The Python MQbtT library¶
This module supports the implementation of the optimizer side communicating
with a miner (be it ESP-miner running on some ESP-based board, or cgminer
running on a Linux machine controlling some USB-connected ASIC miner).
Usage of the library¶
The usage is very simple: first you need to create a concrete subclass of
MQbtTBase that will be used to communicate with the miner:
MQbtTOptimizerto implement the optimizer,MQbtTLogReceiverto receive the logs from the miner,MQbtTMinerto implement a (mock) miner.
The pair id is a string that should be unique for the pair of optimizer and miner, and should be the same for both parties.
- class MQbtTBase(broker_uri: str, pair_id: str, receive_subtopics: list[tuple[str, int]], keepalive: int = 60, clean_session: bool = False)¶
Base class for all MQbtT classes.
This class is not meant to be used directly, but only as a base class for other classes. It provides the basic functionality to connect to the MQTT broker and to send and receive messages.
- Parameters:
broker_uri – the URI of the MQTT broker (example: mqtt://localhost)
pair_id – the pair id of the client
receive_subtopics – a list of (subtopic, qos) to receive from (i.e.: esp, srv or log)
keepalive – maximum number of seconds allowed between communications with the broker, passed to
paho.mqtt.client.Client.connect()clean_session – if
True, the broker discards this client’s subscription and any queued QoS 1/2 backlog the moment it disconnects; ifFalse(the default here), the broker keeps a persistent session across reconnects, so a briefly-disconnected client does not lose QoS 1/2 messages queued for it in the meantime. Concrete subclasses may default this differently according to their own reliability/broker-load trade-off (seeMQbtTLogReceiver).
Note
Receiving methods
This class uses internally a
Queueper subtopic, starts theMQTT Clientand registers acallbackthat pushes all the messages for the subtopic to its queue.Subclasses use such queues to provide a synchronous way to receive messages via
receive_T()methods (where T is the type of message the subclass handles), and check viahas_T()if there are messages to be received.Receiving methods have a
timeoutparameter that (if positive) specifies the maximum number of seconds to wait for the queue to become not empty. If for whatever reason a message does not show up in the queue within the timeout, the receiving method raises aEmptyexception.In case the
timeoutparameter isNone(the default), theblockingparameter can be set totrueso that the call blocks until the queue is not empty; if set tofalse, the call returns immediately (possibly aNoneif the queue is empty), but never raises theEmptyexception.Sending methods
Subclasses provide
send_T()to send messages of type T.Such methods have a
blockingparameter that if set totrue(the default) makes the method block until the message is sent to the MQTT broker.Important notes
There is a subtle difference between receiving and sending methods: on one hand, for receiving methods, blocking or timeouts may depend (indistinguishably) both on the MQTT broker or the sender, this implies that there is no way to distinguish if the reason for the exception is a networking issue, the broker or sender malfunctioning, or simply that the sender is silent. On the other hand, for sending methods, blocking depends just on the MQTT broker (that may be unreachable due to networking issues, or malfunctioning), but never on the receiver.
It is of paramount importance for users of subclasses to receive messages in order for the queue non to grow indefinitely. Once the communication is over, the
stop()method must be called to stop the MQTT client and to disconnect from the broker.- stop()¶
Stop the MQTT client and disconnect from the broker.
Messages format¶
The parties exchange messages of various types:
Requestis sent by the optimizer to the miner,
Messages use dataclasses, in order to reduce the risk of programming
errors, they are immutable and their constructor accepts only keyword
arguments; to create a message, you need to create an instance of the desired
class, for instance:
request = Request(
timestamp_min = 3,
timestamp_max = 10,
nonce_start = 5,
nonce_size = 7,
reset = False
)
and to access the fields, you can use the dot notation:
request.timestamp_min
request.timestamp_max
request.nonce_start
request.nonce_size
request.reset
Messages are defined as follows. Observe that the from_bytes() and
to_bytes() methods are usually not used directly, but are needed by the
send() and receive() methods of the subclasses of
MQbtTBase to serialize and deserialize the messages for
transmission.
- class Request(*, timestamp_min: int = 0, timestamp_max: int = 0, nonce_start: int = 0, nonce_size: int = 0, reset: bool = False)¶
Request message sent by the optimizer to the miner.
- Parameters:
timestamp_min – Minimum timestamp.
timestamp_max – Maximum timestamp.
nonce_start – Minimum nonce.
nonce_size – Maximum nonce.
reset – Whether to reset the state.
- class Reply(kind: Kind = Kind.RESULT, num_shares: int = 0, new_block: bool = False)¶
Reply message from the miner to the optimizer.
- Parameters:
kind – The kind of reply (e.g., RESULT).
num_shares – Number of shares in the reply.
new_block – Whether a new block was found.
The Kind enum is defined as
Finally, shares are represented by
Share message from the miner to the optimizer.
- Parameters:
version – The version rolled by the miner.
prev_block – The hash of the previous block.
merkle_root – The bytes of the merkle root.
time – The timestamp of the share.
pool_time – The timestamp provided by the mining pool.
bits – The difficulty bits.
nonce – The nonce rolled by the miner.
pool_diff – The pool difficulty.
job_id – The job ID.
extranonce2 – The extranonce2.
diff – The difficulty of the share.
zeroes – The (approximate) number of leading binary zeroes in the share.
Deserialize bytes into a Share object.
- Parameters:
b – The bytes to deserialize.
- Returns:
The deserialized Share object.
Serialize the Share to bytes using protobuf.
- Returns:
The serialized protobuf representation of the reply.
The MQbtTOptimizer¶
The central class of the library is the one that allows to implement the optimizer.
- class MQbtTOptimizer(broker_uri: str, pair_id: str, keepalive: int = 60, clean_session: bool = False)¶
Class for the optimizer to send and receive messages to the miner.
- has_reply() bool¶
Check if there is a reply from the miner.
- Returns:
True if there is a reply, False otherwise.
Check if there is a share from the miner.
- Returns:
True if there is a share, False otherwise.
- receive_reply(blocking: bool = True, timeout: float | None = None) Reply | None¶
Receive a reply from the miner.
- Parameters:
blocking – Whether to block until the queue becomes not empty.
timeout – The maximum time to wait for the queue to become not empty.
Receive a share from the miner.
- Parameters:
blocking – Whether to block until a share is received.
A sketch of its usage is given by the following fragment of code
from mqbtt import MQbtTOptimizer, Request, Kind
# create the connection
handle = MQbtTOptimizer('cudone.law.di.unimi.it', 'test')
while True:
# prepare the request
request = Request(
timestamp_min= ...,
timestamp_max= ...,
nonce_start = ...,
nonce_size = ...,
reset=...
)
# send the request
handle.send_request(request)
...
# receive the reply
reply = handle.receive_reply()
# process the reply
if (reply.kind == Kind.RESULT):
... reply.num_shares ...
... reply.new_block ...
# end the communication
handle.stop()
The MQbtTLogReceiver¶
Receiving logs is very easy.
- class MQbtTLogReceiver(broker_uri: str, pair_id: str, keepalive: int = 60, clean_session: bool = True)¶
Class to receive log messages from the MQTT broker.
Unlike
MQbtTBase,clean_sessiondefaults toTruehere: log messages are informational (QoS 1), so it is preferable to bound the broker-side backlog a disconnected or abandoned receiver could accumulate rather than to guarantee delivery of every log line.
A sketch of its usage is given by the following fragment of code
from mqbt import MQbtTLogReceiver
handle = MQbtTLogReceiver('cudone.law.di.unimi.it', 'test')
while True:
message = handle.receive_log()
...
handle.stop()
The MQbtTMiner¶
For debugging purposes, the library provides a class to implement a (mock) miner.
- class MQbtTMiner(broker_uri: str, pair_id: str, keepalive: int = 60, clean_session: bool = False)¶
Class for a (mock) miner to send and receive messages to the optimizer.
A sketch of its usage is pretty symmetrical to the one of the optimizer
from mqbt import MQbtTMiner, Reply
handle = MQbtTMiner(broker_address, pair_id)
while True:
# receive the request from the optimizer
request = handle.receive_request()
# process the request and prepare the reply
... request.timestamp_min ...
... request.timestamp_max ...
... request.nonce_start ...
... request.nonce_size ...
... request.reset ...
reply = Reply(
num_shares= ...,
new_block= ...,
kind= Kind...
)
# send the reply
handle.send_reply(reply)
handle.stop()
The MQbtTDatabase class¶
As seen in the next section, the library has a tool that can be used to record
all the traffic in the mqbtt subtopics in a SQLite
with the following schema
TABLE messages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp REAL NOT NULL,
pair_id TEXT NOT NULL,
subtopic TEXT NOT NULL CHECK (subtopic IN ('esp', 'log', 'shr', 'srv')),
message BLOB NOT NULL
);
where timestamp is the time of the message in seconds since the epoch for the
UTC timezone, pair_id is the pair id of the optimizer and miner, subtopic is
one of the four subtopics of the mqbtt topic, and message is the serialized
message (in bytes).
Once the traffic is recorded, one can conveniently use the replies class to read and deserialize the data from the
database to further process it.
- class MQbtTDataBase(path: str, batch_size: int = 100, batch_timeout: float = 10.0)¶
A SQLite database for MQbtT messages.
- close()¶
Close the database connection.
- insert(record: MQbtTRecord)¶
Insert a mqtt record into the database with batched commits.
- logs(pair_id: str | None = None, min_timestamp: datetime | None = None, max_timestamp: datetime | None = None, text: str | None = None, desc: bool = False) Generator[MQbtTRecord, None, None]¶
Query the database for log messages, auto-deserializing the message payload.
- Parameters:
pair_id – The pair_id to filter by.
min_timestamp – The minimum timestamp to filter by.
max_timestamp – The maximum timestamp to filter by.
text – Optional text to filter log messages by.
desc – Whether to order the results in descending timestamp order.
- records(pair_id: str | None = None, subtopic: str | None = None, min_timestamp: datetime | None = None, max_timestamp: datetime | None = None, desc: bool = False) Generator[MQbtTRecord, None, None]¶
Query the database for messages, auto-deserializing the message payload.
- Parameters:
pair_id – The pair_id to filter by.
subtopic – The subtopic to filter by.
min_timestamp – The minimum timestamp to filter by.
max_timestamp – The maximum timestamp to filter by.
desc – Whether to order the results in descending timestamp order.
The class is conveniente because can be used to obtain a list of deserialized messages represented by the following class
- class MQbtTRecord(*, id: int, pair_id: str, subtopic: str, timestamp: datetime, payload: bytes)¶
A (deserialized) record from a MQbtTDataBase.
- Parameters:
id – The ID of the message in the database.
pair_id – The pair_id of the message.
subtopic – The subtopic of the message.
timestamp – The timestamp of the message recording.
message – The deserialized message payload (may be None).
payload – The raw message payload (bytes, may be None).
External services APIs¶
The library provides some simplified way of calling AxeOS and OpenObserve APIs.
More in detail, for AxeOS:
- axeos_info(pid: str) dict¶
Get system information from the AxeOS API.
- Parameters:
pid – The pair identifier or IP address of the device.
- Returns:
A dictionary containing the system information.
and
- axeos_restart(pid: str) None¶
Restart the device using the AxeOS API.
- Parameters:
pid – The pair identifier or IP address of the device.
Finally, to record an event in OpenObserve history stream:
- o2_history(pid: str, detail: str) None¶
Log an event to OpenObserve history stream.
- Parameters:
pid – The pair identifier or IP address of the device.
detail – The detail to log in the history.
All these functions may raise an APIException in case of
network errors (including timeouts).
A convenience function to get the AxeOS web interface URL from the pair id is also provided:
- pid2url(pid: str) str¶
Converts a pair identifier to the corresponding AxeOS API base URL.
If the
pidis not known, it will be returned as is, assuming it is an IP address.- Parameters:
pid – The pair identifier or IP address of the device.
- Returns:
The base URL of the AxeOS API for the given pair identifier or IP address.
Command line tools¶
The library comprises sveral command line tools that can be used to test and the communication with the miner and the optimizer, as well as automatically upgrade to the latest version.
mqbtt_dump¶
As a convenient debugging tool, the mqbtt_dump command is provided, its usage is
Usage: mqbtt_dump [OPTIONS]
Options:
--version Show version and exit.
--broker TEXT MQTT host, default: cudone.law.di.unimi.it
--pid TEXT Pair id [required]
--mode [log|miner|optimizer] Operation mode
--help Show this message and exit.
Once the broker host and pair id are specified, the mqbtt_dump command, acting
as the operation mode requires, will report:
mqbtt_record¶
To keep track of the traffic, the mqbtt_record command is provided, its usage is
Usage: mqbtt_record [OPTIONS]
Options:
--version Show version and exit.
--broker TEXT MQTT host, default: cudone.law.di.unimi.it
--db PATH Path to the record database. [required]
--exclude TEXT Comma separated list of subtopics to exclude (in: esp, log,
shr, srv).
--help Show this message and exit.
To record the traffic, once broker host and the path to the database where to record the messages are provided, the command will record the traffic in the SQLite database until stopped.
The recorded traffic can then be conveniently read using the replies class as shown in the examples/Analysis.ipynb
Jupyter notebook.
Recovering the logs¶
As an example, once some traffic is in the database, the logs can be retrieved with
the examples/db2logspy script that, invoked like
python examples/db2logs.py traffic.db gamma01
will generate an output similar to the one below
2025-05-22 11:15:46+0200 I (4398308) create_jobs_task_Stefano: nTime/nonce range COMPLETED: 0 ACCEPTED shares found.
2025-05-22 11:15:46+0200 I (4398313) mqbtt_mqtt: Published to mqbtt/gamma01/esp, msg_id=23958
2025-05-22 11:15:46+0200 I (4398316) mqbtt_protocol: Sent reply: kind=1, num_shares=0, new_block=0
2025-05-22 11:15:47+0200 I (4398562) mqbtt_protocol: Received request: timestamp_min=2048, timestamp_max=3071, nonce_start=120, nonce_size=8, reset=0
2025-05-22 11:15:47+0200 I (4398566) create_jobs_task_Stefano: Server asked NEW nTime range: (2048-3071)
2025-05-22 11:15:47+0200 I (4398574) create_jobs_task_Stefano: Server asked NEW Nonce range: 16/512 values (3.12% fullscan) starting at Byte1=240
2025-05-22 11:15:47+0200 I (4398584) create_jobs_task_Stefano: Inter-job time: 4.0 ms
2025-05-22 11:15:47+0200 I (4398591) create_jobs_task_Stefano: Starting Nonce range: 16/512 values (3.12% fullscan) starting at Byte1=240 - nTime range: 2048-3071, old Block
2025-05-22 11:15:47+0200 I (4399069) bm1370Module: Job ID: 48, Core: 20/6, Ver: 00024000
2025-05-22 11:15:47+0200 I (4399072) asic_result: Ver: 20024000 Nonce C5E6FC28 diff 1636.0 of 32768.```
mqbtt_echo¶
This commands provides a sort of fake miner that can be used to test the
communication of the optimizer; for every request it returns the following
reply
Reply(
num_shares=(
request.timestamp_min * 1000000
+ request.timestamp_max * 10000
+ request.nonce_start * 100
+ request.nonce_size
),
new_block=request.timestamp_min % 2 == 0,
kind=Kind(request.timestamp_min % 4),
)
Its usage is
Usage: mqbtt_echo [OPTIONS]
Options:
--version Show version and exit.
--broker TEXT MQTT host, default: cudone.law.di.unimi.it
--pid TEXT Pair identifier
--help Show this message and exit.
mqbtt_upgrade¶
This is experimental, if run it should upgrade the library to the latest version. Needs a working DNS and network connection. Its usage is
Usage: mqbtt_upgrade [OPTIONS]
Options:
--version Show version and exit.
--help Show this message and exit.
mqbtt_restart¶
This is another experimental command that can be used to restart an AxeOS-based miner and log the restart event to OpenObserve. Its usage is
Usage: mqbtt_restart [OPTIONS]
Options:
--version Show version and exit.
--pid TEXT Pair identifier, default: test
--detail TEXT The detail of the reason for which the restart is requested
--help Show this message and exit.
mqbtt_wslog¶
In case you want to receive and store the full length log messages that are routed via websocket you can you the mqbtt_wslog command, its usage is
❯ mqbtt_wslog --help
Usage: mqbtt_wslog [OPTIONS]
Options:
--version Show version and exit.
--pid TEXT Pair identifier, default: test
--file PATH Path to the file where to store the logs. [required]
--help Show this message and exit.
Please not that logs are no more transmitted via MQTT once a websocket connection is opened, so use this command judiciously (during its usage Open Observe will not receive any log messages).
The example code¶
The logger example¶
An example of logger is provided in the examples/logger.py script. It connects
to the broker on cudone.law.di.unimi.it using the pair id of test and echos
every log message it receives.
If the testing Bitaxe is running, running
python examples/logger.py
should produce an output similar to
I (78745803) bm1368Module: Job ID: 00, Core: 1/1, Ver: 05942000
I (78745803) asic_result: Ver: 25942000 Nonce DE740102 diff 19283.4 of 4096.
I (78745813) stratum_api: tx: {"id": 2291, "method": "mining.submit", "params": ["bc1qnp980s5fpp8l94p5cvttmtdqy8rvrq74qly2yrfmzkdsntqzlc5qkc4rkq.bitaxe", "29163cc", "13000000", "67dd7a88", "de740102", "05942000"]}
I (78745973) stratum_task: rx: {"id":2291,"error":null,"result":true}
I (78745983) stratum_task: message result accepted
I (78757453) bm1368Module: Job ID: 28, Core: 40/9, Ver: 08352000
I (78757463) asic_result: Ver: 28352000 Nonce 7E540150 diff 542.9 of 4096.
I (78759513) bm1368Module: Job ID: 20, Core: 75/12, Ver: 00A18000
I (78759523) asic_result: Ver: 20A18000 Nonce D1BE0096 diff 322.8 of 4096.
I (78761213) bm1368Module: Job ID: 68, Core: 46/6, Ver: 0416C000
I (78761223) asic_result: Ver: 2416C000 Nonce 9CD0015C diff 995.4 of 4096.
The optimizer example¶
An example of optimizer is provided in the examples/optimizer.py script. It
connects to the broker on cudone.law.di.unimi.it using the pair id of test;
it then sends ten requests to the miner
Request(
timestamp_min=i, timestamp_max=i + 1,
nonce_start = i//2, nonce_size = i // 2 + 1,
reset=(i % 2 == 0)
)
for i from 0 to 9, and prints for the replies.
The echo fake miner¶
Testing the examples/optimizer.py script with the echo miner requires running
mqbtt_echo paid_ir
and then
python examples/optimizer.py pair_id
the output of the optimizer should be
Sent request: Request(timestamp_min=0, timestamp_max=1, nonce_start=0, nonce_size=1, reset=True)
Received reply: Reply(kind=<Kind.HELLO: 0>, num_shares=10001, new_block=True)
Sent request: Request(timestamp_min=1, timestamp_max=2, nonce_start=0, nonce_size=1, reset=False)
Received reply: Reply(kind=<Kind.RESULT: 1>, num_shares=1020001, new_block=False)
Sent request: Request(timestamp_min=2, timestamp_max=3, nonce_start=1, nonce_size=2, reset=True)
Received reply: Reply(kind=<Kind.RESET: 2>, num_shares=2030102, new_block=True)
Sent request: Request(timestamp_min=3, timestamp_max=4, nonce_start=1, nonce_size=2, reset=False)
Received reply: Reply(kind=<Kind.BYE: 3>, num_shares=3040102, new_block=False)
Sent request: Request(timestamp_min=4, timestamp_max=5, nonce_start=2, nonce_size=3, reset=True)
Received reply: Reply(kind=<Kind.HELLO: 0>, num_shares=4050203, new_block=True)
Sent request: Request(timestamp_min=5, timestamp_max=6, nonce_start=2, nonce_size=3, reset=False)
Received reply: Reply(kind=<Kind.RESULT: 1>, num_shares=5060203, new_block=False)
Sent request: Request(timestamp_min=6, timestamp_max=7, nonce_start=3, nonce_size=4, reset=True)
Received reply: Reply(kind=<Kind.RESET: 2>, num_shares=6070304, new_block=True)
Sent request: Request(timestamp_min=7, timestamp_max=8, nonce_start=3, nonce_size=4, reset=False)
Received reply: Reply(kind=<Kind.BYE: 3>, num_shares=7080304, new_block=False)
Sent request: Request(timestamp_min=8, timestamp_max=9, nonce_start=4, nonce_size=5, reset=True)
Received reply: Reply(kind=<Kind.HELLO: 0>, num_shares=8090405, new_block=True)
Sent request: Request(timestamp_min=9, timestamp_max=10, nonce_start=4, nonce_size=5, reset=False)
Received reply: Reply(kind=<Kind.RESULT: 1>, num_shares=9100405, new_block=False)
while the output of the mqbtt_echo should be
Received request: Request(timestamp_min=0, timestamp_max=1, nonce_start=0, nonce_size=1, reset=True)
Sent reply: Reply(kind=<Kind.HELLO: 0>, num_shares=10001, new_block=True)
Received request: Request(timestamp_min=1, timestamp_max=2, nonce_start=0, nonce_size=1, reset=False)
Sent reply: Reply(kind=<Kind.RESULT: 1>, num_shares=1020001, new_block=False)
Received request: Request(timestamp_min=2, timestamp_max=3, nonce_start=1, nonce_size=2, reset=True)
Sent reply: Reply(kind=<Kind.RESET: 2>, num_shares=2030102, new_block=True)
Received request: Request(timestamp_min=3, timestamp_max=4, nonce_start=1, nonce_size=2, reset=False)
Sent reply: Reply(kind=<Kind.BYE: 3>, num_shares=3040102, new_block=False)
Received request: Request(timestamp_min=4, timestamp_max=5, nonce_start=2, nonce_size=3, reset=True)
Sent reply: Reply(kind=<Kind.HELLO: 0>, num_shares=4050203, new_block=True)
Received request: Request(timestamp_min=5, timestamp_max=6, nonce_start=2, nonce_size=3, reset=False)
Sent reply: Reply(kind=<Kind.RESULT: 1>, num_shares=5060203, new_block=False)
Received request: Request(timestamp_min=6, timestamp_max=7, nonce_start=3, nonce_size=4, reset=True)
Sent reply: Reply(kind=<Kind.RESET: 2>, num_shares=6070304, new_block=True)
Received request: Request(timestamp_min=7, timestamp_max=8, nonce_start=3, nonce_size=4, reset=False)
Sent reply: Reply(kind=<Kind.BYE: 3>, num_shares=7080304, new_block=False)
Received request: Request(timestamp_min=8, timestamp_max=9, nonce_start=4, nonce_size=5, reset=True)
Sent reply: Reply(kind=<Kind.HELLO: 0>, num_shares=8090405, new_block=True)
Received request: Request(timestamp_min=9, timestamp_max=10, nonce_start=4, nonce_size=5, reset=False)
Sent reply: Reply(kind=<Kind.RESULT: 1>, num_shares=9100405, new_block=False)