# The Python MQbtT library This module supports the implementation of the *optimizer* side communicating with a *miner* (be it `ESP-miner` running on some ESP-based board, or `cgminer` running on a Linux machine controlling some USB-connected ASIC miner). ## Usage of the library The usage is very simple: first you need to create a concrete subclass of {class}`~mqbtt.mqbtt.MQbtTBase` that will be used to communicate with the miner: * {class}`~mqbtt.mqbtt.MQbtTOptimizer` to implement the *optimizer*, * {class}`~mqbtt.mqbtt.MQbtTLogReceiver` to receive the *logs from the miner*, * {class}`~mqbtt.mqbtt.MQbtTMiner` to implement a (mock) *miner*. The **pair id** is a string that should be unique for the pair of optimizer and miner, and should be the same for both parties. ```{eval-rst} .. autoclass:: mqbtt.mqbtt.MQbtTBase :members: ``` ### Messages format The parties exchange messages of various types: * {class}`~mqbtt.proto.Request ` is sent by the optimizer to the miner, * {class}`~mqbtt.proto.Reply` and {class}`~mqbtt.proto.Share` are sent by the miner to the optimizer. Messages use {mod}`dataclasses`, in order to reduce the risk of programming errors, they are *immutable* and their constructor accepts only *keyword arguments*; to create a message, you need to create an instance of the desired class, for instance: ```python request = Request( timestamp_min = 3, timestamp_max = 10, nonce_start = 5, nonce_size = 7, reset = False ) ``` and to access the fields, you can use the dot notation: ```python request.timestamp_min request.timestamp_max request.nonce_start request.nonce_size request.reset ``` Messages are defined as follows. Observe that the {meth}`from_bytes` and {meth}`to_bytes` methods are usually not used directly, but are needed by the {meth}`send` and {meth}`receive` methods of the subclasses of {class}`~mqbtt.mqbtt.MQbtTBase` to serialize and deserialize the messages for transmission. ```{eval-rst} .. autoclass:: mqbtt.proto.Request :members: ``` ```{eval-rst} .. autoclass:: mqbtt.proto.Reply :members: ``` The `Kind` enum is defined as ```{eval-rst} .. autoclass:: mqbtt.proto.Kind :members: :undoc-members: ``` Finally, shares are represented by ```{eval-rst} .. autoclass:: mqbtt.proto.Share :members: ``` ### The `MQbtTOptimizer` The central class of the library is the one that allows to implement the optimizer. ```{eval-rst} .. autoclass:: mqbtt.mqbtt.MQbtTOptimizer :members: ``` A sketch of its usage is given by the following fragment of code ```python from mqbtt import MQbtTOptimizer, Request, Kind # create the connection handle = MQbtTOptimizer('cudone.law.di.unimi.it', 'test') while True: # prepare the request request = Request( timestamp_min= ..., timestamp_max= ..., nonce_start = ..., nonce_size = ..., reset=... ) # send the request handle.send_request(request) ... # receive the reply reply = handle.receive_reply() # process the reply if (reply.kind == Kind.RESULT): ... reply.num_shares ... ... reply.new_block ... # end the communication handle.stop() ``` ### The `MQbtTLogReceiver` Receiving logs is very easy. ```{eval-rst} .. autoclass:: mqbtt.mqbtt.MQbtTLogReceiver :members: ``` A sketch of its usage is given by the following fragment of code ```python from mqbt import MQbtTLogReceiver handle = MQbtTLogReceiver('cudone.law.di.unimi.it', 'test') while True: message = handle.receive_log() ... handle.stop() ``` ### The `MQbtTMiner` For debugging purposes, the library provides a class to implement a (mock) miner. ```{eval-rst} .. autoclass:: mqbtt.mqbtt.MQbtTMiner :members: ``` A sketch of its usage is pretty symmetrical to the one of the optimizer ```python from mqbt import MQbtTMiner, Reply handle = MQbtTMiner(broker_address, pair_id) while True: # receive the request from the optimizer request = handle.receive_request() # process the request and prepare the reply ... request.timestamp_min ... ... request.timestamp_max ... ... request.nonce_start ... ... request.nonce_size ... ... request.reset ... reply = Reply( num_shares= ..., new_block= ..., kind= Kind... ) # send the reply handle.send_reply(reply) handle.stop() ``` ### The `MQbtTDatabase` class As seen in the next section, the library has a tool that can be used to record all the traffic in the `mqbtt` subtopics in a [SQLite](https://www.sqlite.org/) with the following schema ```sql TABLE messages ( id INTEGER PRIMARY KEY AUTOINCREMENT, timestamp REAL NOT NULL, pair_id TEXT NOT NULL, subtopic TEXT NOT NULL CHECK (subtopic IN ('esp', 'log', 'shr', 'srv')), message BLOB NOT NULL ); ``` where `timestamp` is the time of the message in seconds since the epoch for the UTC timezone, `pair_id` is the pair id of the optimizer and miner, `subtopic` is one of the four subtopics of the `mqbtt` topic, and `message` is the serialized message (in bytes). Once the traffic is recorded, one can conveniently use the {class}`replies ` class to read and deserialize the data from the database to further process it. ```{eval-rst} .. autoclass:: mqbtt.persist.MQbtTDataBase :members: ``` The class is conveniente because can be used to obtain a list of deserialized messages represented by the following class ```{eval-rst} .. autoclass:: mqbtt.persist.MQbtTRecord :members: ``` ## The `mqbtt_dump` command As a convenient debugging tool, the `mqbtt_dump` command is provided, its usage is ``` Usage: mqbtt_dump [OPTIONS] Options: --version Show version and exit. --broker TEXT MQTT URI --pid TEXT Pair id --record PATH Path to the record file. --mode [log|miner|optimizer] Operation mode --help Show this message and exit. ``` The tool can be used for two purposes: if the `--pid` parameter is provided, it will *inspect* traffic, while if the `--record` parameter is provided, it will record the traffic in a SQLite database. To **inspect** the traffic, once the broker URI and pair id are specified, the `mqbtt_dump` command, acting as the operation mode requires, will report: - `log`: the log messages received from the miner, - `optimizer`: the {class}`replies ` the *optimizer* receives from the miner, - `miner`: the {class}`requests ` the *miner* receives from the optimizer. On the other hand, to **record** the traffic, once broker URI and the path to the database where to record the messages are provided, the `mqbtt_dump` command will record the traffic in the SQLite database until stopped. The recorded traffic can then be conveniently read using the {class}`replies ` class as shown in the `examples/Analysis.ipynb` [Jupyter](https://jupyter.org/) notebook. ## The example code ### The `logger` example An example of logger is provided in the `examples/logger.py` script. It connects to the broker on `cudone.law.di.unimi.it` using the pair id of `test` and echos every log message it receives. If the testing Bitaxe is running, running ```bash python examples/logger.py ``` should produce an output similar to ```bash I (78745803) bm1368Module: Job ID: 00, Core: 1/1, Ver: 05942000 I (78745803) asic_result: Ver: 25942000 Nonce DE740102 diff 19283.4 of 4096. I (78745813) stratum_api: tx: {"id": 2291, "method": "mining.submit", "params": ["bc1qnp980s5fpp8l94p5cvttmtdqy8rvrq74qly2yrfmzkdsntqzlc5qkc4rkq.bitaxe", "29163cc", "13000000", "67dd7a88", "de740102", "05942000"]} I (78745973) stratum_task: rx: {"id":2291,"error":null,"result":true} I (78745983) stratum_task: message result accepted I (78757453) bm1368Module: Job ID: 28, Core: 40/9, Ver: 08352000 I (78757463) asic_result: Ver: 28352000 Nonce 7E540150 diff 542.9 of 4096. I (78759513) bm1368Module: Job ID: 20, Core: 75/12, Ver: 00A18000 I (78759523) asic_result: Ver: 20A18000 Nonce D1BE0096 diff 322.8 of 4096. I (78761213) bm1368Module: Job ID: 68, Core: 46/6, Ver: 0416C000 I (78761223) asic_result: Ver: 2416C000 Nonce 9CD0015C diff 995.4 of 4096. ``` ### The `optimizer` example An example of optimizer is provided in the `examples/optimizer.py` script. It connects to the broker on `cudone.law.di.unimi.it` using the pair id of `test`; it then sends ten requests to the miner ```python Request( timestamp_min=i, timestamp_max=i + 1, nonce_start = i//2, nonce_size = i // 2 + 1, reset=(i % 2 == 0) ) ``` for `i` from `0` to `9`, and prints for the replies. ### The `echo` fake miner To test the communication of the optimizer, a fake miner is provided that, for every `request` returns the following reply ```python Reply( num_shares=( request.timestamp_min * 1000000 + request.timestamp_max * 10000 + request.nonce_start * 100 + request.nonce_size ), new_block=request.timestamp_min % 2 == 0, kind=Kind(request.timestamp_min % 4), ) ``` Testing the `examples/optimizer.py` script with the `echo` miner requires running ```bash python -m mqbtt.echo ``` and then ```bash python examples/optimizer.py ``` the output of the optimizer should be ```bash Sent request: Request(timestamp_min=0, timestamp_max=1, nonce_start=0, nonce_size=1, reset=True) Received reply: Reply(kind=, num_shares=10001, new_block=True) Sent request: Request(timestamp_min=1, timestamp_max=2, nonce_start=0, nonce_size=1, reset=False) Received reply: Reply(kind=, num_shares=1020001, new_block=False) Sent request: Request(timestamp_min=2, timestamp_max=3, nonce_start=1, nonce_size=2, reset=True) Received reply: Reply(kind=, num_shares=2030102, new_block=True) Sent request: Request(timestamp_min=3, timestamp_max=4, nonce_start=1, nonce_size=2, reset=False) Received reply: Reply(kind=, num_shares=3040102, new_block=False) Sent request: Request(timestamp_min=4, timestamp_max=5, nonce_start=2, nonce_size=3, reset=True) Received reply: Reply(kind=, num_shares=4050203, new_block=True) Sent request: Request(timestamp_min=5, timestamp_max=6, nonce_start=2, nonce_size=3, reset=False) Received reply: Reply(kind=, num_shares=5060203, new_block=False) Sent request: Request(timestamp_min=6, timestamp_max=7, nonce_start=3, nonce_size=4, reset=True) Received reply: Reply(kind=, num_shares=6070304, new_block=True) Sent request: Request(timestamp_min=7, timestamp_max=8, nonce_start=3, nonce_size=4, reset=False) Received reply: Reply(kind=, num_shares=7080304, new_block=False) Sent request: Request(timestamp_min=8, timestamp_max=9, nonce_start=4, nonce_size=5, reset=True) Received reply: Reply(kind=, num_shares=8090405, new_block=True) Sent request: Request(timestamp_min=9, timestamp_max=10, nonce_start=4, nonce_size=5, reset=False) Received reply: Reply(kind=, num_shares=9100405, new_block=False) ``` while the output of the `echo` miner should be ```bash Received request: Request(timestamp_min=0, timestamp_max=1, nonce_start=0, nonce_size=1, reset=True) Sent reply: Reply(kind=, num_shares=10001, new_block=True) Received request: Request(timestamp_min=1, timestamp_max=2, nonce_start=0, nonce_size=1, reset=False) Sent reply: Reply(kind=, num_shares=1020001, new_block=False) Received request: Request(timestamp_min=2, timestamp_max=3, nonce_start=1, nonce_size=2, reset=True) Sent reply: Reply(kind=, num_shares=2030102, new_block=True) Received request: Request(timestamp_min=3, timestamp_max=4, nonce_start=1, nonce_size=2, reset=False) Sent reply: Reply(kind=, num_shares=3040102, new_block=False) Received request: Request(timestamp_min=4, timestamp_max=5, nonce_start=2, nonce_size=3, reset=True) Sent reply: Reply(kind=, num_shares=4050203, new_block=True) Received request: Request(timestamp_min=5, timestamp_max=6, nonce_start=2, nonce_size=3, reset=False) Sent reply: Reply(kind=, num_shares=5060203, new_block=False) Received request: Request(timestamp_min=6, timestamp_max=7, nonce_start=3, nonce_size=4, reset=True) Sent reply: Reply(kind=, num_shares=6070304, new_block=True) Received request: Request(timestamp_min=7, timestamp_max=8, nonce_start=3, nonce_size=4, reset=False) Sent reply: Reply(kind=, num_shares=7080304, new_block=False) Received request: Request(timestamp_min=8, timestamp_max=9, nonce_start=4, nonce_size=5, reset=True) Sent reply: Reply(kind=, num_shares=8090405, new_block=True) Received request: Request(timestamp_min=9, timestamp_max=10, nonce_start=4, nonce_size=5, reset=False) Sent reply: Reply(kind=, num_shares=9100405, new_block=False) ```