(First of all, I’m truly sorry for slacking off in the past week. I had another Mid Term Evaluation to deal with: my Master’s Degree one. Combine the pressure of that with some frustrations afterwards and Voila ! Wasted time…

Not that I’m trying to indulge myself. I recognize being a little lazy and careless with my planning, and in the reduced time I got to work on my project, the first big roadblocks slowed my progress down.

Things just started to get as messy as my life… *mandatory lol*

Anyways, as GSoC Mid Term Evaluations are approaching as well, it’s time to shift one gear up to overcome the obstacles and compensate for lagging behind.)

Man, Jupyter’s Wire Protocol surely is chatty !! There’s a big number of messages being exchanged between frontend and backend for some operations, and took me some time to understand it well enough.

More than that, each Jupyter message is itself a ZeroMQ multipart message, sent as a series of frames, as mentioned in a previous article: Identifier, Delimiter, Hash, Header, Parent Header, Metadata and Content.

That many messages reminded me of one song by the always awesome Tarja Turunen (actually only the title, that I used for this post):

(A lot of messages, a lot of letters, y’know… makes sense, right ?… I hope so…)

Musical taste pushing aside, keeping track of the order and content of the protocol messages was hard, but in the end my previous sources of information didn’t fail me.

As the Jupyter Documentation is kinda arid at some places, and the IPython example kernels are outdated, I found myself going back and forth between both to fill the gaps of my understanding. I even installed and runned the Simple Kernel project to see which messages the frontend was sending at initialization (the author implemented some debugging info in its code, which was great for me).

The most used ZeroMQ sockets used in Jupyter are the Shell one, that receives all the non high priority requests from clients (high priority goes through Control socket), and the I/O Pub one, used to publish the kernel state and outputs to all the frontends available. Most of the differences between the various types of messages are in their Content frames

At the processes start, called with a command like

# We use --Session.key="b''" to bypass authentication for now. More on that later
$ jupyter qtconsole --kernel scilab --Session.key="b''"

the qtconsole client application there sends three requests to the given kernel:

  • Kernel Info Request, answered with a Kernel Info Reply, containing information about the language used and the kernel implementation:
content = {
    'protocol_version': '5.0',
    'implementation': 'scilab',
    'implementation_version': '0.1',
    'language_info': {
        'name': 'scilab',
        'version': '6.0',
        'mimetype': '',
        'file_extension': '.sce',
        'pygments_lexer': '',
        'codemirror_mode': '',
        'nbconvert_exporter': '',
    },
    'banner' : 'Scilab Kernel',
    'help_links': []
}
  • History Request, answered with a History Reply, containing a list of all previous commands stored and its outputs, if defined (in the begining, I just return an empty list, but I’ll need to implement a proper history log later):
content = {
    'history': []
}
  • Execute Request with an empty code string

Predictably, as this is the main purpose of a kernel, the more complex communications involve the code execution request by the frontend, that sends the user input string through the Shell socket, which is answered by the kernel with a series of messages as follows:

  • Publishes (at the I/O Pub socket) the status of the kernel as “busy” (processing code)
  • Publishes the execution input so that every client knows what is being processed
  • After the processing (I guess), publishes the stream, indicating the text to be written in the given output channel (like “stdout”)
  • Publishes the execution results, be they text, images, whatever is supported. The type of the output is indicated as well
  • Publishes the status of the kernel as “idle” (ready to accept new commands)
  • Write on the Shell socket the Execute Reply message, containing the result status (like “ok” or “error”) and the updated execution counter (incremented at each succesful call)

Those are more or less implemented, except that we do not do any processing (leveraging Scilab libraries) yet, replying to execution requests instantly. There are other kinds of requests like Connection, Introspection, Code Completion, Code Completeness verification, Comm Info, etc., that still should be coded here.

The good news is that with what we have now, the frontend can get enough feedback to start requesting user input and updating command count for each (failed) execution request:

$ jupyter console --kernel scilab --Session.key="b''"  
[ZMQTerminalIPythonApp] WARNING | Message signing is disabled.  This is insecure and not recommended!
Reading config file /run/user/1000/jupyter/kernel-9595.json
{
        "control_port" : 45893,
        "hb_port" : 60093,
        "iopub_port" : 42615,
        "ip" : "127.0.0.1",
        "kernel_name" : "scilab",
        "key" : "",
        "shell_port" : 55787,
        "signature_scheme" : "hmac-sha256",
        "stdin_port" : 36317,
        "transport" : "tcp"
}
UUID: d69933a6-83de-4e5b-aa09-d2cc0aeccf38
Delimiter: <IDS|MSG>
HMAC: <IDS|MSG>
Header: {
        "date" : "2016-06-10T06:31:56.724735",
        "msg_id" : "1e811623-8530-4e30-9eac-6cc46febeb47",
        "msg_type" : "kernel_info_request",
        "session" : "d69933a6-83de-4e5b-aa09-d2cc0aeccf38",
        "username" : "leonardojc",
        "version" : "5.0"
}
Parent Header: {}
Metadata: {}
Content: {}

Received Shell request !: kernel_info_request
Kernel info request
Sending data:
d69933a6-83de-4e5b-aa09-d2cc0aeccf38
<IDS|MSG>

{"date":"2016-06-10T06:31:56.725778","msg_id":"5020e4e0-7a5b-4f71-b149-153c26b5c2c5","msg_type":"kernel_info_reply","session":"91778d4d-8d9c-4677-934e-c3d737ab61c0","username":"kernel","version":"5.0"}

{"date":"2016-06-10T06:31:56.724735","msg_id":"1e811623-8530-4e30-9eac-6cc46febeb47","msg_type":"kernel_info_request","session":"d69933a6-83de-4e5b-aa09-d2cc0aeccf38","username":"leonardojc","version":"5.0"}

{}

{"banner":"Scilab Kernel","implementation":"scilab","implementation_version":"0.1","language_info":{"codemirror_mode":"","file_extension":".sce","mimetype":"","name":"scilab","nbconvert_exporter":"","pygments_lexer":"","version":"6.0"},"protocol_version":"5.0"}

UUID: d69933a6-83de-4e5b-aa09-d2cc0aeccf38
Delimiter: <IDS|MSG>
HMAC: <IDS|MSG>
Header: {
        "date" : "2016-06-10T06:31:56.767649",
        "msg_id" : "3d85d2e5-aeb9-44e5-9902-5690e5d6c145",
        "msg_type" : "history_request",
        "session" : "d69933a6-83de-4e5b-aa09-d2cc0aeccf38",
        "username" : "leonardojc",
        "version" : "5.0"
}
Parent Header: {}
Metadata: {}
Content: {
        "hist_access_type" : "tail",
        "n" : 1000,
        "output" : false,
        "raw" : true
}

Received Shell request !: history_request
History request
Sending data:
d69933a6-83de-4e5b-aa09-d2cc0aeccf38
<IDS|MSG>

{"date":"2016-06-10T06:31:56.768220","msg_id":"c8d931c9-4be4-42af-a320-62829d80b71c","msg_type":"history_reply","session":"91778d4d-8d9c-4677-934e-c3d737ab61c0","username":"kernel","version":"5.0"}

{"date":"2016-06-10T06:31:56.767649","msg_id":"3d85d2e5-aeb9-44e5-9902-5690e5d6c145","msg_type":"history_request","session":"d69933a6-83de-4e5b-aa09-d2cc0aeccf38","username":"leonardojc","version":"5.0"}

{}

{"history":[]}

Jupyter Console 4.1.1

UUID: d69933a6-83de-4e5b-aa09-d2cc0aeccf38
Delimiter: <IDS|MSG>
HMAC: <IDS|MSG>
Header: {
        "date" : "2016-06-10T06:31:56.830672",
        "msg_id" : "b9584a63-9f62-4772-b266-dd2a94410bb2",
        "msg_type" : "kernel_info_request",
        "session" : "d69933a6-83de-4e5b-aa09-d2cc0aeccf38",
        "username" : "leonardojc",
        "version" : "5.0"
}
Parent Header: {}
Metadata: {}
Content: {}

Received Shell request !: kernel_info_request
Kernel info request
Sending data:
d69933a6-83de-4e5b-aa09-d2cc0aeccf38
<IDS|MSG>

{"date":"2016-06-10T06:31:56.831192","msg_id":"c051f171-42ad-4665-8470-9fae75551d13","msg_type":"kernel_info_reply","session":"91778d4d-8d9c-4677-934e-c3d737ab61c0","username":"kernel","version":"5.0"}

{"date":"2016-06-10T06:31:56.830672","msg_id":"b9584a63-9f62-4772-b266-dd2a94410bb2","msg_type":"kernel_info_request","session":"d69933a6-83de-4e5b-aa09-d2cc0aeccf38","username":"leonardojc","version":"5.0"}

{}

{"banner":"Scilab Kernel","implementation":"scilab","implementation_version":"0.1","language_info":{"codemirror_mode":"","file_extension":".sce","mimetype":"","name":"scilab","nbconvert_exporter":"","pygments_lexer":"","version":"6.0"},"protocol_version":"5.0"}

UUID: d69933a6-83de-4e5b-aa09-d2cc0aeccf38
Delimiter: <IDS|MSG>
HMAC: <IDS|MSG>
Header: {
        "date" : "2016-06-10T06:31:56.831649",
        "msg_id" : "a3909d5d-466b-44ab-8adf-674afd49a116",
        "msg_type" : "history_request",
        "session" : "d69933a6-83de-4e5b-aa09-d2cc0aeccf38",
        "username" : "leonardojc",
        "version" : "5.0"
}
Parent Header: {}
Metadata: {}
Content: {
        "hist_access_type" : "tail",
        "n" : 1000,
        "output" : false,
        "raw" : true
}

Received Shell request !: history_request
History request
Sending data:
d69933a6-83de-4e5b-aa09-d2cc0aeccf38
<IDS|MSG>

{"date":"2016-06-10T06:31:56.832105","msg_id":"3ceaa24a-9ab5-4da3-93cc-d61b8a7e300d","msg_type":"history_reply","session":"91778d4d-8d9c-4677-934e-c3d737ab61c0","username":"kernel","version":"5.0"}

{"date":"2016-06-10T06:31:56.831649","msg_id":"a3909d5d-466b-44ab-8adf-674afd49a116","msg_type":"history_request","session":"d69933a6-83de-4e5b-aa09-d2cc0aeccf38","username":"leonardojc","version":"5.0"}

{}

{"history":[]}


In [1]:

Another good news is that I learned how to use JsonCpp properly for parsing strings and doing serialization with the Reader and FastWriter classes. No more string stream abuses…

That’s what I have for now. I’ll try to come back soon to talk about what is being a major pain: Authentication.

See ya !!