My 2016 GSoC Project

Hello again, folks !

* Insert lame excuse about delay here (Shame on me) *

Nowadays, Scilab’s compilation generates 2 executables: scilab-bin, the full-featured Java GUI interface (with Scinotes, Xcos and help browser tools); and scilab-cli-bin, which opens the shell on a simple terminal, and can even be made to run without Java dependencies, by disabling some features.

From a developer’s point of view, that means that Scilab’s codebase already has the groundwork for handling different interfaces, allowing the definition of independent functions for text input and output, while the rest of processing code remains pretty much the same.

As we plan to integrate the Jupyter Kernel/Console/Notebook functionality into Scilab, first of all, read/write functions that implement the Wire Protocol communication must be provided and set according to the defined interface:

// From scilabRead.h
typedef char* (*SCILAB_INPUT_METHOD)( void );
CONSOLE_IMPEXP void setScilabInputMethod( SCILAB_INPUT_METHOD reader );

// From scilabWrite.hxx
typedef void (*SCILAB_OUTPUT_METHOD)( const char *text );
OUTPUT_STREAM_IMPEXP void setScilabOutputMethod( SCILAB_OUTPUT_METHOD writer );

Even with that kind of facilitators, making my Jupyter messaging work part of Scilab is not such an easy task: as I looked for the components that the kernel logic needs to interoperate with, in order to build a complete implementation, a miriad of files and folders, yet to be fully understood, has shown itself.

Gladly, the directory structure related to this work is not that deep, and it was relatively easy to locate the C and C++ header/source files as they were referenced in the code I was inspecting.

Basically, each component used to build the entire Scilab application has its sources grouped in its own subfolder of the “modules” folder, at the root of Scilab’s source code, and it is firstly compiled to a library, before being linked to the main executable.

$ ls scilab/modules/
action_binding/           fftw/                m2sci/                sound/
api_scilab/               fileio/              matio/                sparse/
arnoldi/                  functions/           mexlib/               special_functions/
ast/                      functions_manager/   modules_manager/      spreadsheet/
atoms/                    genetic_algorithms/  mpi/                  startup/
boolean/                  graph/               optimization/         statistics/
cacsd/                    graphic_export/      output_stream/        string/
call_scilab/              graphic_objects/     overloading/          tclsci/
commons/                  graphics/            parallel/             threads/
compatibility_functions/  gui/                 parameters/           time/
completion/               hdf5/                polynomials/          types/
console/                  helptools/           prebuildjava/         ui_data/
core/                     history_browser/     preferences/          umfpack/
coverage/                 history_manager/     randlib/              windows_tools/
data_structures/          integer/             renderer/             xcos/
demo_tools/               interpolation/       scicos/               xml/
development_tools/        io/                  scicos_blocks/        libscilab-cli.la
differential_equations/   javasci/             scinotes/             libscilab.la
dynamic_link/             jupyter/             scirenderer/          Makefile
elementary_functions/     jvm/                 signal_processing/    Makefile.am
external_objects/         linear_algebra/      simulated_annealing/  Makefile.in
external_objects_java/    localization/        slint/

(As you can see, I already created a module folder for the Jupyter kernel code)

The startup module contains the starting point for the application (the main function, named “scilab.cpp”), which parses the command-line options, defines the used input/output methods and calls the [almost self explanatory] higher level Scilab engine initialization functions from the core module:

From InitScilab.h

// Initialization options structure
typedef struct
{
    char* pstParseFile;
    char* pstFile;
    char* pstExec;
    char* pstLang;
    void* pExpTree;
    int iParseTrace;
    int iPrintAst;
    int iExecAst;
    int iDumpAst;
    int iDumpStack;
    int iTimed;
    int iAstTimed;
    int iExecVerbose;
    int iConsoleMode;
    int iNoJvm;
    int iNoStart;
    int iShowVersion;
    int iSerialize;
    int iKeepConsole;
    int iNoBanner;
    int iMultiLine;
    int isInterruptible;
    int isPrioritary;
    int iStartConsoleThread;
    int iForceQuit;
    int iTimeoutDelay;
    int iCodeAction;
    enum command_origin_t iCommandOrigin;
} ScilabEngineInfo;

ScilabEngineInfo* InitScilabEngineInfo();

int StartScilabEngine(ScilabEngineInfo* _pSEI);
int RunScilabEngine(ScilabEngineInfo* _pSEI);
int ExecExternalCommand(ScilabEngineInfo* _pSEI);

void StopScilabEngine(ScilabEngineInfo* _pSEI);

Even if we could add the Jupyter kernel initialization logic to “scilab.cpp” through preprocessor directives (a.k.a. “#ifdef”s), Clément, my mentor, suggested that I created a separate main for it, inside the same startup directory, copying only what is needed, without all the JVM and OpenMPI handling, from the original one.

From jupyter_scilab_kernel.cpp

// ...Includes and definitions...

int main(int argc, char *argv[])
{
    int iRet = 0;

    ScilabEngineInfo* pSEI = InitScilabEngineInfo();

    // Building Scilab without GUI/JAVA features
    pSEI->iConsoleMode = 1;
    pSEI->iNoJvm = 1;
    setScilabMode( SCILAB_NWNI );

    get_option( argc, argv, pSEI ); // Parse command line arguments

    setScilabInputMethod( &JupyterMessageRead );
    setScilabOutputMethod( &JupyterMessagePrint );

    int val = setjmp( ScilabJmpEnv ); // Do not ask me
    if( !val )
    {
        // First argument should be the JSON config file name
        InitializeJupyter( ( argc > 1 ) ? argv[ 1 ] : NULL );
        
        iRet = StartScilabEngine( pSEI );
        if ( iRet == 0 )
        {
            iRet = RunScilabEngine( pSEI );
        }

        StopScilabEngine( pSEI );
        FREE( pSEI );
        
        TerminateJupyter();
        
        return iRet;
    }
    else
    {
        // We probably had a segfault so print error
        std::wcerr << getLastErrorMessage() << std::endl;
        return val;
    }
}

As we can’t run the connection update loop inside the main function anymore, and wouldn’t want to block execution of other tasks during message receiving/sending as well, the Jupyter kernel, now added as a new class, starts a separate connection thread on its static initializer, to handle the input (commands) and output (results) queue asynchronously.

From JupyterKernel.hxx

#include "JupyterMessage.hxx"
#include "JupyterKernelConnection.hxx"

#include <json/json.h>  // JsonCpp functions

// ...

class JupyterKernel 
{
public:
  static int Initialize( std::string );
  static char* GetInputString();
  static void SetOutputString( const char* );
  static void Terminate();
  
private:
  // Message handler function declarations
  static void HandleShutdownRequest( Json::Value&, Json::Value& );
  static void FillKernelInfo( Json::Value& );
  static void FillConnectionInfo( Json::Value&, Json::Value& );
  static void HandleHistoryRequest( Json::Value&, Json::Value& );
  static void HandleInspectionRequest( Json::Value&, Json::Value& );
  static void HandleCompletenessRequest( Json::Value&, Json::Value& );
  static void HandleCompletionRequest( Json::Value&, Json::Value& );
  
  static void HandleExecutionRequest( JupyterKernelConnection&, JupyterMessage&, Json::Value& );
  
  static std::thread updateThread;
  static void RunUpdateThread( Json::Value );
  static std::mutex processingLock;
  
  static std::mutex inputLock, outputLock;
  static std::queue<std::string> inputCommandQueue, outputResultQueue;

  static bool isRunning;
  
  static const std::string USER_NAME;
};

To effectively make our new module work as a internal library, we mimic the way other modules work, exporting symbols for wrapper C-style functions that call kernel class methods:

From dynlib_jupyter.h

#ifdef _MSC_VER
#ifdef JUPYTER_EXPORTS
#define JUPYTER_IMPEXP __declspec(dllexport)
#else
#define JUPYTER_IMPEXP __declspec(dllimport)
#endif
#else
#define JUPYTER_IMPEXP
#endif

From InitializeJupyter.h

// Calls JupyterKernel::Initialize
JUPYTER_IMPEXP void InitializeJupyter( const char* );
// Calls JupyterKernel::Terminate
JUPYTER_IMPEXP void TerminateJupyter();

From JupyterMessagePrint.h

// Calls JupyterKernel::SetOutputString
JUPYTER_IMPEXP void JupyterMessagePrint( const char *line );

From JupyterMessageRead.h

// Calls JupyterKernel::GetInputString
JUPYTER_IMPEXP char* JupyterMessageRead( void );

With the code implemented so far, we should be able to handle simple “{execute,kernel_info,connect,shutdown}_request” calls properly. What remains to be done, regarding message handling, are command history and code completion/completeness/introspection, for which the needed Scilab functions are still being discovered.

But before that, I pretend to add my new sources to the [apparently complicated] build process, with Clément’s guidance, it seems, based on the makefiles he helpfully provided me.

Thanks for your reading time. Until next post !!

My 2016 GSoC Project - Part XI

New Home, New Rules