Spark_Handbook

The code necessary to build spark, both ours and external can be obtain from the git repo at git@bitbucket.org:bluetradesys/spark.git. To get started use:

git clone --recursive git@bitbucket.org:bluetradesys/spark.git

Note the use of --recursive above. More on this later.

Once the clone is complete the tree will look like this:

spark/
├── src
├── externals
├── docker
└── tools

src is where all source code used in the product lives. externals is a subrepository that contains all the external libraries needed by spark. docker holds the Dockerfiles used to define our build images. tools is used to keep simple tools used during development (not production). A quick rule of thumb is only what is in src is used to build the production package.

src has the following top level structure:

src/ 
├── client
├── lib
└── server

Each of these directories is dedicated to a specific purpose:

• client holds code that only runs on the client. Typically this is C# on Windows only.
• lib holds core multi-platform C++, things like config, logging, the API, the security model and quickbus. Additionally lib also holds the proto definitions (and resulting generated proxies and stubs) for all our API services. Because lib is compiled on both platforms it should only contain code that is needed on both client and server. This is important to reduce compilation times, avoid code bloat and minimize the ammount of external libraries for which we need to keep multi-platform builds. Going forward we expect for most if not all additions to lib to be proto definitions for API services. All the code in lib gets compiled together into a single library, spark_lib.ar on linux and spark_lib.dll on Windows.
• server holds code needed only on the server side. Things like jetfix, market data providers, trading algos, order managing code, etc. Currently all the code in server gets compiled into a single Linux only statically linked executable: sparkd

externals holds our C++ external libraries (we use NuGet for C# externals). In contrast with edge where we used as parallel repo to hold our externals, in spark we use a subrepo. This is why the --recursive flag is needed when cloning. Subrepos also called submodules allows us to tie each commit in spark to a specific commit in externals, freeing us from the burden of maintaining backwards compatibility every time we make any change to externals. This is a big improvement over our old externals but it does come with the cost of having to understand submodules. The basics are:

• Remember to use --recursive to get a full copy of a repo including its submodules
• When you need to change externals you need to first make your commit in externals, push that, then make a second commit in the parent repo to move the submodule pointer to the newest version of externals.

We build our C++ code using cmake. Though far from being an elegant tool cmake is a powerfull, multi-platform and mature build system, which is fast becoming the standard C++ build tool. Using cmake instead of our home grown build system, brings a number of advantages:

• Portability between Windows and Linux. With some care a single cmake project can be used to define builds for Linux and Windows.
• Tool support. cmake projects can be loaded by most tools, including Visual Studio, Visual Studio Code, CLion and emacs.
• Expressability. It is fairly easy to add new generators, switch between build types or add custom steps to the build using cmake.
• Simplicity. Our cmake projects files consist of only a few lines of custom code, in contrast to the thousands in our home grown system.

CMake is a meta-build system, that is we feed cmake a CMakeLists.txt file holding a high-level description of our C++ project and from it cmake generates build files that target the native build system for the specific platform: msbuild proj files on Windows and Makefiles on Linux. To bootstrap cmake we first create a build directory, that is a directory used to house the build output, then from that directory, we run cmake, pointing it to the CMakeLists.txt that defines our project. cmake will then generate the native build files. From that point on we can use the native build tools to actually build the software. For example, on Linux:

ppinto@pp-6_spark:~/Dropbox/bts/spark
$ mkdir build
ppinto@pp-6_spark:~/Dropbox/bts/spark
$ cd build
ppinto@pp-6_spark:~/Dropbox/bts/spark/build
$ cmake .. # point cmake to the directory that holds the main CMakeLists.txt
-- The C compiler identification is GNU 6.3.0
-- The CXX compiler identification is GNU 6.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ppinto/Dropbox/bts/spark/build
ppinto@pp-6_spark:~/Dropbox/bts/spark/build
$ make 
[  0%] Running C++ service compiler on api/services/UserService.proto

On Windows the process is similar but we have to instruct cmake to use the 64 bit compiler (it defaults to x86):

PS C:\Users\ppinto\Dropbox\bts\spark> mkdir build                                                                                                                                                                  
PS C:\Users\ppinto\Dropbox\bts\spark> cd build                                                                                                                                                                  
PS C:\Users\ppinto\Dropbox\bts\spark\build> cmake -G "Visual Studio 15 2017 Win64" ..                                                                                                                           
-- The C compiler identification is MSVC 19.11.25506.0                                                                                                                                                          
-- The CXX compiler identification is MSVC 19.11.25506.0                                                                                                                                                        
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.11.25503/bin/HostX86/x64/cl.exe                                                                
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.11.25503/bin/HostX86/x64/cl.exe -- works                                                       
-- Detecting C compiler ABI info                                                                                                                                                                                
-- Detecting C compiler ABI info - done                                                                                                                                                                         
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.11.25503/bin/HostX86/x64/cl.exe                                                              
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.11.25503/bin/HostX86/x64/cl.exe -- works                                                     
-- Detecting CXX compiler ABI info                                                                                                                                                                              
-- Detecting CXX compiler ABI info - done                                                                                                                                                                       
-- Detecting CXX compile features                                                                                                                                                                               
-- Detecting CXX compile features - done                                                                                                                                                                        
-- Configuring done                                                                                                                                                                                             
-- Generating done                                                                                                                                                                                              
-- Build files have been written to: C:/Users/ppinto/Dropbox/bts/spark/build                                                                                                                                    
PS C:\Users\ppinto\Dropbox\bts\spark\build> msbuild .\spark.sln /m /v:q /nologo

We currently use 3 CMakeLists.txt files to define our build:

• A top level CMakeLists.txt in the root of the spark repo. This defines the overall structure of the spark C++ projects.
• ./src/server/sparkd/CMakeLists.txt is a Linux only project that defines the sparkd daemon, that is the Linux server process that implements the spark backend.
• ./src/lib/CMakeLists.txt defines the spark_lib project holds multi-platform C++ code along with the proto definitions for our API services.

To build the system we use only the top level CMakeLists.txt at the root of the spark repo. The other CMakeLists.txt files are designed to be invoked from the top-level one.

To ensure that our builds are reproducible, that is identical bits are generated regardless of which machine is used to build, as well as encapsulate and name the steps required to bootstrap a build machine we use Docker. Docker allows us to define a light-weight virtual machine, called a container, using a Dockerfile, a simple text-file describing the steps required to configure the container. We have Dockerfiles for both Windows and Linux build machines.

To build from a Linux machine, doesn't matter what kind, you first need to install docker:

$ sudo apt-get install docker.io

You then should login to dockerhub (you might need to create an account there and ask Pete to add it to the BTS team):

$ docker login

You can then launch the build container using the ./tools/docker_shell.sh script:

ppinto@pp-6:~/Dropbox/bts/spark
$ ./tools/docker_shell.sh 
ppinto@pp-6_spark:~/Dropbox/bts/spark
$

Note the prompt change above (from pp-6 to pp-6_spark), indicating that you are now running inside the container. If this is the first time you try to build you will need to instruct cmake to generate the build Makefiles. There is a handy script for that, ./tools/setup-builds.sh:

ppinto@pp-6_spark:~/Dropbox/bts/spark
$ ./tools/setup_builds.sh

This sets up 4 builds one for each combination of compiler (clang or gcc) and configuration (debug or release). To build the gcc debug build you would just do:

ppinto@pp-6_spark:~/Dropbox/bts/spark
$ cd build/gcc_debug/
ppinto@pp-6_spark:~/Dropbox/bts/spark/build/gcc_debug
$ make -j $(nproc)

If you do not want to use a separate Linux machine to build (and run the spark server side) you can install Docker on Windows and run the ./tools/docker_shell.bat script (again, you will have to already have a docker hub login which you previously entered with docker login). This will launch the Linux container and mount your %HOMEDRIVE%%HOMEPATH% in the /root directory. Find your repo in there and proceed as in the Linux machine case.