* The Spark Git Repo ** General outline The code necessary to build spark, both ours and external can be obtain from the git repo at ~git@bitbucket.org:bluetradesys/spark.git~. To get started use: #+BEGIN_EXAMPLE git clone --recursive git@bitbucket.org:bluetradesys/spark.git #+END_EXAMPLE Note the use of ~--recursive~ above. More on this later. Once the clone is complete the tree will look like this: #+BEGIN_EXAMPLE spark/ ├── src ├── externals ├── docker └── tools #+END_EXAMPLE ~src~ is where all source code used in the product lives. ~externals~ is a subrepository that contains all the external libraries needed by spark. ~docker~ holds the Dockerfiles used to define our build images. ~tools~ is used to keep simple tools used during development (not production). A quick rule of thumb is only what is in ~src~ is used to build the production package. ** ~src~ ~src~ has the following top level structure: #+BEGIN_EXAMPLE src/ ├── client ├── lib └── server #+END_EXAMPLE Each of these directories is dedicated to a specific purpose: - ~client~ holds code that *only* runs on the client. Typically this is C# on Windows only. - ~lib~ holds core multi-platform C++, things like config, logging, the API, the security model and quickbus. Additionally ~lib~ also holds the proto definitions (and resulting generated proxies and stubs) for all our API services. Because ~lib~ is compiled on both platforms it should only contain code that is needed on both client and server. This is important to reduce compilation times, avoid code bloat and minimize the ammount of external libraries for which we need to keep multi-platform builds. Going forward we expect for most if not all additions to lib to be proto definitions for API services. All the code in ~lib~ gets compiled together into a single library, ~spark_lib.ar~ on linux and ~spark_lib.dll~ on Windows. - ~server~ holds code needed only on the server side. Things like jetfix, market data providers, trading algos, order managing code, etc. Currently all the code in server gets compiled into a single Linux only statically linked executable: ~sparkd~ ** ~externals~ ~externals~ holds our C++ external libraries (we use NuGet for C# externals). In contrast with edge where we used as parallel repo to hold our externals, in spark we use a subrepo. This is why the ~--recursive~ flag is needed when cloning. Subrepos also called submodules allows us to tie each commit in spark to a specific commit in externals, freeing us from the burden of maintaining backwards compatibility every time we make any change to externals. This is a big improvement over our old externals but it does come with the cost of having to understand submodules. The basics are: - Remember to use ~--recursive~ to get a full copy of a repo including its submodules - When you need to change externals you need to first make your commit in externals, push that, then make a second commit in the parent repo to move the submodule pointer to the newest version of externals. * The Build ** C++ and CMake We build our C++ code using ~cmake~. Though far from being an elegant tool ~cmake~ is a powerfull, multi-platform and mature build system, which is fast becoming the standard C++ build tool. Using ~cmake~ instead of our home grown build system, brings a number of advantages: - Portability between Windows and Linux. With some care a single ~cmake~ project can be used to define builds for Linux and Windows. - Tool support. ~cmake~ projects can be loaded by most tools, including Visual Studio, Visual Studio Code, CLion and emacs. - Expressability. It is fairly easy to add new generators, switch between build types or add custom steps to the build using cmake. - Simplicity. Our cmake projects files consist of only a few lines of custom code, in contrast to the thousands in our home grown system. ~CMake~ is a meta-build system, that is we feed ~cmake~ a ~CMakeLists.txt~ file holding a high-level description of our C++ project and from it ~cmake~ generates build files that target the native build system for the specific platform: msbuild proj files on Windows and Makefiles on Linux. To bootstrap ~cmake~ we first create a build directory, that is a directory used to house the build output, then from that directory, we run ~cmake~, pointing it to the ~CMakeLists.txt~ that defines our project. ~cmake~ will then generate the native build files. From that point on we can use the native build tools to actually build the software. For example, on Linux: #+BEGIN_EXAMPLE ppinto@pp-6_spark:~/Dropbox/bts/spark $ mkdir build ppinto@pp-6_spark:~/Dropbox/bts/spark $ cd build ppinto@pp-6_spark:~/Dropbox/bts/spark/build $ cmake .. # point cmake to the directory that holds the main CMakeLists.txt -- The C compiler identification is GNU 6.3.0 -- The CXX compiler identification is GNU 6.3.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Configuring done -- Generating done -- Build files have been written to: /home/ppinto/Dropbox/bts/spark/build ppinto@pp-6_spark:~/Dropbox/bts/spark/build $ make [ 0%] Running C++ service compiler on api/services/UserService.proto #+END_EXAMPLE On Windows the process is similar but we have to instruct ~cmake~ to use the 64 bit compiler (it defaults to x86): #+BEGIN_EXAMPLE PS C:\Users\ppinto\Dropbox\bts\spark> mkdir build PS C:\Users\ppinto\Dropbox\bts\spark> cd build PS C:\Users\ppinto\Dropbox\bts\spark\build> cmake -G "Visual Studio 15 2017 Win64" .. -- The C compiler identification is MSVC 19.11.25506.0 -- The CXX compiler identification is MSVC 19.11.25506.0 -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.11.25503/bin/HostX86/x64/cl.exe -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.11.25503/bin/HostX86/x64/cl.exe -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.11.25503/bin/HostX86/x64/cl.exe -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.11.25503/bin/HostX86/x64/cl.exe -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Configuring done -- Generating done -- Build files have been written to: C:/Users/ppinto/Dropbox/bts/spark/build PS C:\Users\ppinto\Dropbox\bts\spark\build> msbuild .\spark.sln /m /v:q /nologo #+END_EXAMPLE We currently use 3 ~CMakeLists.txt~ files to define our build: - A top level ~CMakeLists.txt~ in the root of the spark repo. This defines the overall structure of the spark C++ projects. - ~./src/server/sparkd/CMakeLists.txt~ is a Linux only project that defines the sparkd daemon, that is the Linux server process that implements the spark backend. - ~./src/lib/CMakeLists.txt~ defines the ~spark_lib~ project holds multi-platform C++ code along with the proto definitions for our API services. To build the system we use only the top level ~CMakeLists.txt~ at the root of the spark repo. The other ~CMakeLists.txt~ files are designed to be invoked from the top-level one. ** Docker To ensure that our builds are reproducible, that is identical bits are generated regardless of which machine is used to build, as well as encapsulate and name the steps required to bootstrap a build machine we use Docker. Docker allows us to define a light-weight virtual machine, called a container, using a Dockerfile, a simple text-file describing the steps required to configure the container. We have Dockerfiles for both Windows and Linux build machines. ** Using Docker from a Linux machine to build the server side To build from a Linux machine, doesn't matter what kind, you first need to install docker: #+BEGIN_EXAMPLE $ sudo apt-get install docker.io #+END_EXAMPLE You then should login to dockerhub (you might need to create an account there and ask Pete to add it to the BTS team): #+BEGIN_EXAMPLE $ docker login #+END_EXAMPLE You can then launch the build container using the ~./tools/docker_shell.sh~ script: #+BEGIN_EXAMPLE ppinto@pp-6:~/Dropbox/bts/spark $ ./tools/docker_shell.sh ppinto@pp-6_spark:~/Dropbox/bts/spark $ #+END_EXAMPLE Note the prompt change above (from pp-6 to pp-6_spark), indicating that you are now running inside the container. If this is the first time you try to build you will need to instruct ~cmake~ to generate the build Makefiles. There is a handy script for that, ~./tools/setup-builds.sh~: #+BEGIN_EXAMPLE ppinto@pp-6_spark:~/Dropbox/bts/spark $ ./tools/setup_builds.sh #+END_EXAMPLE This sets up 4 builds one for each combination of compiler (clang or gcc) and configuration (debug or release). To build the gcc debug build you would just do: #+BEGIN_EXAMPLE ppinto@pp-6_spark:~/Dropbox/bts/spark $ cd build/gcc_debug/ ppinto@pp-6_spark:~/Dropbox/bts/spark/build/gcc_debug $ make -j $(nproc) #+END_EXAMPLE ** Using Docker from a Windows machine to build the server side If you do not want to use a separate Linux machine to build (and run the spark server side) you can install Docker on Windows and run the ~./tools/docker_shell.bat~ script (again, you will have to already have a docker hub login which you previously entered with ~docker login~). This will launch the Linux container and mount your ~%HOMEDRIVE%%HOMEPATH%~ in the /root directory. Find your repo in there and proceed as in the Linux machine case. * Unit testing For Unit testing we use the [[https://github.com/philsquared/Catch][Catch]] library. There are several examples in the code base already. To invoke tests use: #+BEGIN_EXAMPLE $ sparkd test [test tags] #+END_EXAMPLE