petefisher78/spark
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
be3daae1757fe9d08077ca94de1a6f4f8372ea40
spark/doc/dev/Spark_Handbook.html
T
Pedro Pinto 3060386a23 Doc change
2017-08-22 15:39:45 -04:00

506 lines
21 KiB
HTML
Raw Blame History

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Spark_Handbook</title>
<!-- 2017-08-22 Tue 15:38 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<meta name="author" content="ppinto" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<h1 class="title">Spark_Handbook</h1>
<div id="table-of-contents">
<h2>Table of Contents</h2>
<div id="text-table-of-contents">
<ul>
<li><a href="#sec-1">1. The Spark Git Repo</a>
<ul>
<li><a href="#sec-1-1">1.1. General outline</a></li>
<li><a href="#sec-1-2">1.2. <code>src</code></a></li>
<li><a href="#sec-1-3">1.3. <code>externals</code></a></li>
</ul>
</li>
<li><a href="#sec-2">2. The Build</a>
<ul>
<li><a href="#sec-2-1">2.1. C++ and CMake</a></li>
<li><a href="#sec-2-2">2.2. Docker</a></li>
<li><a href="#sec-2-3">2.3. Using Docker from a Linux machine to build the server side</a></li>
<li><a href="#sec-2-4">2.4. Using Docker from a Windows machine to build the server side</a></li>
</ul>
</li>
<li><a href="#sec-3">3. Unit testing</a></li>
</ul>
</div>
</div>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1"><span class="section-number-2">1</span> The Spark Git Repo</h2>
<div class="outline-text-2" id="text-1">
</div><div id="outline-container-sec-1-1" class="outline-3">
<h3 id="sec-1-1"><span class="section-number-3">1.1</span> General outline</h3>
<div class="outline-text-3" id="text-1-1">
<p>
The code necessary to build spark, both ours and external can be
obtain from the git repo at
<code>git@bitbucket.org:bluetradesys/spark.git</code>. To get started use:
</p>
<pre class="example">
git clone --recursive git@bitbucket.org:bluetradesys/spark.git
</pre>
<p>
Note the use of <code>--recursive</code> above. More on this later.
</p>
<p>
Once the clone is complete the tree will look like this:
</p>
<pre class="example">
spark/
├── src
├── externals
├── docker
└── tools
</pre>
<p>
<code>src</code> is where all source code used in the product lives. <code>externals</code>
is a subrepository that contains all the external libraries needed by
spark. <code>docker</code> holds the Dockerfiles used to define our build
images. <code>tools</code> is used to keep simple tools used during development
(not production). A quick rule of thumb is only what is in <code>src</code> is
used to build the production package.
</p>
</div>
</div>
<div id="outline-container-sec-1-2" class="outline-3">
<h3 id="sec-1-2"><span class="section-number-3">1.2</span> <code>src</code></h3>
<div class="outline-text-3" id="text-1-2">
<p>
<code>src</code> has the following top level structure:
</p>
<pre class="example">
src/
├── client
├── lib
└── server
</pre>
<p>
Each of these directories is dedicated to a specific purpose:
</p>
<ul class="org-ul">
<li><code>client</code> holds code that <b>only</b> runs on the client. Typically this
is C# on Windows only.
</li>
<li><code>lib</code> holds core multi-platform C++, things like config, logging,
the API, the security model and quickbus. Additionally <code>lib</code> also
holds the proto definitions (and resulting generated proxies and
stubs) for all our API services. Because <code>lib</code> is compiled on both
platforms it should only contain code that is needed on both client
and server. This is important to reduce compilation times, avoid
code bloat and minimize the ammount of external libraries for which
we need to keep multi-platform builds. Going forward we expect for
most if not all additions to lib to be proto definitions for API
services. All the code in <code>lib</code> gets compiled together into a single
library, <code>spark_lib.ar</code> on linux and <code>spark_lib.dll</code> on Windows.
</li>
<li><code>server</code> holds code needed only on the server side. Things like
jetfix, market data providers, trading algos, order managing code,
etc. Currently all the code in server gets compiled into a single
Linux only statically linked executable: <code>sparkd</code>
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-1-3" class="outline-3">
<h3 id="sec-1-3"><span class="section-number-3">1.3</span> <code>externals</code></h3>
<div class="outline-text-3" id="text-1-3">
<p>
<code>externals</code> holds our C++ external libraries (we use NuGet for C#
externals). In contrast with edge where we used as parallel repo to
hold our externals, in spark we use a subrepo. This is why the
<code>--recursive</code> flag is needed when cloning. Subrepos also called
submodules allows us to tie each commit in spark to a specific commit
in externals, freeing us from the burden of maintaining backwards
compatibility every time we make any change to externals. This is a
big improvement over our old externals but it does come with the cost
of having to understand submodules. The basics are:
</p>
<ul class="org-ul">
<li>Remember to use <code>--recursive</code> to get a full copy of a repo including
its submodules
</li>
<li>When you need to change externals you need to first make your commit
in externals, push that, then make a second commit in the parent
repo to move the submodule pointer to the newest version of externals.
</li>
</ul>
</div>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2"><span class="section-number-2">2</span> The Build</h2>
<div class="outline-text-2" id="text-2">
</div><div id="outline-container-sec-2-1" class="outline-3">
<h3 id="sec-2-1"><span class="section-number-3">2.1</span> C++ and CMake</h3>
<div class="outline-text-3" id="text-2-1">
<p>
We build our C++ code using <code>cmake</code>. Though far from being an elegant
tool <code>cmake</code> is a powerfull, multi-platform and mature build system,
which is fast becoming the standard C++ build tool. Using <code>cmake</code>
instead of our home grown build system, brings a number of advantages:
</p>
<ul class="org-ul">
<li>Portability between Windows and Linux. With some care a single
<code>cmake</code> project can be used to define builds for Linux and Windows.
</li>
<li>Tool support. <code>cmake</code> projects can be loaded by most tools,
including Visual Studio, Visual Studio Code, CLion and emacs.
</li>
<li>Expressability. It is fairly easy to add new generators, switch
between build types or add custom steps to the build using cmake.
</li>
<li>Simplicity. Our cmake projects files consist of only a few lines of
custom code, in contrast to the thousands in our home grown system.
</li>
</ul>
<p>
<code>CMake</code> is a meta-build system, that is we feed <code>cmake</code> a
<code>CMakeLists.txt</code> file holding a high-level description of our C++
project and from it <code>cmake</code> generates build files that target the
native build system for the specific platform: msbuild proj files on
Windows and Makefiles on Linux. To bootstrap <code>cmake</code> we first create a
build directory, that is a directory used to house the build output,
then from that directory, we run <code>cmake</code>, pointing it to the
<code>CMakeLists.txt</code> that defines our project. <code>cmake</code> will then generate
the native build files. From that point on we can use the native build
tools to actually build the software. For example, on Linux:
</p>
<pre class="example">
ppinto@pp-6_spark:~/Dropbox/bts/spark
$ mkdir build
ppinto@pp-6_spark:~/Dropbox/bts/spark
$ cd build
ppinto@pp-6_spark:~/Dropbox/bts/spark/build
$ cmake .. # point cmake to the directory that holds the main CMakeLists.txt
-- The C compiler identification is GNU 6.3.0
-- The CXX compiler identification is GNU 6.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ppinto/Dropbox/bts/spark/build
ppinto@pp-6_spark:~/Dropbox/bts/spark/build
$ make
[ 0%] Running C++ service compiler on api/services/UserService.proto
</pre>
<p>
On Windows the process is similar but we have to instruct <code>cmake</code> to use
the 64 bit compiler (it defaults to x86):
</p>
<pre class="example">
PS C:\Users\ppinto\Dropbox\bts\spark&gt; mkdir build
PS C:\Users\ppinto\Dropbox\bts\spark&gt; cd build
PS C:\Users\ppinto\Dropbox\bts\spark\build&gt; cmake -G "Visual Studio 15 2017 Win64" ..
-- The C compiler identification is MSVC 19.11.25506.0
-- The CXX compiler identification is MSVC 19.11.25506.0
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.11.25503/bin/HostX86/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.11.25503/bin/HostX86/x64/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.11.25503/bin/HostX86/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.11.25503/bin/HostX86/x64/cl.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/ppinto/Dropbox/bts/spark/build
PS C:\Users\ppinto\Dropbox\bts\spark\build&gt; msbuild .\spark.sln /m /v:q /nologo
</pre>
<p>
We currently use 3 <code>CMakeLists.txt</code> files to define our build:
</p>
<ul class="org-ul">
<li>A top level <code>CMakeLists.txt</code> in the root of the spark repo. This
defines the overall structure of the spark C++ projects.
</li>
<li><code>./src/server/sparkd/CMakeLists.txt</code> is a Linux only project that
defines the sparkd daemon, that is the Linux server process that
implements the spark backend.
</li>
<li><code>./src/lib/CMakeLists.txt</code> defines the <code>spark_lib</code> project holds
multi-platform C++ code along with the proto definitions for our API
services.
</li>
</ul>
<p>
To build the system we use only the top level <code>CMakeLists.txt</code> at the
root of the spark repo. The other <code>CMakeLists.txt</code> files are designed
to be invoked from the top-level one.
</p>
</div>
</div>
<div id="outline-container-sec-2-2" class="outline-3">
<h3 id="sec-2-2"><span class="section-number-3">2.2</span> Docker</h3>
<div class="outline-text-3" id="text-2-2">
<p>
To ensure that our builds are reproducible, that is identical bits are
generated regardless of which machine is used to build, as well as
encapsulate and name the steps required to bootstrap a build machine
we use Docker. Docker allows us to define a light-weight virtual
machine, called a container, using a Dockerfile, a simple text-file
describing the steps required to configure the container. We have
Dockerfiles for both Windows and Linux build machines.
</p>
</div>
</div>
<div id="outline-container-sec-2-3" class="outline-3">
<h3 id="sec-2-3"><span class="section-number-3">2.3</span> Using Docker from a Linux machine to build the server side</h3>
<div class="outline-text-3" id="text-2-3">
<p>
To build from a Linux machine, doesn't matter what kind, you first
need to install docker:
</p>
<pre class="example">
$ sudo apt-get install docker.io
</pre>
<p>
You then should login to dockerhub (you might need to create an
account there and ask Pete to add it to the BTS team):
</p>
<pre class="example">
$ docker login
</pre>
<p>
You can then launch the build container using the
<code>./tools/docker_shell.sh</code> script:
</p>
<pre class="example">
ppinto@pp-6:~/Dropbox/bts/spark
$ ./tools/docker_shell.sh
ppinto@pp-6_spark:~/Dropbox/bts/spark
$
</pre>
<p>
Note the prompt change above (from pp-6 to pp-6<sub>spark</sub>), indicating
that you are now running inside the container. If this is the first
time you try to build you will need to instruct <code>cmake</code> to generate
the build Makefiles. There is a handy script for that, <code>./tools/setup-builds.sh</code>:
</p>
<pre class="example">
ppinto@pp-6_spark:~/Dropbox/bts/spark
$ ./tools/setup_builds.sh
</pre>
<p>
This sets up 4 builds one for each combination of compiler (clang or
gcc) and configuration (debug or release). To build the gcc debug
build you would just do:
</p>
<pre class="example">
ppinto@pp-6_spark:~/Dropbox/bts/spark
$ cd build/gcc_debug/
ppinto@pp-6_spark:~/Dropbox/bts/spark/build/gcc_debug
$ make -j $(nproc)
</pre>
</div>
</div>
<div id="outline-container-sec-2-4" class="outline-3">
<h3 id="sec-2-4"><span class="section-number-3">2.4</span> Using Docker from a Windows machine to build the server side</h3>
<div class="outline-text-3" id="text-2-4">
<p>
If you do not want to use a separate Linux machine to build (and run
the spark server side) you can install Docker on Windows and run the
<code>./tools/docker_shell.bat</code> script (again, you will have to already
have a docker hub login which you previously entered with <code>docker
login</code>). This will launch the Linux container and mount your
<code>%HOMEDRIVE%%HOMEPATH%</code> in the /root directory. Find your repo in
there and proceed as in the Linux machine case.
</p>
</div>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3"><span class="section-number-2">3</span> Unit testing</h2>
<div class="outline-text-2" id="text-3">
<p>
For Unit testing we use the <a href="https://github.com/philsquared/Catch">Catch</a> library. There are several examples
in the code base already. To invoke tests use:
</p>
<pre class="example">
$ sparkd test [test tags]
</pre>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="author">Author: ppinto</p>
<p class="date">Created: 2017-08-22 Tue 15:38</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 25.1.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>
Reference in new issue View Git Blame Copy Permalink