MarketDataGateway still exists, but is greatly simplified. It creates
several `IMarketDataProvider`s, each of which generates a list of
`IMarketDataSource`s. The market data sources provide market data to an
`IMarketDataListener`.
* EmumbaMarketDataProvider
* CCApiMarketDataProvider
* PyMarketDataProvider
* OnixsMarketDataProvider
* ReplayMarketDataProvider
As a side effect, individual py market data sources are now explicitly
limited to a single venue (it probably should have been this way all
along, since the shared memory writers do not expect updates to involve
multiple venues).
Migrates the following:
* Delegate.h
* bts_array_size.h
* string_utils.h
Note: File names in the new lib are snake_case (this seems to be the
standard naming convention for c++).
Rather than performing this step in the root CMakeLists, as we previously did
in the "server_layout" target. This effectively replaces the "server_layout" target.
Reorganizing the sparkd proj with stuttering dirs broke the build because
the generated protos were being dumped into a dir with the same name
as the "sparkd" executable.
So we can be more precise when generating protobufs. For example, now we
can specify different include dirs when generating code for different
group of proto messages - an obvious improvement over the previous
system, which relied on a hardcoded superset of include dirs that would
support every proto message in our code base.
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
To deploy to aws, connect to our aws vpn then run:
tools/deploy-docker-compose.sh
The deploy script does the following:
* builds sparkd
* builds datastream apps (pubsub, md-publisher, and webapp)
* uploads the build to aws
* bounces all services (sparkd, pubsub, md-publisher, and webapp)
Note: the name 'md-publisher' refers to DataStreamPublisher w/ MarketData
configuration
Bugs (fixed):
* md-publisher silently fails if it tries to connect before those services are "ready".
The Docker compose "depends_on" property ensures sparkd and pubsub
containers are up, but doesn't prevent startup race bugs (e.g., if
md-publisher and pubsub start around the same time, and md-publisher
tries to connect before pubsub has started accepting connections). To
address these race conditions, we use a nifty tool called "dockerize"
References:
* https://github.com/jwilder/dockerize