Commit Graph
13630 Commits
Author SHA1 Message Date
John Colvin d41fa8a5fb cleanup: Remove unused includes
Mostly collection macro headers
2024-10-17 05:29:46 +00:00
John Colvin 1da52e9a5a cleanup (build): Move protobuf service generation cmake into lib_spark
Because lib_spark owns all the services
2024-10-17 05:29:46 +00:00
John Colvin 53739e3919 refactor: Rename protobuf out dir "include" -> "protobufs"
This is a more accurate and precise name. The dir contains c++ code generated
from .proto files - both sources (.cc) and headers (.pb.h).
2024-10-17 05:29:45 +00:00
John Colvin 18ea0ec425 refactor: Move ValueWrapper into lib_system
So other projects can use it
2024-10-17 05:29:45 +00:00
John Colvin 0b3a3d8833 Include bts_assert.h in bts_fatal.h
We shouldn't have to include bts_assert.h when we're already including bts_fatal.h
2024-10-17 05:29:45 +00:00
John Colvin b8d35cc0f9 refactor: Split MarketDataGateway up into providers/sources
MarketDataGateway still exists, but is greatly simplified. It creates
several `IMarketDataProvider`s, each of which generates a list of
`IMarketDataSource`s. The market data sources provide market data to an
`IMarketDataListener`.

* EmumbaMarketDataProvider
* CCApiMarketDataProvider
* PyMarketDataProvider
* OnixsMarketDataProvider
* ReplayMarketDataProvider

As a side effect, individual py market data sources are now explicitly
limited to a single venue (it probably should have been this way all
along, since the shared memory writers do not expect updates to involve
multiple venues).
2024-10-17 05:29:45 +00:00
John Colvin eb81da9863 cleanup: Remove superfluous code 2024-10-17 05:29:45 +00:00
John Colvin 9ea79c0bc9 improvement: Set SHARED_MEMORY_PREFIX to an environment variable
So we can effectively define it at runtime. We don't want to have to
produce different builds just to change the shared mem prefix.
2024-10-17 05:29:45 +00:00
John Colvin 621cdb90a4 fix: Always prepend proc name to shared mem prefix
Our shared memory model isn't really designed to be shared across processes.
Maybe some day it will.
2024-10-17 05:29:45 +00:00
John Colvin 3ded2653ad cleanup: Remove redundant macro def 2024-10-17 05:29:45 +00:00
John Colvin e92a91b450 Skip md recording thread loop if no recorders configured to run 2024-10-17 05:29:45 +00:00
John Colvin 9d66f268ec Handle signal interrupts explicitly in sparkd
Otherwise the python interpretter, which is instantiated when we load
python md plugins, prevents the application from exiting.
2024-10-17 05:29:45 +00:00
John Colvin a81f176de6 feat: Add support for python md plugins 2024-10-17 05:29:45 +00:00
John Colvin 3ff3f62be3 fix build: Ensure py scripts use unix line endings
I'm not sure why, but my sparkd build (linux) suddenly began failing to
generate cpp headers for protos. The error message indicated it was due
to an unexpected carriage return (\r) in plugin.py:

/usr/bin/env: ‘python\r’: No such file or directory
2024-10-17 05:24:19 +00:00
John Colvin 09b28704e6 refactor: Add lib_system proj for common utils
Migrates the following:
* Delegate.h
* bts_array_size.h
* string_utils.h

Note: File names in the new lib are snake_case (this seems to be the
standard naming convention for c++).
2024-10-17 05:24:19 +00:00
John Colvin 586fdefc45 cleanup: Trivial whitespace cleanup in sparkd CMakeLists 2024-10-16 18:35:26 +00:00
John Colvin 99f13b5ba5 build: Add cmake fns to symlink and install local libs
The code needed to do this (inline) is ugly and verbose
2024-10-16 18:35:26 +00:00
John Colvin 5c9624380e cleanup: Use conventional UPPER_CASE var names in sparkd CMakeLists 2024-10-16 18:35:26 +00:00
John Colvin ca6429e483 cleanup: Trivial whitespace cleanup in Protobuf.cmake 2024-10-16 18:35:26 +00:00
John Colvin 9f1e4d8b3e build: Clean up data_files cmake proj (support --target clean) 2024-10-16 18:35:16 +00:00
John Colvin 7e79430aeb refactor: Organize libspark proj w/ stuttering include
This new structure provides several benefits:

* Allows us to install multiple CMake projects in the same destination
  dir (without include path conflicts).
  - This makes shipping the trading algo sdk a trivial excercise. Simply
    copy the lib and include dirs from the install dest. That's it!
* Puts us in a position to break up the monolithic libspark into smaller
  modules, which are easier to manage.
2024-10-16 18:35:15 +00:00
John Colvin 031b0ac521 vscode: Add debug cfg to skip sparkd build 2024-10-14 17:21:49 +00:00
John Colvin 9ec5bf4ea5 fix build: Add ShmSegmentName.cpp to windows client sources 2024-10-14 17:21:49 +00:00
John Colvin 5507743201 Make a few timestamp fns const
They should have been const in the first place
2024-10-14 17:21:49 +00:00
John Colvin e0e4bf9764 Add config profiles to libspark
This is part of a larger effort to unify the libspark build, and eliminate
compile time differences between the builds used by our various services
2024-10-14 17:21:49 +00:00
John Colvin 95ba1bb0bd Add api server to docker compose 2024-10-14 17:21:49 +00:00
John Colvin a524943f47 Rename pingtest "sparkd-ping-test", make it depend on sparkd
Add it to a "ping-test" profile, as we may add similar tests for other services
2024-10-14 17:21:49 +00:00
John Colvin 19804866ad cleanup: Remove unnecessary env vars from webapp docker compose 2024-10-14 17:21:49 +00:00
John Colvin ef5835a83c Change api server ports so they don't conflict w/ ds webapp 2024-10-14 17:21:49 +00:00
John Colvin 86b9c67c54 vscode: Fix sparkd debug config 2024-10-14 17:21:49 +00:00
John Colvin 0a6d960522 build: Resolve app data dir and shared mem prefix at runtime
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
2024-10-14 17:21:49 +00:00
John Colvin eb8be7a98f Don't log "initializing spark config" if it has already been initialized 2024-10-14 17:21:49 +00:00
John Colvin 8c05bcf4f7 vscode: Add debug cfg for spark api server 2024-10-14 17:21:49 +00:00
John Colvin b788505646 Use dependency injection in spark api server 2024-10-14 17:21:49 +00:00
John Colvin fcf5707ef1 Add SparkLoggerAdapter for Msft ILogger 2024-10-14 17:21:49 +00:00
John Colvin 38dcfeae79 Display exception stack trace w/ 1 call per line in ds webapp 2024-10-14 17:21:49 +00:00
John Colvin be4556a6b2 Remove unused ErrorView from ds webapp 2024-10-14 17:21:49 +00:00
John Colvin 939b783583 Propagate ViewStream errors to webapp frontend 2024-10-14 17:21:49 +00:00
John Colvin 466d3eb432 Add ErrorChanged event to propagate DataStream exceptions 2024-10-14 17:21:49 +00:00
John Colvin d3f069439d Set columns to empty array rather than null in ds webapp
If no data has been received. Since the 'columns' property was marked
'required', this eliminates a warning that was being logged to console.
2024-10-14 17:21:49 +00:00
John Colvin bc9102b7b5 Improve alignment/spacing in ds webapp 2024-10-14 17:21:49 +00:00
John Colvin b0bc68997b Configure datastream deployment with docker-compose
To deploy to aws, connect to our aws vpn then run:
tools/deploy-docker-compose.sh

The deploy script does the following:
* builds sparkd
* builds datastream apps (pubsub, md-publisher, and webapp)
* uploads the build to aws
* bounces all services (sparkd, pubsub, md-publisher, and webapp)

Note: the name 'md-publisher' refers to DataStreamPublisher w/ MarketData
configuration
2024-10-14 17:21:49 +00:00
John Colvin ad1e9b1fe2 Remove "BTS Cloud Streams" title from webapp
Space is precious when viewing on phone (tbh, it didn't look that good
anyways).
2024-10-14 17:21:48 +00:00
John Colvin 1efbca9a49 fix: Ensure md-publisher waits for pubsub and sparkd
Bugs (fixed):
  * md-publisher silently fails if it tries to connect before those services are "ready".

The Docker compose "depends_on" property ensures sparkd and pubsub
containers are up, but doesn't prevent startup race bugs (e.g., if
md-publisher and pubsub start around the same time, and md-publisher
tries to connect before pubsub has started accepting connections). To
address these race conditions, we use a nifty tool called "dockerize"

References:
  * https://github.com/jwilder/dockerize
2024-10-14 17:21:47 +00:00
John Colvin 00f825802d Replace ErrorView component w/ robust QueryStatusView 2024-10-14 17:21:47 +00:00
John Colvin 4dc95e1979 Fix typos in query button theme 2024-10-14 17:21:47 +00:00
John Colvin fd0f95a884 fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2024-10-14 17:21:47 +00:00
John Colvin 00f866d3f5 Change VITE_DATASTREAM_WEBSOCKET_URL to localhost 2024-10-14 17:21:47 +00:00
John Colvin 17b2d56391 Assert that local ip address is successfully obtained 2024-10-14 17:21:47 +00:00
John Colvin 0d9699b79c improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't been
tested yet...so we should assume it doesn't work).
2024-10-14 17:21:47 +00:00