Commit Graph
12065 Commits
Author SHA1 Message Date
John Colvin 1c2dc2c39c build: Configure admin webapp build entirely in CMakeLists
Get rid of build_admin_web_app.py
2024-09-18 14:02:39 +00:00
John Colvin 3faffbe22a refactor: Move domain types into new lib_domain
Also rename files in snake_case (standard naming convention for protos)
https://protobuf.dev/programming-guides/style/
2024-09-18 14:02:39 +00:00
John Colvin 7864c3238c refactor: Move collections into lib_system 2024-09-18 14:02:39 +00:00
John Colvin cf4a8599e7 refactor: Move a handful of types/utils to lib_system 2024-09-18 14:02:39 +00:00
John Colvin f1451b102d cleanup: Remove unused includes
Mostly collection macro headers
2024-09-18 14:02:39 +00:00
John Colvin 23b2758a05 cleanup (build): Move protobuf service generation cmake into lib_spark
Because lib_spark owns all the services
2024-09-18 14:02:39 +00:00
John Colvin 5c3a57ceb5 refactor: Rename protobuf out dir "include" -> "protobufs"
This is a more accurate and precise name. The dir contains c++ code generated
from .proto files - both sources (.cc) and headers (.pb.h).
2024-09-18 14:02:39 +00:00
John Colvin 888a6bc536 refactor: Move ValueWrapper into lib_system
So other projects can use it
2024-09-18 14:02:39 +00:00
John Colvin 0514d8f434 Include bts_assert.h in bts_fatal.h
We shouldn't have to include bts_assert.h when we're already including bts_fatal.h
2024-09-18 14:02:39 +00:00
John Colvin cf0adfd910 refactor: Split MarketDataGateway up into providers/sources
MarketDataGateway still exists, but is greatly simplified. It creates
several `IMarketDataProvider`s, each of which generates a list of
`IMarketDataSource`s. The market data sources provide market data to an
`IMarketDataListener`.

* EmumbaMarketDataProvider
* CCApiMarketDataProvider
* PyMarketDataProvider
* OnixsMarketDataProvider
* ReplayMarketDataProvider

As a side effect, individual py market data sources are now explicitly
limited to a single venue (it probably should have been this way all
along, since the shared memory writers do not expect updates to involve
multiple venues).
2024-09-18 14:02:39 +00:00
John Colvin d9c8c61e47 fix build: Remove old install target
Leftover from before recent build refactoring.
2024-09-18 14:02:39 +00:00
John Colvin f4cef5aadb cleanup: Remove superfluous code 2024-09-18 14:02:39 +00:00
John Colvin 6e5ddb734f fix: Always prepend proc name to shared mem prefix
Our shared memory model isn't really designed to be shared across processes.
Maybe some day it will.
2024-09-18 14:02:39 +00:00
John Colvin b4345cfa24 cleanup: Remove redundant macro def 2024-09-18 14:02:39 +00:00
John Colvin 54566ade60 Skip md recording thread loop if no recorders configured to run 2024-09-18 14:02:39 +00:00
John Colvin 06a80040ff Handle signal interrupts explicitly in sparkd
Otherwise the python interpretter, which is instantiated when we load
python md plugins, prevents the application from exiting.
2024-09-18 14:02:39 +00:00
John Colvin 2e9b4c78f3 feat: Add support for python md plugins 2024-09-18 14:02:39 +00:00
John Colvin 17ca432f87 refactor: Add lib_system proj for common utils
Migrates the following:
* Delegate.h
* bts_array_size.h
* string_utils.h

Note: File names in the new lib are snake_case (this seems to be the
standard naming convention for c++).
2024-09-18 14:02:39 +00:00
John Colvin e0ab5713b6 build: Clean up libspark CMakeLists 2024-09-18 14:02:39 +00:00
John Colvin c31b52e1df cleanup: Trivial whitespace cleanup in sparkd CMakeLists 2024-09-18 14:02:39 +00:00
John Colvin 3afc447538 cleanup: Use conventional UPPER_CASE var names in sparkd CMakeLists 2024-09-18 14:02:39 +00:00
John Colvin 5243490d5f build: Set up necessary paths, symlinks for sparkd in sparkd cmake proj
Rather than performing this step in the root CMakeLists, as we previously did
in the "server_layout" target. This effectively replaces the "server_layout" target.
2024-09-18 14:02:39 +00:00
John Colvin 0d04eea483 build: Clean up data_files cmake proj (support --target clean) 2024-09-18 14:02:39 +00:00
John Colvin 3abfac1a20 fix build: Generate proto hdrs in "include" dir (child of proj's bin dir)
Reorganizing the sparkd proj with stuttering dirs broke the build because
the generated protos were being dumped into a dir with the same name
as the "sparkd" executable.
2024-09-18 14:02:38 +00:00
John Colvin 574694e06e fix build: Update sparkd & risk webapp after moving libspark 2024-09-18 14:02:38 +00:00
John Colvin c30bba7d5c refactor: Parse input args in Protobuf_Generate_cpp
So we can be more precise when generating protobufs. For example, now we
can specify different include dirs when generating code for different
group of proto messages - an obvious improvement over the previous
system, which relied on a hardcoded superset of include dirs that would
support every proto message in our code base.
2024-09-18 14:02:38 +00:00
John Colvin 48695d3f7f refactor: Orgnaize libspark proj w/ stuttering include 2024-09-18 14:02:38 +00:00
John Colvin 3ef40387f3 build: Add install targets for libspark 2024-09-18 14:02:38 +00:00
John Colvin fcdd869831 fix build: Add ShmSegmentName.cpp to windows client sources 2024-09-18 14:02:38 +00:00
John Colvin 6a7efacac9 Make a few timestamp fns const
They should have been const in the first place
2024-09-18 14:02:38 +00:00
John Colvin 489b289a6d Add config profiles to libspark (check pt)
This is part of a larger effort to unify the libspark build, and eliminate
compile time differences between the builds used by our various services
2024-09-18 14:02:38 +00:00
John Colvin e75cc05137 Add api server to docker compose 2024-09-18 14:02:38 +00:00
John Colvin 89143b25c7 Change api server ports so they don't conflict w/ ds webapp 2024-09-18 14:02:38 +00:00
John Colvin 103725887f build: Resolve app data dir and shared mem prefix at runtime
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
2024-09-18 14:02:38 +00:00
John Colvin 1ce7ddd252 Don't log "initializing spark config" if it has already been initialized 2024-09-18 14:02:37 +00:00
John Colvin 5561c54345 Use dependency injection in spark api server 2024-09-18 14:02:37 +00:00
John Colvin c7269176ca Add SparkLoggerAdapter for Msft ILogger 2024-09-18 14:02:37 +00:00
John Colvin ec34e8b2f2 Display exception stack trace w/ 1 call per line in ds webapp 2024-09-18 14:02:37 +00:00
John Colvin a54f9530b9 Remove unused ErrorView from ds webapp 2024-09-18 14:02:37 +00:00
John Colvin 53850044c0 Propagate ViewStream errors to webapp frontend 2024-09-18 14:02:37 +00:00
John Colvin 8e91091e3c Add ErrorChanged event to propagate DataStream exceptions 2024-09-18 14:02:37 +00:00
John Colvin 92ffade5da Set columns to empty array rather than null in ds webapp
If no data has been received. Since the 'columns' property was marked
'required', this eliminates a warning that was being logged to console.
2024-09-18 14:02:37 +00:00
John Colvin b636cf76f2 Improve alignment/spacing in ds webapp 2024-09-18 14:02:37 +00:00
John Colvin ac001b65c6 Configure datastream deployment with docker-compose
To deploy to aws, connect to our aws vpn then run:
tools/deploy-docker-compose.sh

The deploy script does the following:
* builds sparkd
* builds datastream apps (pubsub, md-publisher, and webapp)
* uploads the build to aws
* bounces all services (sparkd, pubsub, md-publisher, and webapp)

Note: the name 'md-publisher' refers to DataStreamPublisher w/ MarketData
configuration
2024-09-18 14:02:37 +00:00
John Colvin bc3869f087 Remove "BTS Cloud Streams" title from webapp
Space is precious when viewing on phone (tbh, it didn't look that good
anyways).
2024-09-18 14:02:37 +00:00
John Colvin 66583356ac fix: Ensure md-publisher waits for pubsub and sparkd
Bugs (fixed):
  * md-publisher silently fails if it tries to connect before those services are "ready".

The Docker compose "depends_on" property ensures sparkd and pubsub
containers are up, but doesn't prevent startup race bugs (e.g., if
md-publisher and pubsub start around the same time, and md-publisher
tries to connect before pubsub has started accepting connections). To
address these race conditions, we use a nifty tool called "dockerize"

References:
  * https://github.com/jwilder/dockerize
2024-09-18 14:02:37 +00:00
John Colvin ef11c6e418 Replace ErrorView component w/ robust QueryStatusView 2024-09-18 14:02:37 +00:00
John Colvin 361e4bdd48 Fix typos in query button theme 2024-09-18 14:02:37 +00:00
John Colvin 44ad5861fc fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2024-09-18 14:02:37 +00:00
John Colvin 0ea6563ddb Change VITE_DATASTREAM_WEBSOCKET_URL to localhost 2024-09-18 14:02:37 +00:00