Commit Graph
13596 Commits
Author SHA1 Message Date
John Colvin 2e9b4c78f3 feat: Add support for python md plugins 2024-09-18 14:02:39 +00:00
John Colvin 2b62668df8 fix build: Ensure py scripts use unix line endings
I'm not sure why, but my sparkd build (linux) suddenly began failing to
generate cpp headers for protos. The error message indicated it was due
to an unexpected carriage return (\r) in plugin.py:

/usr/bin/env: ‘python\r’: No such file or directory
2024-09-18 14:02:39 +00:00
John Colvin 17ca432f87 refactor: Add lib_system proj for common utils
Migrates the following:
* Delegate.h
* bts_array_size.h
* string_utils.h

Note: File names in the new lib are snake_case (this seems to be the
standard naming convention for c++).
2024-09-18 14:02:39 +00:00
John Colvin 2c8833bdfe vscode: Fix sparkd path in launch.json 2024-09-18 14:02:39 +00:00
John Colvin e0ab5713b6 build: Clean up libspark CMakeLists 2024-09-18 14:02:39 +00:00
John Colvin c31b52e1df cleanup: Trivial whitespace cleanup in sparkd CMakeLists 2024-09-18 14:02:39 +00:00
John Colvin 2e7478629f build: Add cmake fns to symlink and install local libs
The code needed to do this (inline) is ugly and verbose
2024-09-18 14:02:39 +00:00
John Colvin 3afc447538 cleanup: Use conventional UPPER_CASE var names in sparkd CMakeLists 2024-09-18 14:02:39 +00:00
John Colvin 56887327e1 cleanup: Trivial whitespace cleanup in Protobuf.cmake 2024-09-18 14:02:39 +00:00
John Colvin 6c89cd6717 build: Remove explicit targets from sparkd build preset
So it builds the default targets (all). They were only listed explicitly
in the first place so we could build specific targets in vs code using
the dropdown
2024-09-18 14:02:39 +00:00
John Colvin 5243490d5f build: Set up necessary paths, symlinks for sparkd in sparkd cmake proj
Rather than performing this step in the root CMakeLists, as we previously did
in the "server_layout" target. This effectively replaces the "server_layout" target.
2024-09-18 14:02:39 +00:00
John Colvin 0d04eea483 build: Clean up data_files cmake proj (support --target clean) 2024-09-18 14:02:39 +00:00
John Colvin 3abfac1a20 fix build: Generate proto hdrs in "include" dir (child of proj's bin dir)
Reorganizing the sparkd proj with stuttering dirs broke the build because
the generated protos were being dumped into a dir with the same name
as the "sparkd" executable.
2024-09-18 14:02:38 +00:00
John Colvin 574694e06e fix build: Update sparkd & risk webapp after moving libspark 2024-09-18 14:02:38 +00:00
John Colvin c30bba7d5c refactor: Parse input args in Protobuf_Generate_cpp
So we can be more precise when generating protobufs. For example, now we
can specify different include dirs when generating code for different
group of proto messages - an obvious improvement over the previous
system, which relied on a hardcoded superset of include dirs that would
support every proto message in our code base.
2024-09-18 14:02:38 +00:00
John Colvin 48695d3f7f refactor: Orgnaize libspark proj w/ stuttering include 2024-09-18 14:02:38 +00:00
John Colvin 3ef40387f3 build: Add install targets for libspark 2024-09-18 14:02:38 +00:00
John Colvin 776625891e vscode: Add debug cfg to skip sparkd build 2024-09-18 14:02:38 +00:00
John Colvin fcdd869831 fix build: Add ShmSegmentName.cpp to windows client sources 2024-09-18 14:02:38 +00:00
John Colvin 6a7efacac9 Make a few timestamp fns const
They should have been const in the first place
2024-09-18 14:02:38 +00:00
John Colvin 489b289a6d Add config profiles to libspark (check pt)
This is part of a larger effort to unify the libspark build, and eliminate
compile time differences between the builds used by our various services
2024-09-18 14:02:38 +00:00
John Colvin e75cc05137 Add api server to docker compose 2024-09-18 14:02:38 +00:00
John Colvin f75fe665c1 Rename pingtest "sparkd-ping-test", make it depend on sparkd
Add it to a "ping-test" profile, as we may add similar tests for other services
2024-09-18 14:02:38 +00:00
John Colvin 58bd35be01 cleanup: Remove unnecessary env vars from webapp docker compose 2024-09-18 14:02:38 +00:00
John Colvin 89143b25c7 Change api server ports so they don't conflict w/ ds webapp 2024-09-18 14:02:38 +00:00
John Colvin aec005505f vscode: Fix sparkd debug config 2024-09-18 14:02:38 +00:00
John Colvin 103725887f build: Resolve app data dir and shared mem prefix at runtime
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
2024-09-18 14:02:38 +00:00
John Colvin 1ce7ddd252 Don't log "initializing spark config" if it has already been initialized 2024-09-18 14:02:37 +00:00
John Colvin 3e1000e8e7 vscode: Add debug cfg for spark api server 2024-09-18 14:02:37 +00:00
John Colvin 5561c54345 Use dependency injection in spark api server 2024-09-18 14:02:37 +00:00
John Colvin c7269176ca Add SparkLoggerAdapter for Msft ILogger 2024-09-18 14:02:37 +00:00
John Colvin ec34e8b2f2 Display exception stack trace w/ 1 call per line in ds webapp 2024-09-18 14:02:37 +00:00
John Colvin a54f9530b9 Remove unused ErrorView from ds webapp 2024-09-18 14:02:37 +00:00
John Colvin 53850044c0 Propagate ViewStream errors to webapp frontend 2024-09-18 14:02:37 +00:00
John Colvin 8e91091e3c Add ErrorChanged event to propagate DataStream exceptions 2024-09-18 14:02:37 +00:00
John Colvin 92ffade5da Set columns to empty array rather than null in ds webapp
If no data has been received. Since the 'columns' property was marked
'required', this eliminates a warning that was being logged to console.
2024-09-18 14:02:37 +00:00
John Colvin b636cf76f2 Improve alignment/spacing in ds webapp 2024-09-18 14:02:37 +00:00
John Colvin ac001b65c6 Configure datastream deployment with docker-compose
To deploy to aws, connect to our aws vpn then run:
tools/deploy-docker-compose.sh

The deploy script does the following:
* builds sparkd
* builds datastream apps (pubsub, md-publisher, and webapp)
* uploads the build to aws
* bounces all services (sparkd, pubsub, md-publisher, and webapp)

Note: the name 'md-publisher' refers to DataStreamPublisher w/ MarketData
configuration
2024-09-18 14:02:37 +00:00
John Colvin bc3869f087 Remove "BTS Cloud Streams" title from webapp
Space is precious when viewing on phone (tbh, it didn't look that good
anyways).
2024-09-18 14:02:37 +00:00
John Colvin 66583356ac fix: Ensure md-publisher waits for pubsub and sparkd
Bugs (fixed):
  * md-publisher silently fails if it tries to connect before those services are "ready".

The Docker compose "depends_on" property ensures sparkd and pubsub
containers are up, but doesn't prevent startup race bugs (e.g., if
md-publisher and pubsub start around the same time, and md-publisher
tries to connect before pubsub has started accepting connections). To
address these race conditions, we use a nifty tool called "dockerize"

References:
  * https://github.com/jwilder/dockerize
2024-09-18 14:02:37 +00:00
John Colvin ef11c6e418 Replace ErrorView component w/ robust QueryStatusView 2024-09-18 14:02:37 +00:00
John Colvin 361e4bdd48 Fix typos in query button theme 2024-09-18 14:02:37 +00:00
John Colvin 44ad5861fc fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2024-09-18 14:02:37 +00:00
John Colvin 0ea6563ddb Change VITE_DATASTREAM_WEBSOCKET_URL to localhost 2024-09-18 14:02:37 +00:00
John Colvin ff7f8fce49 Assert that local ip address is successfully obtained 2024-09-18 14:02:37 +00:00
John Colvin cc28ba41d1 improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't been
tested yet...so we should assume it doesn't work).
2024-09-18 14:02:37 +00:00
John Colvin 6fcec2c370 Assert TcpConnection endpoint matches expected 2024-09-18 14:02:37 +00:00
John Colvin 76393216c1 Fix typo in log msg (was missing interpolation '$') 2024-09-18 14:02:37 +00:00
John Colvin 3e792a41e0 Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2024-09-18 14:02:37 +00:00
John Colvin 298273de91 fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2024-09-18 14:02:37 +00:00