Commit Graph
13511 Commits
Author SHA1 Message Date
John Colvin 7da3da74a9 build: Remove explicit targets from sparkd build preset
So it builds the default targets (all). They were only listed explicitly
in the first place so we could build specific targets in vs code using
the dropdown
2024-08-23 16:29:04 +00:00
John Colvin 9d56826797 build: Set up necessary paths, symlinks for sparkd in sparkd cmake proj
Rather than performing this step in the root CMakeLists, as we previously did
in the "server_layout" target. This effectively replaces the "server_layout" target.
2024-08-23 16:29:04 +00:00
John Colvin 92159c01ba build: Clean up data_files cmake proj (support --target clean) 2024-08-23 16:22:42 +00:00
John Colvin 8d260a979c fix build: Generate proto hdrs in "include" dir (child of proj's bin dir)
Reorganizing the sparkd proj with stuttering dirs broke the build because
the generated protos were being dumped into a dir with the same name
as the "sparkd" executable.
2024-08-23 16:22:42 +00:00
John Colvin 89305eb7cf fix build: Update sparkd & risk webapp after moving libspark 2024-08-23 16:22:42 +00:00
John Colvin bb1e113893 refactor: Parse input args in Protobuf_Generate_cpp
So we can be more precise when generating protobufs. For example, now we
can specify different include dirs when generating code for different
group of proto messages - an obvious improvement over the previous
system, which relied on a hardcoded superset of include dirs that would
support every proto message in our code base.
2024-08-23 16:22:42 +00:00
John Colvin 28f27deb2e refactor: Orgnaize libspark proj w/ stuttering include 2024-08-23 16:22:41 +00:00
John Colvin 9a1cc9664c build: Add install targets for libspark 2024-08-23 14:40:50 +00:00
John Colvin 2e7189463a vscode: Add debug cfg to skip sparkd build 2024-08-13 17:39:33 +00:00
John Colvin 8cde5bcd41 fix build: Add ShmSegmentName.cpp to windows client sources 2024-08-13 17:39:33 +00:00
John Colvin 43d0022878 Make a few timestamp fns const
They should have been const in the first place
2024-08-13 17:39:32 +00:00
John Colvin 65133e8a5e Add config profiles to libspark (check pt)
This is part of a larger effort to unify the libspark build, and eliminate
compile time differences between the builds used by our various services
2024-08-13 17:39:32 +00:00
John Colvin 781bc52ea1 Add api server to docker compose 2024-08-13 17:39:32 +00:00
John Colvin 8de20e51d7 Rename pingtest "sparkd-ping-test", make it depend on sparkd
Add it to a "ping-test" profile, as we may add similar tests for other services
2024-08-13 17:39:32 +00:00
John Colvin 8e57209e52 cleanup: Remove unnecessary env vars from webapp docker compose 2024-08-13 17:39:32 +00:00
John Colvin c5eba43ace Change api server ports so they don't conflict w/ ds webapp 2024-08-13 17:39:32 +00:00
John Colvin 86810bb445 vscode: Fix sparkd debug config 2024-08-13 17:39:32 +00:00
John Colvin ae2c116840 build: Resolve app data dir and shared mem prefix at runtime
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
2024-08-13 17:39:32 +00:00
John Colvin 65aeac3c73 Don't log "initializing spark config" if it has already been initialized 2024-07-31 08:32:04 -04:00
John Colvin 8001c47abc vscode: Add debug cfg for spark api server 2024-07-31 00:08:12 -04:00
John Colvin e775bc19b8 Use dependency injection in spark api server 2024-07-31 00:02:11 -04:00
John Colvin a89c4a5e1e Add SparkLoggerAdapter for Msft ILogger 2024-07-31 00:00:29 -04:00
John Colvin d07eff613f Display exception stack trace w/ 1 call per line in ds webapp 2024-07-30 18:26:58 +00:00
John Colvin fc86a505ee Remove unused ErrorView from ds webapp 2024-07-30 18:25:16 +00:00
John Colvin 06eaec5165 Propagate ViewStream errors to webapp frontend 2024-07-30 18:24:24 +00:00
John Colvin 8e181a4060 Add ErrorChanged event to propagate DataStream exceptions 2024-07-30 17:22:01 +00:00
John Colvin 7c6cbbe83c Set columns to empty array rather than null in ds webapp
If no data has been received. Since the 'columns' property was marked
'required', this eliminates a warning that was being logged to console.
2024-07-30 14:19:51 +00:00
John Colvin 8e7552b144 Improve alignment/spacing in ds webapp 2024-07-30 14:16:34 +00:00
John Colvin 52c1d17ae6 Configure datastream deployment with docker-compose
To deploy to aws, connect to our aws vpn then run:
tools/deploy-docker-compose.sh

The deploy script does the following:
* builds sparkd
* builds datastream apps (pubsub, md-publisher, and webapp)
* uploads the build to aws
* bounces all services (sparkd, pubsub, md-publisher, and webapp)

Note: the name 'md-publisher' refers to DataStreamPublisher w/ MarketData
configuration
2024-07-30 13:59:23 +00:00
John Colvin 62a60267f4 Remove "BTS Cloud Streams" title from webapp
Space is precious when viewing on phone (tbh, it didn't look that good
anyways).
2024-07-30 05:30:26 +00:00
John Colvin f80c876df8 fix: Ensure md-publisher waits for pubsub and sparkd
Bugs (fixed):
  * md-publisher silently fails if it tries to connect before those services are "ready".

The Docker compose "depends_on" property ensures sparkd and pubsub
containers are up, but doesn't prevent startup race bugs (e.g., if
md-publisher and pubsub start around the same time, and md-publisher
tries to connect before pubsub has started accepting connections). To
address these race conditions, we use a nifty tool called "dockerize"

References:
  * https://github.com/jwilder/dockerize
2024-07-29 15:54:26 +00:00
John Colvin f59751b6e1 Replace ErrorView component w/ robust QueryStatusView 2024-07-29 15:48:01 +00:00
John Colvin 10e8298560 Fix typos in query button theme 2024-07-29 15:38:37 +00:00
John Colvin 6261760e48 fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2024-07-29 04:18:19 +00:00
John Colvin 57d17bea3b Change VITE_DATASTREAM_WEBSOCKET_URL to localhost 2024-07-29 04:11:35 +00:00
John Colvin 0b4675c26d Assert that local ip address is successfully obtained 2024-07-29 04:11:35 +00:00
John Colvin 5b2a40ad7f improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't been
tested yet...so we should assume it doesn't work).
2024-07-29 04:10:05 +00:00
John Colvin 6c453ea8aa Assert TcpConnection endpoint matches expected 2024-07-29 03:48:34 +00:00
John Colvin 5823bf1690 Fix typo in log msg (was missing interpolation '$') 2024-07-29 03:48:34 +00:00
John Colvin dbf941ff0e Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2024-07-29 03:48:34 +00:00
John Colvin be7148dcec fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2024-07-29 03:48:34 +00:00
John Colvin 4baa54631e build: Upgrade packages (deps) in DataStreamTests 2024-07-29 03:48:34 +00:00
John Colvin bd0d9ef88d cleanup: Remove unused ViewStreamDatabase config 2024-07-29 03:48:34 +00:00
John Colvin efe6b73935 vscode: Add PubSubServer debug cfg to skip build 2024-07-29 03:48:34 +00:00
John Colvin 5064bbb926 Configure apps as a distributed system via Docker compose 2024-07-29 03:48:34 +00:00
John Colvin 25b92951d8 Remove reference to spark protobuf Config in ViewStream
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2024-07-29 03:24:47 +00:00
John Colvin 8389208ff0 refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2024-07-29 03:24:47 +00:00
John Colvin 0dfd3926ad Remove inappropriate "Unable to dispose subscription" log msg 2024-07-29 03:24:47 +00:00
John Colvin 638d46469f fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2024-07-29 03:13:04 +00:00
John Colvin e6478327d6 Move misplaced log abt initializing query text to appropriate place 2024-07-28 18:10:56 +00:00