Commit Graph
13608 Commits
Author SHA1 Message Date
John Colvin 9ec5bf4ea5 fix build: Add ShmSegmentName.cpp to windows client sources 2024-10-14 17:21:49 +00:00
John Colvin 5507743201 Make a few timestamp fns const
They should have been const in the first place
2024-10-14 17:21:49 +00:00
John Colvin e0e4bf9764 Add config profiles to libspark
This is part of a larger effort to unify the libspark build, and eliminate
compile time differences between the builds used by our various services
2024-10-14 17:21:49 +00:00
John Colvin 95ba1bb0bd Add api server to docker compose 2024-10-14 17:21:49 +00:00
John Colvin a524943f47 Rename pingtest "sparkd-ping-test", make it depend on sparkd
Add it to a "ping-test" profile, as we may add similar tests for other services
2024-10-14 17:21:49 +00:00
John Colvin 19804866ad cleanup: Remove unnecessary env vars from webapp docker compose 2024-10-14 17:21:49 +00:00
John Colvin ef5835a83c Change api server ports so they don't conflict w/ ds webapp 2024-10-14 17:21:49 +00:00
John Colvin 86b9c67c54 vscode: Fix sparkd debug config 2024-10-14 17:21:49 +00:00
John Colvin 0a6d960522 build: Resolve app data dir and shared mem prefix at runtime
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
2024-10-14 17:21:49 +00:00
John Colvin eb8be7a98f Don't log "initializing spark config" if it has already been initialized 2024-10-14 17:21:49 +00:00
John Colvin 8c05bcf4f7 vscode: Add debug cfg for spark api server 2024-10-14 17:21:49 +00:00
John Colvin b788505646 Use dependency injection in spark api server 2024-10-14 17:21:49 +00:00
John Colvin fcf5707ef1 Add SparkLoggerAdapter for Msft ILogger 2024-10-14 17:21:49 +00:00
John Colvin 38dcfeae79 Display exception stack trace w/ 1 call per line in ds webapp 2024-10-14 17:21:49 +00:00
John Colvin be4556a6b2 Remove unused ErrorView from ds webapp 2024-10-14 17:21:49 +00:00
John Colvin 939b783583 Propagate ViewStream errors to webapp frontend 2024-10-14 17:21:49 +00:00
John Colvin 466d3eb432 Add ErrorChanged event to propagate DataStream exceptions 2024-10-14 17:21:49 +00:00
John Colvin d3f069439d Set columns to empty array rather than null in ds webapp
If no data has been received. Since the 'columns' property was marked
'required', this eliminates a warning that was being logged to console.
2024-10-14 17:21:49 +00:00
John Colvin bc9102b7b5 Improve alignment/spacing in ds webapp 2024-10-14 17:21:49 +00:00
John Colvin b0bc68997b Configure datastream deployment with docker-compose
To deploy to aws, connect to our aws vpn then run:
tools/deploy-docker-compose.sh

The deploy script does the following:
* builds sparkd
* builds datastream apps (pubsub, md-publisher, and webapp)
* uploads the build to aws
* bounces all services (sparkd, pubsub, md-publisher, and webapp)

Note: the name 'md-publisher' refers to DataStreamPublisher w/ MarketData
configuration
2024-10-14 17:21:49 +00:00
John Colvin ad1e9b1fe2 Remove "BTS Cloud Streams" title from webapp
Space is precious when viewing on phone (tbh, it didn't look that good
anyways).
2024-10-14 17:21:48 +00:00
John Colvin 1efbca9a49 fix: Ensure md-publisher waits for pubsub and sparkd
Bugs (fixed):
  * md-publisher silently fails if it tries to connect before those services are "ready".

The Docker compose "depends_on" property ensures sparkd and pubsub
containers are up, but doesn't prevent startup race bugs (e.g., if
md-publisher and pubsub start around the same time, and md-publisher
tries to connect before pubsub has started accepting connections). To
address these race conditions, we use a nifty tool called "dockerize"

References:
  * https://github.com/jwilder/dockerize
2024-10-14 17:21:47 +00:00
John Colvin 00f825802d Replace ErrorView component w/ robust QueryStatusView 2024-10-14 17:21:47 +00:00
John Colvin 4dc95e1979 Fix typos in query button theme 2024-10-14 17:21:47 +00:00
John Colvin fd0f95a884 fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2024-10-14 17:21:47 +00:00
John Colvin 00f866d3f5 Change VITE_DATASTREAM_WEBSOCKET_URL to localhost 2024-10-14 17:21:47 +00:00
John Colvin 17b2d56391 Assert that local ip address is successfully obtained 2024-10-14 17:21:47 +00:00
John Colvin 0d9699b79c improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't been
tested yet...so we should assume it doesn't work).
2024-10-14 17:21:47 +00:00
John Colvin 87c3471c9c Assert TcpConnection endpoint matches expected 2024-10-14 17:21:46 +00:00
John Colvin 2bb6458c44 Fix typo in log msg (was missing interpolation '$') 2024-10-14 17:21:46 +00:00
John Colvin 768ec3ca34 Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2024-10-14 17:21:46 +00:00
John Colvin b10135aded fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2024-10-14 17:21:46 +00:00
John Colvin 00dbbfa230 build: Upgrade packages (deps) in DataStreamTests 2024-10-14 17:21:46 +00:00
John Colvin f4570b2083 cleanup: Remove unused ViewStreamDatabase config 2024-10-14 17:21:46 +00:00
John Colvin 5e79a58253 vscode: Add PubSubServer debug cfg to skip build 2024-10-14 17:21:46 +00:00
John Colvin 61ddefa862 Configure apps as a distributed system via Docker compose 2024-10-14 17:21:46 +00:00
John Colvin 490e425fc3 Remove reference to spark protobuf Config in ViewStream
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2024-10-14 17:21:46 +00:00
John Colvin 0382d5de3b refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2024-10-14 17:21:46 +00:00
John Colvin 0ae1d61ce1 Remove inappropriate "Unable to dispose subscription" log msg 2024-10-14 17:21:46 +00:00
John Colvin 3c1d849989 fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2024-10-14 17:21:46 +00:00
John Colvin 1b26f4b114 Move misplaced log abt initializing query text to appropriate place 2024-10-14 17:21:46 +00:00
John Colvin 8a2dea53e6 Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2024-10-14 17:21:46 +00:00
John Colvin e396ebb181 cleanup: Remove unused code from ViewStreamDatabase 2024-10-14 17:21:46 +00:00
John Colvin 7f9be44cc2 Include subscription id in ds websocket updates 2024-10-14 17:21:46 +00:00
John Colvin 0f88a15c57 fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2024-10-14 17:21:46 +00:00
John Colvin b84dfce39d Log event data in websocket open, close, error callbacks 2024-10-14 17:21:46 +00:00
John Colvin 86bddb7fd6 refactor: Bring back Material DataGrid
Fix the column resize bugs that made it previously unusable
2024-10-14 17:21:46 +00:00
John Colvin 3b4e424dd3 refactor: Improve react webapp state management
* Add queryTextResetTime as part of the CannedQuery state
  (the QueryControl compares the canned query click time with the time of the
  last user input, and applies the text from the most recent change)

* Replace 'isRunning' (bool) with a 'queryState' (string) to reflect the subscription status
  value received from the server. Less indirection this way.

* Replace inappropriate useMemo call w/ useEffect
2024-10-14 17:21:46 +00:00
John Colvin 96dee6ac0e cleanup: Get rid of magic const (subscriptionId) in webapp App.jsx 2024-10-14 17:21:46 +00:00
John Colvin e0ea6418bd build: Copy configuration profiles to publish dir 2024-10-14 17:21:46 +00:00