Commit Graph
282 Commits
Author SHA1 Message Date
John Colvin 2ddf14d2b5 cleanup: Move global var into function scope in vite cfg 2024-10-30 19:35:05 +00:00
John Colvin 51aa380c3f Update node pkgs in datastream webapp 2024-10-30 19:35:05 +00:00
John Colvin c58a2f90e0 refactor: Organize libspark proj w/ stuttering include
This new structure provides several benefits:

* Allows us to install multiple CMake projects in the same destination
  dir (without include path conflicts).
  - This makes shipping the trading algo sdk a trivial excercise. Simply
    copy the lib and include dirs from the install dest. That's it!
* Puts us in a position to break up the monolithic libspark into smaller
  modules, which are easier to manage.
2024-10-30 19:31:40 +00:00
John Colvin 3af14cecd7 build: Resolve app data dir and shared mem prefix at runtime
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
2024-10-30 19:31:40 +00:00
John Colvin 5c5a0ca09a Display exception stack trace w/ 1 call per line in ds webapp 2024-10-30 19:31:40 +00:00
John Colvin 114873eb35 Remove unused ErrorView from ds webapp 2024-10-30 19:31:40 +00:00
John Colvin a9799b2a00 Propagate ViewStream errors to webapp frontend 2024-10-30 19:31:40 +00:00
John Colvin d5aeaa5e0f Add ErrorChanged event to propagate DataStream exceptions 2024-10-30 19:31:40 +00:00
John Colvin 8a8c491fb1 Set columns to empty array rather than null in ds webapp
If no data has been received. Since the 'columns' property was marked
'required', this eliminates a warning that was being logged to console.
2024-10-30 19:31:39 +00:00
John Colvin 450a6c62b6 Improve alignment/spacing in ds webapp 2024-10-30 19:31:39 +00:00
John Colvin a3887e6246 Configure datastream deployment with docker-compose
To deploy to aws, connect to our aws vpn then run:
tools/deploy-docker-compose.sh

The deploy script does the following:
* builds sparkd
* builds datastream apps (pubsub, md-publisher, and webapp)
* uploads the build to aws
* bounces all services (sparkd, pubsub, md-publisher, and webapp)

Note: the name 'md-publisher' refers to DataStreamPublisher w/ MarketData
configuration
2024-10-30 19:31:39 +00:00
John Colvin 982ddd24d4 Remove "BTS Cloud Streams" title from webapp
Space is precious when viewing on phone (tbh, it didn't look that good
anyways).
2024-10-30 19:31:39 +00:00
John Colvin 20e959fea3 fix: Ensure md-publisher waits for pubsub and sparkd
Bugs (fixed):
  * md-publisher silently fails if it tries to connect before those services are "ready".

The Docker compose "depends_on" property ensures sparkd and pubsub
containers are up, but doesn't prevent startup race bugs (e.g., if
md-publisher and pubsub start around the same time, and md-publisher
tries to connect before pubsub has started accepting connections). To
address these race conditions, we use a nifty tool called "dockerize"

References:
  * https://github.com/jwilder/dockerize
2024-10-30 19:31:39 +00:00
John Colvin a108823ea5 Replace ErrorView component w/ robust QueryStatusView 2024-10-30 19:31:39 +00:00
John Colvin 52c74cd800 Fix typos in query button theme 2024-10-30 19:31:39 +00:00
John Colvin a473cbc703 fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2024-10-30 19:31:39 +00:00
John Colvin 63c648a659 Change VITE_DATASTREAM_WEBSOCKET_URL to localhost 2024-10-30 19:31:39 +00:00
John Colvin 3ffe02254e Assert that local ip address is successfully obtained 2024-10-30 19:31:39 +00:00
John Colvin f5de2b5355 improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't been
tested yet...so we should assume it doesn't work).
2024-10-30 19:31:39 +00:00
John Colvin 979b51fb20 Assert TcpConnection endpoint matches expected 2024-10-30 19:31:39 +00:00
John Colvin 4efb0bfc42 Fix typo in log msg (was missing interpolation '$') 2024-10-30 19:31:39 +00:00
John Colvin f925be32a6 Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2024-10-30 19:31:39 +00:00
John Colvin f3b5ff8970 fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2024-10-30 19:31:39 +00:00
John Colvin fdea340492 build: Upgrade packages (deps) in DataStreamTests 2024-10-30 19:31:39 +00:00
John Colvin 5ce2ffa5d6 cleanup: Remove unused ViewStreamDatabase config 2024-10-30 19:31:39 +00:00
John Colvin a1d7607065 Configure apps as a distributed system via Docker compose 2024-10-30 19:31:39 +00:00
John Colvin 233d089c34 Remove reference to spark protobuf Config in ViewStream
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2024-10-30 19:31:39 +00:00
John Colvin 759f6dd59a refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2024-10-30 19:31:39 +00:00
John Colvin 80622ddaba Remove inappropriate "Unable to dispose subscription" log msg 2024-10-30 19:31:39 +00:00
John Colvin a90e8807ad Move misplaced log abt initializing query text to appropriate place 2024-10-30 19:31:39 +00:00
John Colvin 6dc25bab01 Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2024-10-30 19:31:39 +00:00
John Colvin 00042d2471 cleanup: Remove unused code from ViewStreamDatabase 2024-10-30 19:31:39 +00:00
John Colvin de9d47cd00 Include subscription id in ds websocket updates 2024-10-30 19:31:39 +00:00
John Colvin 461f690463 fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2024-10-30 19:31:39 +00:00
John Colvin a425d3e6ba Log event data in websocket open, close, error callbacks 2024-10-30 19:31:39 +00:00
John Colvin fe81b3107f refactor: Bring back Material DataGrid
Fix the column resize bugs that made it previously unusable
2024-10-30 19:31:39 +00:00
John Colvin d5d12c56c9 refactor: Improve react webapp state management
* Add queryTextResetTime as part of the CannedQuery state
  (the QueryControl compares the canned query click time with the time of the
  last user input, and applies the text from the most recent change)

* Replace 'isRunning' (bool) with a 'queryState' (string) to reflect the subscription status
  value received from the server. Less indirection this way.

* Replace inappropriate useMemo call w/ useEffect
2024-10-30 19:31:39 +00:00
John Colvin 8979d5def8 cleanup: Get rid of magic const (subscriptionId) in webapp App.jsx 2024-10-30 19:31:39 +00:00
John Colvin 5da807b5f0 cleanup: Remove unnecessary deps from csprojs 2024-10-30 19:31:39 +00:00
John Colvin 212bd432af fix: Downgrade Microsoft.Extensions.Hosting to 7.0.1
A handful of csprojs were referencing version 8.0.0. Since that version
targets net8.0, I have no idea why it didn't cause the build to fail.
Instead, this framework mismatch led to a mysterious runtime crash with
the following error, despite the dll in question being present in the
build dir:

Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'System.Diagnostics.DiagnosticSource, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
The system cannot find the file specified.

It is worth noting that this crash would only happen for exes published
as part of the DataStreamSuite.sln with the "--self-contained" flag. For
example, the Bts.DataStreamSuite.Server exe would always fail if
published via:

dotnet publish --configuration Release --sc --framework net7.0 --runtime linux-x64 <DataStreamSuite.sln>
2024-10-30 19:31:39 +00:00
John Colvin 4361ac6519 Add HttpsRedirection middleware first in the webapp pipeline
Middleware handlers are invoked in the order in which they are added. Any middleware
that is added before HttpsRedirection may can prevent the request from being redirected.
As of now, only development-specific middleware is added ahead of HttpsRedircetion.
2024-10-30 19:31:09 +00:00
John Colvin 93dc4c9c1f build: Copy configuration profiles to build dir
So they are readily available to publish/deploy scripts.
2024-10-30 19:31:09 +00:00
John Colvin ec88ffdbf3 fix: Catch exception when creating TcpClient 2024-10-30 19:31:09 +00:00
John Colvin 26fecc73c6 fix: Catch WebSocketException in DataStreamSubscription 2024-10-30 19:31:09 +00:00
John Colvin 05715eb489 fix: Add null check to NonIncGroupByStream 2024-10-30 19:31:09 +00:00
John Colvin 70019c59e5 fix: Add null check to TableAccumulator2 2024-10-30 19:31:09 +00:00
John Colvin 6caa98a6b3 fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2024-10-30 19:31:09 +00:00
John Colvin 40930628fb fix: Add null check in TimeBasedFilter.cs 2024-10-30 19:31:09 +00:00
John Colvin d439e3e695 Reduce task wait/timeout code in AsyncTCPConnector 2024-10-30 19:31:09 +00:00
John Colvin aae50cd5f8 Add a few canned queries to AwsProd.json for ds webapp 2024-10-30 19:31:09 +00:00