Commit Graph
14347 Commits
Author SHA1 Message Date
John Colvin 7401b90486 Add ErrorChanged event to propagate DataStream exceptions 2025-07-23 16:39:28 -04:00
John Colvin f5c2537b26 Set columns to empty array rather than null in ds webapp
If no data has been received. Since the 'columns' property was marked
'required', this eliminates a warning that was being logged to console.
2025-07-23 16:34:04 -04:00
John Colvin 993ab27323 Improve alignment/spacing in ds webapp 2025-07-23 16:34:04 -04:00
John Colvin 8e1243696f Remove "BTS Cloud Streams" title from webapp
Space is precious when viewing on phone (tbh, it didn't look that good
anyways).
2025-07-23 16:31:31 -04:00
John Colvin ec37426c35 fix: Ensure md-publisher waits for pubsub and sparkd
Bugs (fixed):
  * md-publisher silently fails if it tries to connect before those services are "ready".

The Docker compose "depends_on" property ensures sparkd and pubsub
containers are up, but doesn't prevent startup race bugs (e.g., if
md-publisher and pubsub start around the same time, and md-publisher
tries to connect before pubsub has started accepting connections). To
address these race conditions, we use a nifty tool called "dockerize"

References:
  * https://github.com/jwilder/dockerize
2025-07-23 16:31:31 -04:00
John Colvin 3d72ee3cc8 Replace ErrorView component w/ robust QueryStatusView 2025-07-23 16:31:31 -04:00
John Colvin 409d650652 Fix typos in query button theme 2025-07-23 16:31:31 -04:00
John Colvin c51225b98f fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2025-07-23 16:31:31 -04:00
John Colvin 3bb153ea4a Change VITE_DATASTREAM_WEBSOCKET_URL to localhost 2025-07-23 16:31:31 -04:00
John Colvin c4ac7a4166 Assert that local ip address is successfully obtained 2025-07-23 16:31:31 -04:00
John Colvin 07fcee15f6 improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't been
tested yet...so we should assume it doesn't work).
2025-07-23 16:31:31 -04:00
John Colvin ac9c407f64 Assert TcpConnection endpoint matches expected 2025-07-23 16:31:31 -04:00
John Colvin f1a7d23893 Fix typo in log msg (was missing interpolation '$') 2025-07-23 16:31:31 -04:00
John Colvin edc18b15c2 Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2025-07-23 16:31:31 -04:00
John Colvin 8ee7fa99b5 fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2025-07-23 16:29:07 -04:00
John Colvin aa6a09c47f build: Upgrade packages (deps) in DataStreamTests 2025-07-23 16:29:06 -04:00
John Colvin 7014898f1f cleanup: Remove unused ViewStreamDatabase config 2025-07-23 16:29:06 -04:00
John Colvin 2cdd394745 vscode: Add PubSubServer debug cfg to skip build 2025-07-23 16:29:06 -04:00
John Colvin 3c0c082dd1 deploy: Add datastream services to Docker compose 2025-07-23 16:27:41 -04:00
John Colvin af693ee229 Remove reference to spark protobuf Config in ViewStream
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-07-23 16:13:26 -04:00
John Colvin 10270854f2 refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-07-23 16:13:26 -04:00
John Colvin 14715f7ea7 Remove inappropriate "Unable to dispose subscription" log msg 2025-07-23 16:13:26 -04:00
John Colvin 2a1161a41b fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-07-23 16:13:26 -04:00
John Colvin 2879967d6a Move misplaced log abt initializing query text to appropriate place 2025-07-23 16:12:57 -04:00
John Colvin 1fc8afd6af Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-07-23 16:12:57 -04:00
John Colvin be35726582 cleanup: Remove unused code from ViewStreamDatabase 2025-07-23 16:12:57 -04:00
John Colvin 77ffd8145b Include subscription id in ds websocket updates 2025-07-23 16:12:57 -04:00
John Colvin c6308db55f fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2025-07-23 16:12:57 -04:00
John Colvin b7aac2e431 Log event data in websocket open, close, error callbacks 2025-07-23 16:12:57 -04:00
John Colvin e72ef3b775 refactor: Bring back Material DataGrid
Fix the column resize bugs that made it previously unusable
2025-07-23 16:12:57 -04:00
John Colvin 9ec0610be4 refactor: Improve react webapp state management
* Add queryTextResetTime as part of the CannedQuery state
  (the QueryControl compares the canned query click time with the time of the
  last user input, and applies the text from the most recent change)

* Replace 'isRunning' (bool) with a 'queryState' (string) to reflect the subscription status
  value received from the server. Less indirection this way.

* Replace inappropriate useMemo call w/ useEffect
2025-07-23 16:12:57 -04:00
John Colvin be502be538 cleanup: Get rid of magic const (subscriptionId) in webapp App.jsx 2025-07-23 16:12:57 -04:00
John Colvin af819af516 build: Copy configuration profiles to publish dir 2025-07-23 16:12:57 -04:00
John Colvin 69da3d46ab cleanup: Remove unnecessary deps from csprojs 2025-07-23 16:12:57 -04:00
John Colvin 64b4eec999 build: Change publish_data_stream.py to publish the sln
We were publishing individual projects as a workaround for the
dependency issue (bug) that caused some projects to incorrectly target
net8.0. The fact that publishing individual projects "just worked" was
probably dumb luck. Now that the underlying issue has been addressed, we
can resume publishing the sln as a whole.
2025-07-23 16:11:50 -04:00
John Colvin 563610c2cf fix: Downgrade Microsoft.Extensions.Hosting to 7.0.1
A handful of csprojs were referencing version 8.0.0. Since that version
targets net8.0, I have no idea why it didn't cause the build to fail.
Instead, this framework mismatch led to a mysterious runtime crash with
the following error, despite the dll in question being present in the
build dir:

Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'System.Diagnostics.DiagnosticSource, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
The system cannot find the file specified.

It is worth noting that this crash would only happen for exes published
as part of the DataStreamSuite.sln with the "--self-contained" flag. For
example, the Bts.DataStreamSuite.Server exe would always fail if
published via:

dotnet publish --configuration Release --sc --framework net7.0 --runtime linux-x64 <DataStreamSuite.sln>
2025-07-23 16:09:22 -04:00
John Colvin c46e694ccd Add HttpsRedirection middleware first in the webapp pipeline
Middleware handlers are invoked in the order in which they are added. Any middleware
that is added before HttpsRedirection may can prevent the request from being redirected.
As of now, only development-specific middleware is added ahead of HttpsRedircetion.
2025-07-23 16:09:22 -04:00
John Colvin b6ee5335e5 build: Copy configuration profiles to build dir
So they are readily available to publish/deploy scripts.
2025-07-23 16:09:22 -04:00
John Colvin 4272f6dc41 Fix benign typo in Scanner/Parser msbuild targets
The output path referenced a non-existant msbuild metadata prop "RelativeOutputDir"
(the actual prop is called "RelativeDir"). Also, the OutputFile path is relative to the
project dir (the 'MSBuildProjectDir' prop), not the file being included.
2025-07-23 16:09:22 -04:00
John Colvin f4e4b56b3e fix: Catch exception when creating TcpClient 2025-07-23 16:08:14 -04:00
John Colvin 542df5ff82 fix: Catch WebSocketException in DataStreamSubscription 2025-07-23 16:08:14 -04:00
John Colvin 997a97be40 fix: Add null check to NonIncGroupByStream 2025-07-23 16:08:14 -04:00
John Colvin 025eb4a4fc fix: Add null check to TableAccumulator2 2025-07-23 16:08:14 -04:00
John Colvin 844c842357 fix: Add self-signed certificate to Dockerfile for ds webapp 2025-07-23 16:08:14 -04:00
John Colvin a7d873f42f fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-07-23 16:08:14 -04:00
John Colvin c8970408a4 fix: Add null check in TimeBasedFilter.cs 2025-07-23 16:08:14 -04:00
John Colvin 3cef34179e Reduce task wait/timeout code in AsyncTCPConnector 2025-07-23 16:08:14 -04:00
John Colvin 349a058567 Add a few canned queries to AwsProd.json for ds webapp 2025-07-23 16:08:14 -04:00
John Colvin 32599bdeff fix: Copy canned queries to clipboard on click
Instead of trying to update the text directly, which causes the front end
to overwrite the user's changes everytime the data updates from then on.
2025-07-23 16:08:14 -04:00
John Colvin 89845ace1c fix: Convert invalid doubles to nan in ds webapp serializer 2025-07-23 16:07:35 -04:00