Commit Graph
13552 Commits
Author SHA1 Message Date
John Colvin 07c09b748b Assert TcpConnection endpoint matches expected 2024-09-22 18:46:10 +00:00
John Colvin e71f36d012 Fix typo in log msg (was missing interpolation '$') 2024-09-22 18:46:10 +00:00
John Colvin c320bd23d6 Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2024-09-22 18:46:10 +00:00
John Colvin b9ba9b0430 fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2024-09-22 18:46:10 +00:00
John Colvin 4eb80820bf build: Upgrade packages (deps) in DataStreamTests 2024-09-22 18:46:10 +00:00
John Colvin f58b40f6cc cleanup: Remove unused ViewStreamDatabase config 2024-09-22 18:46:10 +00:00
John Colvin f941c100b7 vscode: Add PubSubServer debug cfg to skip build 2024-09-22 18:46:10 +00:00
John Colvin 314244c693 Configure apps as a distributed system via Docker compose 2024-09-22 18:46:08 +00:00
John Colvin 6798efd9f5 Remove reference to spark protobuf Config in ViewStream
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2024-09-22 18:36:36 +00:00
John Colvin d6b6ab08ac refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2024-09-22 18:36:36 +00:00
John Colvin 13bb4d41a8 Remove inappropriate "Unable to dispose subscription" log msg 2024-09-22 18:36:36 +00:00
John Colvin 2cf72277b3 fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2024-09-22 18:36:36 +00:00
John Colvin d81782a1b8 Move misplaced log abt initializing query text to appropriate place 2024-09-22 18:36:36 +00:00
John Colvin 64f5ed4bd6 Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2024-09-22 18:36:36 +00:00
John Colvin 080b8177b5 cleanup: Remove unused code from ViewStreamDatabase 2024-09-22 18:36:36 +00:00
John Colvin 9ec0a5b8e5 Include subscription id in ds websocket updates 2024-09-22 18:36:36 +00:00
John Colvin aaa9995ad0 fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2024-09-22 18:36:36 +00:00
John Colvin e763d79f26 Log event data in websocket open, close, error callbacks 2024-09-22 18:36:36 +00:00
John Colvin d20580cebb refactor: Bring back Material DataGrid
Fix the column resize bugs that made it previously unusable
2024-09-22 18:36:36 +00:00
John Colvin 727e6a94e6 refactor: Improve react webapp state management
* Add queryTextResetTime as part of the CannedQuery state
  (the QueryControl compares the canned query click time with the time of the
  last user input, and applies the text from the most recent change)

* Replace 'isRunning' (bool) with a 'queryState' (string) to reflect the subscription status
  value received from the server. Less indirection this way.

* Replace inappropriate useMemo call w/ useEffect
2024-09-22 18:36:36 +00:00
John Colvin d73a0de07d cleanup: Get rid of magic const (subscriptionId) in webapp App.jsx 2024-09-22 18:36:36 +00:00
John Colvin f5f9692c33 build: Copy configuration profiles to publish dir 2024-09-22 18:36:36 +00:00
John Colvin 4583a09b70 cleanup: Remove unnecessary deps from csprojs 2024-09-22 18:36:36 +00:00
John Colvin 0236bb9e58 build: Change publish_data_stream.py to publish the sln
We were publishing individual projects as a workaround for the
dependency issue (bug) that caused some projects to incorrectly target
net8.0. The fact that publishing individual projects "just worked" was
probably dumb luck. Now that the underlying issue has been addressed, we
can resume publishing the sln as a whole.
2024-09-22 18:36:36 +00:00
John Colvin b29f32e75d dev: Install docker in devcontainer
So we can start various services like sparkd, pubsub, ds publishers, ds
webapp, etc. from inside the devcontainer.
2024-09-22 18:36:36 +00:00
John Colvin 9d1c00f4d6 dev: Remove dotnet8 from devcontainer
We'll upgrade to dotnet8 eventually, but now isn't the time (too much
more important work to be done).
2024-09-22 18:36:36 +00:00
John Colvin 361da57419 dev: Run devcontainer on host network
So we don't need to worry about publishing the right ports (and don't
need to rebuild the container everytime the list of ports changes).
2024-09-22 18:36:36 +00:00
John Colvin ce8ae14a54 fix: Downgrade Microsoft.Extensions.Hosting to 7.0.1
A handful of csprojs were referencing version 8.0.0. Since that version
targets net8.0, I have no idea why it didn't cause the build to fail.
Instead, this framework mismatch led to a mysterious runtime crash with
the following error, despite the dll in question being present in the
build dir:

Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'System.Diagnostics.DiagnosticSource, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
The system cannot find the file specified.

It is worth noting that this crash would only happen for exes published
as part of the DataStreamSuite.sln with the "--self-contained" flag. For
example, the Bts.DataStreamSuite.Server exe would always fail if
published via:

dotnet publish --configuration Release --sc --framework net7.0 --runtime linux-x64 <DataStreamSuite.sln>
2024-09-22 18:36:36 +00:00
John Colvin 0e7b7c5599 Add HttpsRedirection middleware first in the webapp pipeline
Middleware handlers are invoked in the order in which they are added. Any middleware
that is added before HttpsRedirection may can prevent the request from being redirected.
As of now, only development-specific middleware is added ahead of HttpsRedircetion.
2024-09-22 18:36:36 +00:00
John Colvin 2fecb73e88 build: Copy configuration profiles to build dir
So they are readily available to publish/deploy scripts.
2024-09-22 18:36:36 +00:00
John Colvin b45d93342e Fix benign typo in Scanner/Parser msbuild targets
The output path referenced a non-existant msbuild metadata prop "RelativeOutputDir"
(the actual prop is called "RelativeDir"). Also, the OutputFile path is relative to the
project dir (the 'MSBuildProjectDir' prop), not the file being included.
2024-09-22 18:36:36 +00:00
John Colvin 048834c2c4 fix: Catch exception when creating TcpClient 2024-09-22 18:36:36 +00:00
John Colvin ba6938b09a fix: Catch WebSocketException in DataStreamSubscription 2024-09-22 18:36:36 +00:00
John Colvin 52eaa5622b fix: Add null check to NonIncGroupByStream 2024-09-22 18:36:36 +00:00
John Colvin 012ddeff7e fix: Add null check to TableAccumulator2 2024-09-22 18:36:36 +00:00
John Colvin 727235950f fix: Add self-signed certificate to Dockerfile for ds webapp 2024-09-22 18:36:36 +00:00
John Colvin 8b779597db fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2024-09-22 18:36:36 +00:00
John Colvin 5eecd2ffbc fix: Add null check in TimeBasedFilter.cs 2024-09-22 18:36:36 +00:00
John Colvin b12c28a074 Reduce task wait/timeout code in AsyncTCPConnector 2024-09-22 18:36:36 +00:00
John Colvin 02d19c522d Add a few canned queries to AwsProd.json for ds webapp 2024-09-22 18:36:36 +00:00
John Colvin a00e849192 fix: Copy canned queries to clipboard on click
Instead of trying to update the text directly, which causes the front end
to overwrite the user's changes everytime the data updates from then on.
2024-09-22 18:36:36 +00:00
John Colvin d0601ad32d fix: Convert invalid doubles to nan in ds webapp serializer 2024-09-22 18:36:36 +00:00
John Colvin 149ab49212 Increase datagrid maxheight in ds webapp to 1000px 2024-09-22 18:36:36 +00:00
John Colvin 23c9f0a9de Improve error display in ds webapp 2024-09-22 18:36:36 +00:00
John Colvin 22b04c8ec8 fix: Dispose existing sub on subscribe req in ds webapp
Even if new sub has parse error. This fixes a bug where the previous
subscription would continue firing and updating the frontend, when it
should just display the error info to the user and cease updates.
2024-09-22 18:36:36 +00:00
John Colvin fdba4601ab fix: Get rid of "UseSpa" calls in webapp Production environment
This was leftover from a misguided attempt to fix a bug that turned out
to be due to an omitted environment var:
ASPNETCORE_HOSTINGSTARTUPASSEMBLIES=Microsoft.AspNetCore.SpaProxy
2024-09-22 18:36:35 +00:00
John Colvin dd2dffd337 Define webapp specific env vars in generic datastream app deploy script
For now, this makes things easier, as I don't need to write a special script just for the webapp
2024-09-22 18:36:35 +00:00
John Colvin 2c58415a4f Add config profile and Dockerfile args to deploy script 2024-09-22 18:36:35 +00:00
John Colvin f85f57d2a5 Copy DataStream WebApp 'dist' to 'wwwroot' in deploy script 2024-09-22 18:36:35 +00:00
John Colvin 4d5f87df33 Tweak DataGrid style in DataStream WebApp 2024-09-22 18:36:35 +00:00