Commit Graph
13558 Commits
Author SHA1 Message Date
John Colvin a667d486cb dev: Install docker in devcontainer
So we can start various services like sparkd, pubsub, ds publishers, ds
webapp, etc. from inside the devcontainer.
2024-10-21 13:29:58 +00:00
John Colvin 44610a8569 dev: Remove dotnet8 from devcontainer
We'll upgrade to dotnet8 eventually, but now isn't the time (too much
more important work to be done).
2024-10-21 13:29:58 +00:00
John Colvin 7b7fbbaf8b dev: Run devcontainer on host network
So we don't need to worry about publishing the right ports (and don't
need to rebuild the container everytime the list of ports changes).
2024-10-21 13:29:58 +00:00
John Colvin c45ef25a31 fix: Downgrade Microsoft.Extensions.Hosting to 7.0.1
A handful of csprojs were referencing version 8.0.0. Since that version
targets net8.0, I have no idea why it didn't cause the build to fail.
Instead, this framework mismatch led to a mysterious runtime crash with
the following error, despite the dll in question being present in the
build dir:

Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'System.Diagnostics.DiagnosticSource, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
The system cannot find the file specified.

It is worth noting that this crash would only happen for exes published
as part of the DataStreamSuite.sln with the "--self-contained" flag. For
example, the Bts.DataStreamSuite.Server exe would always fail if
published via:

dotnet publish --configuration Release --sc --framework net7.0 --runtime linux-x64 <DataStreamSuite.sln>
2024-10-21 13:29:58 +00:00
John Colvin 431ef0ae70 Add HttpsRedirection middleware first in the webapp pipeline
Middleware handlers are invoked in the order in which they are added. Any middleware
that is added before HttpsRedirection may can prevent the request from being redirected.
As of now, only development-specific middleware is added ahead of HttpsRedircetion.
2024-10-21 13:29:58 +00:00
John Colvin a770724de1 build: Copy configuration profiles to build dir
So they are readily available to publish/deploy scripts.
2024-10-21 13:29:58 +00:00
John Colvin c8c464c30e Fix benign typo in Scanner/Parser msbuild targets
The output path referenced a non-existant msbuild metadata prop "RelativeOutputDir"
(the actual prop is called "RelativeDir"). Also, the OutputFile path is relative to the
project dir (the 'MSBuildProjectDir' prop), not the file being included.
2024-10-21 13:29:58 +00:00
John Colvin 5a9930334e fix: Catch exception when creating TcpClient 2024-10-21 13:29:58 +00:00
John Colvin 69ba13f3a1 fix: Catch WebSocketException in DataStreamSubscription 2024-10-21 13:29:58 +00:00
John Colvin 9b0e1642a9 fix: Add null check to NonIncGroupByStream 2024-10-21 13:29:58 +00:00
John Colvin bec2932203 fix: Add null check to TableAccumulator2 2024-10-21 13:29:58 +00:00
John Colvin e76c4d3aaf fix: Add self-signed certificate to Dockerfile for ds webapp 2024-10-21 13:29:58 +00:00
John Colvin 25d2e8048e fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2024-10-21 13:29:58 +00:00
John Colvin 0e8726c78c fix: Add null check in TimeBasedFilter.cs 2024-10-21 13:29:58 +00:00
John Colvin a1d3c50aef Reduce task wait/timeout code in AsyncTCPConnector 2024-10-21 13:29:58 +00:00
John Colvin d84d993b7d Add a few canned queries to AwsProd.json for ds webapp 2024-10-21 13:29:58 +00:00
John Colvin c58e9b9e96 fix: Copy canned queries to clipboard on click
Instead of trying to update the text directly, which causes the front end
to overwrite the user's changes everytime the data updates from then on.
2024-10-21 13:29:58 +00:00
John Colvin 21240312e7 fix: Convert invalid doubles to nan in ds webapp serializer 2024-10-21 13:29:58 +00:00
John Colvin 5f75a110b7 Increase datagrid maxheight in ds webapp to 1000px 2024-10-21 13:29:58 +00:00
John Colvin 9faf0d69d4 Improve error display in ds webapp 2024-10-21 13:29:58 +00:00
John Colvin c81e84fce1 fix: Dispose existing sub on subscribe req in ds webapp
Even if new sub has parse error. This fixes a bug where the previous
subscription would continue firing and updating the frontend, when it
should just display the error info to the user and cease updates.
2024-10-21 13:29:58 +00:00
John Colvin d1dc6c6e6b fix: Get rid of "UseSpa" calls in webapp Production environment
This was leftover from a misguided attempt to fix a bug that turned out
to be due to an omitted environment var:
ASPNETCORE_HOSTINGSTARTUPASSEMBLIES=Microsoft.AspNetCore.SpaProxy
2024-10-21 13:29:58 +00:00
John Colvin aa7dda3dca Define webapp specific env vars in generic datastream app deploy script
For now, this makes things easier, as I don't need to write a special script just for the webapp
2024-10-21 13:29:58 +00:00
John Colvin af2536bbb9 Add config profile and Dockerfile args to deploy script 2024-10-21 13:29:58 +00:00
John Colvin b6cb7fcd1a Copy DataStream WebApp 'dist' to 'wwwroot' in deploy script 2024-10-21 13:29:58 +00:00
John Colvin bda7052e40 Tweak DataGrid style in DataStream WebApp 2024-10-21 13:29:58 +00:00
John Colvin 4710d3e072 Add AwsProd config for DataStream WebApp 2024-10-21 13:29:58 +00:00
John Colvin bafc7340fa fix: Add VITE_ prefix to imported env vars
For the DataStream WebApp client
2024-10-21 13:29:58 +00:00
John Colvin 0f2f4df3c5 Fix urls for DataStreamPublisher AwsProd.json 2024-10-21 13:29:58 +00:00
John Colvin bf66d39b27 Add benzinga news streams to AwsProd.json
This is a config profile for DataStreamPublisher, designed to be deployed
to aws (on public internet).
2024-10-21 13:29:58 +00:00
John Colvin a34836540e fix build warnings: Make props nullable strings in SharedStreamConfig 2024-10-21 13:29:58 +00:00
John Colvin e855f136a1 whitespace: Format PubSubServer.json (indentation issues) 2024-10-21 13:29:58 +00:00
John Colvin 41115204f3 fix import from environment variable 2024-10-21 13:29:58 +00:00
John Colvin 4be87728ce Add a few tasks to run DataStream WebApp 2024-10-21 13:29:58 +00:00
John Colvin 9f7353d5b6 fix: Import websocket url from environment var 2024-10-21 13:29:58 +00:00
John Colvin 894add7df6 Add scripts to deploy data stream apps 2024-10-21 13:29:58 +00:00
John Colvin 73ce6d129e Add SpaServer to Production environment
For Bts.DataStreamWebApp.Server
2024-10-21 13:29:58 +00:00
John Colvin ad2bd712c5 Change DirectoryBrowser requestPath to "/browse"
So we can view a psuedo file explorer for the webapp by requesting the "/browse" path
(only activated in Development environment).
2024-10-21 13:29:58 +00:00
John Colvin d78079f442 Publish Bts.DataStreamWebApp.Server along with PubSub & DataStream 2024-10-21 13:29:58 +00:00
John Colvin 928ba5d056 cleanup: Remove unused code in Bts.DataStreamWebApp.Client 2024-10-21 13:29:58 +00:00
John Colvin 762a0914f3 Capture OperationCanceledException (instead of TaskCanceledException)
The former is the base class for the latter, so this will widen the umbrella to catch a few corner cases
that would otherwise been handled by the generic "Exception" catch block. One corner case I know
of is when an op is canceled via "CancellationToken.ThrowIfCancellationRequested". In that case,
an OperationCanceledException is thrown (not a TaskCanceledException).

https://stackoverflow.com/questions/34359129/operationcanceledexception-vs-taskcanceledexception-when-task-is-canceled
2024-10-21 13:29:58 +00:00
John Colvin 6a18eaf122 Hardcode webSocketUrl to aws server in DataStreamWebApp client 2024-10-21 13:29:58 +00:00
John Colvin 1b40083008 Assert that System publishers 'doWork' is always called on same thread 2024-10-21 13:29:58 +00:00
John Colvin b381ce0154 fix: Make InternalPublisherQueue thread safe 2024-10-21 13:29:58 +00:00
John Colvin 2fef1ceb99 Improve error reporting in DataStreamWebSocketServer 2024-10-21 13:29:58 +00:00
John Colvin 08c2b2a7f2 data stream web app client check pt 2024-10-21 13:29:58 +00:00
John Colvin b983edadf0 fix build: Fix Spark build after making config init async 2024-10-21 13:29:58 +00:00
John Colvin a81c4e90a7 fix Change Configuration.initialize to be async 2024-10-21 13:29:58 +00:00
John Colvin ad3e94865c refactor: Tweak SparkCoreServices to synchronize Configuration.init globally 2024-10-21 13:29:58 +00:00
John Colvin b4979410e4 refactor: Change Configuration.initialize to be async 2024-10-21 13:29:58 +00:00