Commit Graph
12759 Commits
Author SHA1 Message Date
John Colvin 3dcde6d089 refactor: Remove spark api server dependency on SecurityMD
This is part of a larger effort to decouple the api server from spark grpc
(protobuf) services
2025-12-05 15:56:23 +00:00
John Colvin e0e9989eda refactor: Move ObservableViewStream into DataStreamCore 2025-12-05 15:56:23 +00:00
John Colvin 5f4a811598 feat: Add ParquetReader/Writer streams 2025-12-05 15:56:23 +00:00
John Colvin 5fb64e114c refactor: Hide md client behind interface, create via dep injection 2025-12-05 15:56:21 +00:00
John Colvin 0571429375 refactor: Decouple ViewStreamDatabase from TableStream, ISubscriberCore 2025-12-05 15:54:03 +00:00
John Colvin 8d10d512dd cleanup: Drop ".exe" from log dir name on windows 2025-12-05 15:54:03 +00:00
John Colvin 7d90f8c664 feat: Add SpreadBook stream 2025-12-05 15:54:03 +00:00
John Colvin 5412a90c2e fix: Check settings Subject.IsDisposed before using 2025-12-05 15:54:03 +00:00
John Colvin 50b4cf7556 ui (launcher): Move "X" button to left of report menu items
This is a more natural location. Also, it looked weird on the right because the
menu labels aren't fixed width, so the buttons weren't vertically aligned.
2025-12-05 15:54:03 +00:00
John Colvin 247dcf7d31 refactor: Use HostBuilder to launch spark w/ streams 2025-12-05 15:54:03 +00:00
John Colvin 05d2a54e8a build: Update System.Composition pkg refs to 8.0.0 2025-12-05 15:54:03 +00:00
John Colvin 6b27106f44 cleanup: Mark a few classes "internal" in Spark
* BookmarkShellModule
* TabDocument
  - In xaml only (it was already marked internal in xaml.cs)
* WindowManager
* WorkspaceViewModel
* WorkspacesDialogView
* WorkspacesDialogViewModel
* WorkspacesModel
2025-12-05 15:54:03 +00:00
John Colvin aadce1c989 refactor: Consolidate ShellBootstrapper and base class into one 2025-12-05 15:54:03 +00:00
John Colvin b532836d9c feat: Add Stream Reports shell window 2025-12-05 15:54:03 +00:00
John Colvin 3135778120 build: Define custom "main" (entry point) for Spark exe
This is a prelude to integrating streams into spark. The plan is to use
an ms HostBuilder to instantiate stream services/dependencies alongside
the spark Application object.
2025-12-05 15:54:03 +00:00
John Colvin 26e500368f cleanup: Get rid of stunnel code
This looks like it hasn't been hooked up for years...it would only start
if a stunnel.conf file was detected, which hasn't been around for a very
long time.
2025-12-05 15:54:03 +00:00
John Colvin 8957ef6230 fix: Only create mutex if spark app is NOT running in vs designer
I accidentally droped the '!' in a cleanup commit 4 years ago
2025-12-05 15:54:03 +00:00
John Colvin 982369fcc3 fix (dsqa): Resolve name conflicts with new BlueOrb streams 2025-12-05 15:54:03 +00:00
John Colvin d1186a81f8 fix: Fix typo in "DateTimeExpansionStream" type name
Previously, it was "DateTimeExpanstion" (erroneous "t")
2025-12-05 15:54:03 +00:00
John Colvin 8a5e4048f8 cleanup: Remove "partial" class modifiers from OrderBook vm classes
They simply weren't necessary.
2025-12-05 15:54:03 +00:00
John Colvin 4b1f31f2b8 diag: Reduce log noise in AsyncTCPConnection_AsyncWrapper 2025-12-05 15:54:03 +00:00
John Colvin 3636d1c785 refactor: Rename snake_case fns PascalCase 2025-12-05 15:54:03 +00:00
John Colvin 833101f026 refactor: Use dependency injection in DSQA 2025-12-05 15:54:03 +00:00
John Colvin f5ecbe8c80 refactor: Load IDataStreamFactory(s) with dependency injection
So we can create ViewStreams that depend on other services, config, etc.
Previously, the PrimitiveManager relied purely on static construction of
types through reflection.
2025-12-05 15:54:01 +00:00
John Colvin c839652ea3 refactor: Make CircularBufferTable immutable 2025-12-05 15:53:32 +00:00
John Colvin f39e661602 refactor: Change namespace to Bts.DataStreamCore for a few types
Since they're in the DataStreamCore csproj
2025-12-05 15:53:32 +00:00
John Colvin a9d55392c2 dev qol: Mark ExceptionAssert 'name' param w/ CallerArgumentExpression 2025-12-05 15:53:32 +00:00
John Colvin 55c0ffbe13 refactor: Replace if/else with switch cases 2025-12-05 15:53:32 +00:00
John Colvin 284784df4d fix: Ensure IDGenerator instance is created before deref 2025-12-05 15:53:32 +00:00
John Colvin d67b9ec0a8 fix: Emplace MDConsumerHandle instead of copy assign 2025-12-05 15:53:32 +00:00
John Colvin ad69f58d5f cleanup: Remove unnecessary code
Trim includes as well
2025-12-05 15:53:32 +00:00
John Colvin ad7fafd35d cleanup: Remove redundant macro def 2025-12-05 15:53:32 +00:00
John Colvin 0e84e388dc improvement: Abort md rec loop if nothing configured to record 2025-12-05 15:53:32 +00:00
John Colvin d5911a7657 build: Clean up data_files cmake proj (support --target clean) 2025-12-05 15:53:32 +00:00
John Colvin 96e7c5102c Add api server to docker compose 2025-12-05 15:53:32 +00:00
John Colvin ea21c3e7ff build: Resolve app data dir and shared mem prefix at runtime
We can't define them at compile time anymore since we build multiple exes that
depend on libspark in the same dir (e.g., api-server, pubsub, sparkd, etc.),
and each one needs its own shared memory and app data filespaces.

For shared memory:
  * Prefix all shm files with the proc name and an environment variable,
    BTS_SHARED_MEMORY_PREFIX (if defined). The env var is only necessary
    when 2 exe's with the same name run simultaneously.

For app data:
  * On linux, the default is:
    /home/<user_name>/.local/share/bts/<proc_name>
  * On windows, the default is:
    /User/<user_name>/AppData/Roaming/bts/<proc_name>
  * This path can be overridden in the environment variable:
    <proc_name>__APPDATA
    - For example, SPARKD__APPDATA, PUBSUB__APPDATA, etc.
2025-12-05 15:53:32 +00:00
John Colvin 26853d597a Don't log "initializing spark config" if it has already been initialized 2025-12-05 15:53:32 +00:00
John Colvin 87d26c076b refactor: Use dependency injection in spark api server 2025-12-05 15:53:32 +00:00
John Colvin f98208a8a6 refactor: Add handy WebsocketExtensions
* IsSocketStateOpen
* IsClosedProperly
2025-12-05 15:53:32 +00:00
John Colvin afd107402a fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2025-12-05 15:53:32 +00:00
John Colvin d628f38bb1 diag: Assert that local ip address is successfully obtained 2025-12-05 15:53:32 +00:00
John Colvin 4ade38ef9a improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2025-12-05 15:53:32 +00:00
John Colvin 0f9ea5139c diag: Assert TcpConnection endpoint matches expected 2025-12-05 15:53:32 +00:00
John Colvin 789f74e3f6 diag: Catch exceptions on AsyncTCPConnector_AsyncWrapper bg thread
Since it spawns a new thread on each attempt, uncaught exceptions were
previously undetected, and caused silent failures down the pipe. This
should make it easier to track such bugs down.
2025-12-05 15:53:32 +00:00
John Colvin 75d95080b7 fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2025-12-05 15:53:32 +00:00
John Colvin 9287a57be3 build: Upgrade packages (deps) in DataStreamTests 2025-12-05 15:53:32 +00:00
John Colvin cb20975417 cleanup: Remove unused ViewStreamDatabase config 2025-12-05 15:53:32 +00:00
John Colvin 25b13d9dff cleanup: Remove reference to spark protobuf Config in ViewStream
In general, we should keep BlueOrb dependencies on spark/sparkd to a minimum.
The dotnet dependency injection architecture provides a robust config framework
we can use instead, based on json files, environment variables, and command-line
arguments.
2025-12-05 15:53:32 +00:00
John Colvin d7b0296fbe refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-12-05 15:53:32 +00:00
John Colvin 69b1ac243e fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-12-05 15:53:31 +00:00