John Colvin
ea0a1fbcd3
refactor: Load IDataStreamFactory(s) with dependency injection
...
So we can create ViewStreams that depend on other services, config, etc.
Previously, the PrimitiveManager relied purely on static construction of
types through reflection.
2025-11-19 13:53:18 -05:00
John Colvin
7566b2560f
refactor: Change namespace to Bts.DataStreamCore for a few types
...
Since they're in the DataStreamCore csproj
2025-11-19 13:29:16 -05:00
John Colvin
6f2d47c630
Make 'name' default to CallerArgumentExpression for convenience
2025-11-19 13:29:16 -05:00
John Colvin
61456be744
refactor: Replace if/else with switch cases
2025-11-19 13:29:16 -05:00
John Colvin
583420b161
cleanup: Remove over-engineered subcommands
...
We don't need explicit subcommands to 'run onixs' md (just use config).
2025-11-19 13:29:16 -05:00
John Colvin
090aacd0c7
fix: Ensure IDGenerator instance is created before deref
2025-11-19 13:29:16 -05:00
John Colvin
79be65b94c
fix: Expand undefined vars to empty string
...
Bugs (fixed):
* Md shared memory file names have unexpanded variable: '${BTS_SHARED_MEMORY_PREFIX}'
2025-11-19 13:29:16 -05:00
John Colvin
70ed999116
fix: Emplace MDConsumerHandle instead of copy assign
2025-11-19 13:29:16 -05:00
John Colvin
166651a45e
cleanup: Remove unnecessary code
...
Trim includes as well
2025-11-19 13:29:16 -05:00
John Colvin
79c9dd8221
improvement: Set SHARED_MEMORY_PREFIX to an environment variable
...
So we can effectively define it at runtime. We don't want to have to
produce different builds just to change the shared mem prefix.
2025-11-19 13:29:16 -05:00
John Colvin
aba6adc141
fix: Always prepend proc name to shared mem prefix
...
Our shared memory model isn't really designed to be shared across
processes. Maybe some day it will.
2025-11-19 13:29:16 -05:00
John Colvin
9352976190
cleanup: Remove redundant macro def
2025-11-19 13:29:16 -05:00
John Colvin
1840386e62
Skip md recording thread loop if no recorders configured to run
2025-11-19 13:29:16 -05:00
John Colvin
247251a52a
cleanup: Trivial whitespace cleanup in Protobuf.cmake
2025-11-19 13:29:15 -05:00
John Colvin
f31b887638
build: Clean up data_files cmake proj (support --target clean)
2025-11-19 13:29:15 -05:00
John Colvin
760510e4d9
fix build: Add ShmSegmentName.cpp to windows client sources
2025-11-19 13:29:15 -05:00
John Colvin
3b6e0d004b
Add api server to docker compose
2025-11-19 13:29:15 -05:00
John Colvin
0b630264d2
Change api server ports so they don't conflict w/ ds webapp
2025-11-19 13:29:15 -05:00
John Colvin
733fd3e378
build: Resolve app data dir and shared mem prefix at runtime
...
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
2025-11-19 13:29:15 -05:00
John Colvin
010af2f175
Don't log "initializing spark config" if it has already been initialized
2025-11-19 13:29:15 -05:00
John Colvin
e2127bbe94
vscode: Add debug cfg for spark api server
2025-11-19 13:29:15 -05:00
John Colvin
f6b07508cd
Use dependency injection in spark api server
2025-11-19 13:29:12 -05:00
John Colvin
2fb254594b
Add SparkLoggerAdapter for Msft ILogger
2025-11-19 13:23:43 -05:00
John Colvin
43b6211913
fix: Don't assert on exception in BooleanExpression.evaluate
...
Bugs (fixed):
* DataStream WebApp Server sometimes crashes when the user enters a
bad query.
2025-11-19 13:23:43 -05:00
John Colvin
5b793b5a0f
Assert that local ip address is successfully obtained
2025-11-19 13:23:43 -05:00
John Colvin
fbee781ab3
improvement: Resolve hostnames when comparing ip addrs
...
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2025-11-19 13:23:43 -05:00
John Colvin
d3df3f974a
Assert TcpConnection endpoint matches expected
2025-11-19 13:23:43 -05:00
John Colvin
075619c375
Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
...
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2025-11-19 13:23:43 -05:00
John Colvin
a5e6a71e0e
fix: Format PubSubRoute as json in log msgs
...
It was previously using the default ToString, which simply returns the type name
2025-11-19 13:23:43 -05:00
John Colvin
64feb470da
build: Upgrade packages (deps) in DataStreamTests
2025-11-19 13:23:43 -05:00
John Colvin
f81dc08621
cleanup: Remove unused ViewStreamDatabase config
2025-11-19 13:23:43 -05:00
John Colvin
9ff23db7a2
vscode: Add PubSubServer debug cfg to skip build
2025-11-19 13:23:43 -05:00
John Colvin
d603adcf7a
Remove reference to spark protobuf Config in ViewStream
...
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-19 13:23:43 -05:00
John Colvin
7932ede1e8
refactor: Run RemoteQueryManager as an IHostedService
...
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-11-19 13:23:42 -05:00
John Colvin
3e2bf37cc3
fix: Ensure spark config init and login are only performed once
...
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.
It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-11-19 13:23:42 -05:00
John Colvin
95683de1c6
Assert ViewStreamDatabase.instance is only created once
...
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-19 13:23:42 -05:00
John Colvin
00c8081516
cleanup: Remove unused code from ViewStreamDatabase
2025-11-19 13:23:40 -05:00
John Colvin
9b1aef6f22
fix: Define BoooleanConstant before StringConstant
...
Otherwise the former masks the latter and prevents it from matching
2025-11-19 13:21:51 -05:00
John Colvin
7c5459291e
cleanup: Remove unnecessary deps from csprojs
2025-11-19 13:21:51 -05:00
John Colvin
0871a15961
fix: Catch exception when creating TcpClient
2025-11-19 13:21:51 -05:00
John Colvin
1fab109987
fix: Make SystemSubscriptionMapPublisher threadsafe
...
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-11-19 13:21:51 -05:00
John Colvin
12ea9a060a
fix: Add a few null checks
...
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-11-19 13:21:51 -05:00
John Colvin
a3794fdac4
Reduce task wait/timeout code in AsyncTCPConnector
2025-11-19 13:21:51 -05:00
John Colvin
4e7eab8e47
fix build warnings: Make props nullable strings in SharedStreamConfig
2025-11-19 13:21:51 -05:00
John Colvin
754dfa87c7
Assert that System publishers 'doWork' is always called on same thread
2025-11-19 13:21:51 -05:00
John Colvin
ef51954549
deploy: Add pubsub and md-publisher services to compose.yaml
2025-11-19 13:21:50 -05:00
John Colvin
50a1dfae14
refactor: Model config profiles after dotnet appsettings
...
Namely, in DataStreamPublisher and PubSubServer. This is part of a
broader paradigm shift towards dotnet factory pattern.
https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines
* Rename Configurations/<name>.json -> appsettings.<name>.json
* Move reusable startup code into new Bts.Extensions.Hosting proj
2025-11-19 13:21:50 -05:00
John Colvin
6e09854337
Add ErrorChanged event to propagate DataStream exceptions
2025-11-19 13:21:50 -05:00
John Colvin
df049622cb
fix build: Fix Spark build after making config init async
2025-11-19 13:21:48 -05:00
John Colvin
0f0420ba4c
refactor: Change Configuration.initialize to be async
...
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-11-19 13:18:10 -05:00