John Colvin
78138eff26
feat: Add DataStream WebApp
...
Included in DataStreamSuite.sln, but not DataStreamQueryAnalyzer.sln
2025-11-19 13:55:41 -05:00
John Colvin
f7c9ee7f6a
Reduce log noise in AsyncTCPConnection_AsyncWrapper
2025-11-19 13:55:41 -05:00
John Colvin
054b819eb9
TEMP changes to appsettings.json (at least i should review them before making permanent)
2025-11-19 13:55:41 -05:00
John Colvin
8787c6752b
Add ColumnIndex class to resolve column names to indices
...
Improves performance compared to accessing columns by name
2025-11-19 13:55:41 -05:00
John Colvin
5cd390d78a
Increase SQLiteEnumerator page size to 65536
...
This performs better, and doesn't seem to be too large in practice
2025-11-19 13:55:41 -05:00
John Colvin
8804dbe093
Serialize TimeSpan w/ microsecond precision in DataPacket json
2025-11-19 13:55:41 -05:00
John Colvin
fd24abc085
Format DateTimes w/ microsecond precision in DataPacketJsonConverter
2025-11-19 13:55:41 -05:00
John Colvin
dce7e73d84
Add BlueColumnTable for pandas integration
...
This will make it easier/more natural to support pandas column expressions
2025-11-19 13:55:41 -05:00
John Colvin
6bb142f51f
Add c# Variant class, modeled after std::variant in c++
2025-11-19 13:55:41 -05:00
John Colvin
b7df16ffd7
refactor: Rename snake_case fns PascalCase
2025-11-19 13:55:41 -05:00
John Colvin
63aaeae401
refactor: Use dependency injection in DSQA
2025-11-19 13:55:08 -05:00
John Colvin
ea0a1fbcd3
refactor: Load IDataStreamFactory(s) with dependency injection
...
So we can create ViewStreams that depend on other services, config, etc.
Previously, the PrimitiveManager relied purely on static construction of
types through reflection.
2025-11-19 13:53:18 -05:00
John Colvin
7566b2560f
refactor: Change namespace to Bts.DataStreamCore for a few types
...
Since they're in the DataStreamCore csproj
2025-11-19 13:29:16 -05:00
John Colvin
6f2d47c630
Make 'name' default to CallerArgumentExpression for convenience
2025-11-19 13:29:16 -05:00
John Colvin
61456be744
refactor: Replace if/else with switch cases
2025-11-19 13:29:16 -05:00
John Colvin
583420b161
cleanup: Remove over-engineered subcommands
...
We don't need explicit subcommands to 'run onixs' md (just use config).
2025-11-19 13:29:16 -05:00
John Colvin
090aacd0c7
fix: Ensure IDGenerator instance is created before deref
2025-11-19 13:29:16 -05:00
John Colvin
79be65b94c
fix: Expand undefined vars to empty string
...
Bugs (fixed):
* Md shared memory file names have unexpanded variable: '${BTS_SHARED_MEMORY_PREFIX}'
2025-11-19 13:29:16 -05:00
John Colvin
70ed999116
fix: Emplace MDConsumerHandle instead of copy assign
2025-11-19 13:29:16 -05:00
John Colvin
166651a45e
cleanup: Remove unnecessary code
...
Trim includes as well
2025-11-19 13:29:16 -05:00
John Colvin
aba6adc141
fix: Always prepend proc name to shared mem prefix
...
Our shared memory model isn't really designed to be shared across
processes. Maybe some day it will.
2025-11-19 13:29:16 -05:00
John Colvin
9352976190
cleanup: Remove redundant macro def
2025-11-19 13:29:16 -05:00
John Colvin
1840386e62
Skip md recording thread loop if no recorders configured to run
2025-11-19 13:29:16 -05:00
John Colvin
f31b887638
build: Clean up data_files cmake proj (support --target clean)
2025-11-19 13:29:15 -05:00
John Colvin
760510e4d9
fix build: Add ShmSegmentName.cpp to windows client sources
2025-11-19 13:29:15 -05:00
John Colvin
3b6e0d004b
Add api server to docker compose
2025-11-19 13:29:15 -05:00
John Colvin
0b630264d2
Change api server ports so they don't conflict w/ ds webapp
2025-11-19 13:29:15 -05:00
John Colvin
733fd3e378
build: Resolve app data dir and shared mem prefix at runtime
...
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
2025-11-19 13:29:15 -05:00
John Colvin
010af2f175
Don't log "initializing spark config" if it has already been initialized
2025-11-19 13:29:15 -05:00
John Colvin
f6b07508cd
Use dependency injection in spark api server
2025-11-19 13:29:12 -05:00
John Colvin
2fb254594b
Add SparkLoggerAdapter for Msft ILogger
2025-11-19 13:23:43 -05:00
John Colvin
43b6211913
fix: Don't assert on exception in BooleanExpression.evaluate
...
Bugs (fixed):
* DataStream WebApp Server sometimes crashes when the user enters a
bad query.
2025-11-19 13:23:43 -05:00
John Colvin
5b793b5a0f
Assert that local ip address is successfully obtained
2025-11-19 13:23:43 -05:00
John Colvin
fbee781ab3
improvement: Resolve hostnames when comparing ip addrs
...
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2025-11-19 13:23:43 -05:00
John Colvin
d3df3f974a
Assert TcpConnection endpoint matches expected
2025-11-19 13:23:43 -05:00
John Colvin
075619c375
Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
...
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2025-11-19 13:23:43 -05:00
John Colvin
a5e6a71e0e
fix: Format PubSubRoute as json in log msgs
...
It was previously using the default ToString, which simply returns the type name
2025-11-19 13:23:43 -05:00
John Colvin
64feb470da
build: Upgrade packages (deps) in DataStreamTests
2025-11-19 13:23:43 -05:00
John Colvin
f81dc08621
cleanup: Remove unused ViewStreamDatabase config
2025-11-19 13:23:43 -05:00
John Colvin
d603adcf7a
Remove reference to spark protobuf Config in ViewStream
...
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-19 13:23:43 -05:00
John Colvin
7932ede1e8
refactor: Run RemoteQueryManager as an IHostedService
...
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-11-19 13:23:42 -05:00
John Colvin
3e2bf37cc3
fix: Ensure spark config init and login are only performed once
...
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.
It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-11-19 13:23:42 -05:00
John Colvin
95683de1c6
Assert ViewStreamDatabase.instance is only created once
...
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-19 13:23:42 -05:00
John Colvin
00c8081516
cleanup: Remove unused code from ViewStreamDatabase
2025-11-19 13:23:40 -05:00
John Colvin
9b1aef6f22
fix: Define BoooleanConstant before StringConstant
...
Otherwise the former masks the latter and prevents it from matching
2025-11-19 13:21:51 -05:00
John Colvin
7c5459291e
cleanup: Remove unnecessary deps from csprojs
2025-11-19 13:21:51 -05:00
John Colvin
0871a15961
fix: Catch exception when creating TcpClient
2025-11-19 13:21:51 -05:00
John Colvin
1fab109987
fix: Make SystemSubscriptionMapPublisher threadsafe
...
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-11-19 13:21:51 -05:00
John Colvin
12ea9a060a
fix: Add a few null checks
...
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-11-19 13:21:51 -05:00
John Colvin
a3794fdac4
Reduce task wait/timeout code in AsyncTCPConnector
2025-11-19 13:21:51 -05:00