Commit Graph
12763 Commits
Author SHA1 Message Date
John Colvin 78138eff26 feat: Add DataStream WebApp
Included in DataStreamSuite.sln, but not DataStreamQueryAnalyzer.sln
2025-11-19 13:55:41 -05:00
John Colvin f7c9ee7f6a Reduce log noise in AsyncTCPConnection_AsyncWrapper 2025-11-19 13:55:41 -05:00
John Colvin 054b819eb9 TEMP changes to appsettings.json (at least i should review them before making permanent) 2025-11-19 13:55:41 -05:00
John Colvin 8787c6752b Add ColumnIndex class to resolve column names to indices
Improves performance compared to accessing columns by name
2025-11-19 13:55:41 -05:00
John Colvin 5cd390d78a Increase SQLiteEnumerator page size to 65536
This performs better, and doesn't seem to be too large in practice
2025-11-19 13:55:41 -05:00
John Colvin 8804dbe093 Serialize TimeSpan w/ microsecond precision in DataPacket json 2025-11-19 13:55:41 -05:00
John Colvin fd24abc085 Format DateTimes w/ microsecond precision in DataPacketJsonConverter 2025-11-19 13:55:41 -05:00
John Colvin dce7e73d84 Add BlueColumnTable for pandas integration
This will make it easier/more natural to support pandas column expressions
2025-11-19 13:55:41 -05:00
John Colvin 6bb142f51f Add c# Variant class, modeled after std::variant in c++ 2025-11-19 13:55:41 -05:00
John Colvin b7df16ffd7 refactor: Rename snake_case fns PascalCase 2025-11-19 13:55:41 -05:00
John Colvin 63aaeae401 refactor: Use dependency injection in DSQA 2025-11-19 13:55:08 -05:00
John Colvin ea0a1fbcd3 refactor: Load IDataStreamFactory(s) with dependency injection
So we can create ViewStreams that depend on other services, config, etc.
Previously, the PrimitiveManager relied purely on static construction of
types through reflection.
2025-11-19 13:53:18 -05:00
John Colvin 7566b2560f refactor: Change namespace to Bts.DataStreamCore for a few types
Since they're in the DataStreamCore csproj
2025-11-19 13:29:16 -05:00
John Colvin 6f2d47c630 Make 'name' default to CallerArgumentExpression for convenience 2025-11-19 13:29:16 -05:00
John Colvin 61456be744 refactor: Replace if/else with switch cases 2025-11-19 13:29:16 -05:00
John Colvin 583420b161 cleanup: Remove over-engineered subcommands
We don't need explicit subcommands to 'run onixs' md (just use config).
2025-11-19 13:29:16 -05:00
John Colvin 090aacd0c7 fix: Ensure IDGenerator instance is created before deref 2025-11-19 13:29:16 -05:00
John Colvin 79be65b94c fix: Expand undefined vars to empty string
Bugs (fixed):
  * Md shared memory file names have unexpanded variable: '${BTS_SHARED_MEMORY_PREFIX}'
2025-11-19 13:29:16 -05:00
John Colvin 70ed999116 fix: Emplace MDConsumerHandle instead of copy assign 2025-11-19 13:29:16 -05:00
John Colvin 166651a45e cleanup: Remove unnecessary code
Trim includes as well
2025-11-19 13:29:16 -05:00
John Colvin aba6adc141 fix: Always prepend proc name to shared mem prefix
Our shared memory model isn't really designed to be shared across
processes. Maybe some day it will.
2025-11-19 13:29:16 -05:00
John Colvin 9352976190 cleanup: Remove redundant macro def 2025-11-19 13:29:16 -05:00
John Colvin 1840386e62 Skip md recording thread loop if no recorders configured to run 2025-11-19 13:29:16 -05:00
John Colvin f31b887638 build: Clean up data_files cmake proj (support --target clean) 2025-11-19 13:29:15 -05:00
John Colvin 760510e4d9 fix build: Add ShmSegmentName.cpp to windows client sources 2025-11-19 13:29:15 -05:00
John Colvin 3b6e0d004b Add api server to docker compose 2025-11-19 13:29:15 -05:00
John Colvin 0b630264d2 Change api server ports so they don't conflict w/ ds webapp 2025-11-19 13:29:15 -05:00
John Colvin 733fd3e378 build: Resolve app data dir and shared mem prefix at runtime
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
2025-11-19 13:29:15 -05:00
John Colvin 010af2f175 Don't log "initializing spark config" if it has already been initialized 2025-11-19 13:29:15 -05:00
John Colvin f6b07508cd Use dependency injection in spark api server 2025-11-19 13:29:12 -05:00
John Colvin 2fb254594b Add SparkLoggerAdapter for Msft ILogger 2025-11-19 13:23:43 -05:00
John Colvin 43b6211913 fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2025-11-19 13:23:43 -05:00
John Colvin 5b793b5a0f Assert that local ip address is successfully obtained 2025-11-19 13:23:43 -05:00
John Colvin fbee781ab3 improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2025-11-19 13:23:43 -05:00
John Colvin d3df3f974a Assert TcpConnection endpoint matches expected 2025-11-19 13:23:43 -05:00
John Colvin 075619c375 Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2025-11-19 13:23:43 -05:00
John Colvin a5e6a71e0e fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2025-11-19 13:23:43 -05:00
John Colvin 64feb470da build: Upgrade packages (deps) in DataStreamTests 2025-11-19 13:23:43 -05:00
John Colvin f81dc08621 cleanup: Remove unused ViewStreamDatabase config 2025-11-19 13:23:43 -05:00
John Colvin d603adcf7a Remove reference to spark protobuf Config in ViewStream
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-19 13:23:43 -05:00
John Colvin 7932ede1e8 refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-11-19 13:23:42 -05:00
John Colvin 3e2bf37cc3 fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-11-19 13:23:42 -05:00
John Colvin 95683de1c6 Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-19 13:23:42 -05:00
John Colvin 00c8081516 cleanup: Remove unused code from ViewStreamDatabase 2025-11-19 13:23:40 -05:00
John Colvin 9b1aef6f22 fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2025-11-19 13:21:51 -05:00
John Colvin 7c5459291e cleanup: Remove unnecessary deps from csprojs 2025-11-19 13:21:51 -05:00
John Colvin 0871a15961 fix: Catch exception when creating TcpClient 2025-11-19 13:21:51 -05:00
John Colvin 1fab109987 fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-11-19 13:21:51 -05:00
John Colvin 12ea9a060a fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-11-19 13:21:51 -05:00
John Colvin a3794fdac4 Reduce task wait/timeout code in AsyncTCPConnector 2025-11-19 13:21:51 -05:00