Commit Graph
14348 Commits
Author SHA1 Message Date
John Colvin 797ba111ec fix: Emplace MDConsumerHandle instead of copy assign 2025-11-03 16:19:49 -05:00
John Colvin a51b015907 cleanup: Remove unnecessary code
Trim includes as well
2025-11-03 16:19:49 -05:00
John Colvin ec057605a5 improvement: Set SHARED_MEMORY_PREFIX to an environment variable
So we can effectively define it at runtime. We don't want to have to
produce different builds just to change the shared mem prefix.
2025-11-03 16:19:49 -05:00
John Colvin e0903d1c82 fix: Always prepend proc name to shared mem prefix
Our shared memory model isn't really designed to be shared across
processes. Maybe some day it will.
2025-11-03 16:19:49 -05:00
John Colvin 1616e1a8b9 cleanup: Remove redundant macro def 2025-11-03 16:19:49 -05:00
John Colvin cfd5579703 Skip md recording thread loop if no recorders configured to run 2025-11-03 16:19:49 -05:00
John Colvin 849737b2ef cleanup: Trivial whitespace cleanup in Protobuf.cmake 2025-11-03 16:19:49 -05:00
John Colvin 85a5b72ec5 build: Clean up data_files cmake proj (support --target clean) 2025-11-03 16:19:49 -05:00
John Colvin e9ac78b438 fix build: Add ShmSegmentName.cpp to windows client sources 2025-11-03 16:19:49 -05:00
John Colvin e5288f1be2 Add api server to docker compose 2025-11-03 16:19:49 -05:00
John Colvin 50c2a57c5c Change api server ports so they don't conflict w/ ds webapp 2025-11-03 16:19:49 -05:00
John Colvin 8f293f73e3 build: Resolve app data dir and shared mem prefix at runtime
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
2025-11-03 16:19:49 -05:00
John Colvin ac42732a19 Don't log "initializing spark config" if it has already been initialized 2025-11-03 16:19:48 -05:00
John Colvin 97246e7ce7 vscode: Add debug cfg for spark api server 2025-11-03 16:19:48 -05:00
John Colvin 0815f9316d Use dependency injection in spark api server 2025-11-03 16:19:48 -05:00
John Colvin cc00ff2a0a Add SparkLoggerAdapter for Msft ILogger 2025-11-03 16:19:48 -05:00
John Colvin 28545b5099 fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2025-11-03 16:19:48 -05:00
John Colvin 04c1ad7be8 Assert that local ip address is successfully obtained 2025-11-03 16:19:48 -05:00
John Colvin 2441a5267d improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2025-11-03 16:19:48 -05:00
John Colvin 12f44f63aa Assert TcpConnection endpoint matches expected 2025-11-03 16:19:48 -05:00
John Colvin c2fe578ccb Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2025-11-03 16:19:48 -05:00
John Colvin a01fc6cc15 fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2025-11-03 16:19:48 -05:00
John Colvin d80cd9c0fd build: Upgrade packages (deps) in DataStreamTests 2025-11-03 16:19:48 -05:00
John Colvin d707847079 cleanup: Remove unused ViewStreamDatabase config 2025-11-03 16:19:48 -05:00
John Colvin 96d98a239f vscode: Add PubSubServer debug cfg to skip build 2025-11-03 16:19:48 -05:00
John Colvin 118e2a9f2e Remove reference to spark protobuf Config in ViewStream
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-03 16:19:48 -05:00
John Colvin 1481a3fd07 refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-11-03 16:19:45 -05:00
John Colvin a5e58c26ce fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-11-03 16:13:44 -05:00
John Colvin 8637bb9254 Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-03 16:13:44 -05:00
John Colvin 7c61c2ef6d cleanup: Remove unused code from ViewStreamDatabase 2025-11-03 16:13:44 -05:00
John Colvin 0668e0cd8c fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2025-11-03 16:13:44 -05:00
John Colvin 3ae88248cc cleanup: Remove unnecessary deps from csprojs 2025-11-03 16:13:43 -05:00
John Colvin 1367cea9ca fix: Catch exception when creating TcpClient 2025-11-03 16:13:43 -05:00
John Colvin b384d89820 fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-11-03 16:13:43 -05:00
John Colvin d669d24aea fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-11-03 16:13:43 -05:00
John Colvin e5bc6667e7 Reduce task wait/timeout code in AsyncTCPConnector 2025-11-03 16:13:43 -05:00
John Colvin 41512cb811 fix build warnings: Make props nullable strings in SharedStreamConfig 2025-11-03 16:13:43 -05:00
John Colvin 01d600b821 Assert that System publishers 'doWork' is always called on same thread 2025-11-03 16:13:43 -05:00
John Colvin d6197f7d1f deploy: Add pubsub and md-publisher services to compose.yaml 2025-11-03 16:13:43 -05:00
John Colvin 9e1cb14523 refactor: Model config profiles after dotnet appsettings
Namely, in DataStreamPublisher and PubSubServer. This is part of a
broader paradigm shift towards dotnet factory pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Rename Configurations/<name>.json -> appsettings.<name>.json
* Move reusable startup code into new Bts.Extensions.Hosting proj
2025-11-03 16:13:41 -05:00
John Colvin cd7c04dce5 Add ErrorChanged event to propagate DataStream exceptions 2025-11-03 16:13:19 -05:00
John Colvin 6aee76590f fix build: Fix Spark build after making config init async 2025-11-03 16:13:19 -05:00
John Colvin fc79c451b2 refactor: Change Configuration.initialize to be async
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-11-03 16:13:19 -05:00
John Colvin b916f1b836 vscode: Add 'isSparkApiEnabled' input to launch.json 2025-11-03 16:13:19 -05:00
John Colvin 826e03af94 cleanup: Remove unused PubSubConfig props 2025-11-03 16:13:19 -05:00
John Colvin 57217c9adf Capture OperationCanceledException (instead of TaskCanceledException)
The former is the base class for the latter, so this will widen the umbrella to catch a few corner cases
that would otherwise been handled by the generic "Exception" catch block. One corner case I know
of is when an op is canceled via "CancellationToken.ThrowIfCancellationRequested". In that case,
an OperationCanceledException is thrown (not a TaskCanceledException).

https://stackoverflow.com/questions/34359129/operationcanceledexception-vs-taskcanceledexception-when-task-is-canceled
2025-11-03 16:13:19 -05:00
John Colvin deae8c7fb0 Fix dsqa config json 2025-11-03 16:13:18 -05:00
John Colvin 81d3b10c2b fix: Always call 'handle_internal_pubs' in PubSubMessageReactor main loop 2025-11-03 16:13:18 -05:00
John Colvin f2cc576c51 Fix mismatched types in PubSubServerTagStatsStream schema 2025-11-03 16:13:18 -05:00
John Colvin b25038fac6 Add DataStreamPublisher service modules designed for Dependency Injection
* Change DataStreamPublisher json fields to match cs class property names verbatim
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
2025-11-03 16:13:18 -05:00