Commit Graph
14339 Commits
Author SHA1 Message Date
John Colvin 1c579ee063 refactor: Change namespace to Bts.DataStreamCore for a few types
Since they're in the DataStreamCore csproj
2026-01-04 21:08:12 +00:00
John Colvin 7508fa0b7f dev qol: Mark ExceptionAssert 'name' param w/ CallerArgumentExpression 2026-01-04 21:08:12 +00:00
John Colvin 1f18ec4198 refactor: Replace if/else with switch cases 2026-01-04 21:08:12 +00:00
John Colvin 81d7820cca fix: Ensure IDGenerator instance is created before deref 2026-01-04 21:08:12 +00:00
John Colvin e125be1f85 fix: Emplace MDConsumerHandle instead of copy assign 2026-01-04 21:08:12 +00:00
John Colvin 040c8cab59 cleanup: Remove unnecessary code
Trim includes as well
2026-01-04 21:08:12 +00:00
John Colvin 32a12cf1f7 cleanup: Remove redundant macro def 2026-01-04 21:08:12 +00:00
John Colvin db604fbed0 improve: Abort md rec loop if nothing configured to record 2026-01-04 21:08:12 +00:00
John Colvin cc0a3ea15f cleanup: Trivial whitespace cleanup in Protobuf.cmake 2026-01-04 21:08:12 +00:00
John Colvin 4066cd50ce build: Clean up data_files cmake proj (support --target clean) 2026-01-04 21:08:12 +00:00
John Colvin f1ccff9134 build: Resolve app data dir and shared mem prefix at runtime
We can't define them at compile time anymore since we build multiple exes that
depend on libspark in the same dir (e.g., api-server, pubsub, sparkd, etc.),
and each one needs its own shared memory and app data filespaces.

For shared memory:
  * Prefix all shm files with the proc name and an environment variable,
    BTS_SHARED_MEMORY_PREFIX (if defined). The env var is only necessary
    when 2 exe's with the same name run simultaneously.

For app data:
  * On linux, the default is:
    /home/<user_name>/.local/share/bts/<proc_name>
  * On windows, the default is:
    /User/<user_name>/AppData/Roaming/bts/<proc_name>
  * This path can be overridden in the environment variable:
    <proc_name>__APPDATA
    - For example, SPARKD__APPDATA, PUBSUB__APPDATA, etc.
2026-01-04 21:08:12 +00:00
John Colvin 51ab182b2f Don't log "initializing spark config" if it has already been initialized 2026-01-04 21:08:12 +00:00
John Colvin 856cd552eb refactor: Add Api Server service modules for Dependency Injection
* Part of a broader paradigm shift towards Dependency Injection and
  Factory Pattern.
* Adds "api-server" service to docker compose
* Replaces MarketDataClient w/ IMarketDataClient in a few streams
* Replaces ViewStreamDatabase w/ IWorkScheduler in a few streams
2026-01-04 21:08:12 +00:00
John Colvin 997647e917 refactor: Add handy WebsocketExtensions
* IsSocketStateOpen
* IsClosedProperly
2026-01-04 21:08:12 +00:00
John Colvin 687a9741db fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2026-01-04 21:08:12 +00:00
John Colvin 7faa0ade24 diag: Assert that local ip address is successfully obtained 2026-01-04 21:08:12 +00:00
John Colvin 4d6d68aff6 improve: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2026-01-04 21:08:12 +00:00
John Colvin dcf291d6b2 diag: Assert TcpConnection endpoint matches expected 2026-01-04 21:08:12 +00:00
John Colvin a6b83988e7 diag: Catch exceptions on AsyncTCPConnector_AsyncWrapper bg thread
Since it spawns a new thread on each attempt, uncaught exceptions were
previously undetected, and caused silent failures down the pipe. This
should make it easier to track such bugs down.
2026-01-04 21:08:12 +00:00
John Colvin 2c097500f4 fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2026-01-04 21:08:12 +00:00
John Colvin 81a212e7a3 build: Upgrade packages (deps) in DataStreamTests 2026-01-04 21:08:12 +00:00
John Colvin 449453625c vscode: Add PubSubServer debug cfg to skip build 2026-01-04 21:08:12 +00:00
John Colvin 5058187d5b cleanup: Remove reference to spark protobuf Config in ViewStream
In general, we should keep BlueOrb dependencies on spark/sparkd to a minimum.
The dotnet dependency injection architecture provides a robust config framework
we can use instead, based on json files, environment variables, and command-line
arguments.
2026-01-04 21:08:12 +00:00
John Colvin 0253a0f60f refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2026-01-04 21:08:12 +00:00
John Colvin 6986cb5e8e fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2026-01-04 21:08:12 +00:00
John Colvin 35a45c35ba refactor: Move position model to spark_api
So we can use it in the new sdk (c# server-side api) for Ophil
2026-01-04 21:08:12 +00:00
John Colvin 970555d2a1 diag: Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2026-01-04 21:08:12 +00:00
John Colvin 1d46ea061a cleanup: Remove unused code from ViewStreamDatabase 2026-01-04 21:08:12 +00:00
John Colvin 4bd12dc465 fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2026-01-04 21:08:12 +00:00
John Colvin ad1d404314 cleanup: Remove unnecessary deps from csprojs 2026-01-04 21:08:12 +00:00
John Colvin a7b6da27ae fix: Catch exception when creating TcpClient 2026-01-04 21:08:12 +00:00
John Colvin de4a55b34c fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2026-01-04 21:08:12 +00:00
John Colvin 2e14598667 fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2026-01-04 21:08:12 +00:00
John Colvin 3038d9154f improve: Reduce task wait/timeout code in AsyncTCPConnector 2026-01-04 21:08:12 +00:00
John Colvin 5805758187 diag: Assert System publishers 'doWork' is always called on same thread 2026-01-04 21:08:12 +00:00
John Colvin 2c1b7b95a4 improve: Propagate DataStream exceptions via ErrorChanged event 2026-01-04 21:08:12 +00:00
John Colvin 60af47c4eb cleanup: Remove unused PubSubConfig props 2026-01-04 21:08:12 +00:00
John Colvin b0d0594e3d fix: Always call 'handle_internal_pubs' in PubSubMessageReactor main loop 2026-01-04 21:08:12 +00:00
John Colvin b29b2a0adb fix: Fix a few column types in in PubSubServerTagStatsStream
* total_bytes      (int      -> long)
* last_update_time (DateTime -> long)
2026-01-04 21:08:11 +00:00
John Colvin 76fa35382e refactor: Add DataStream service modules for Dependency Injection
Namely, in DataStreamPublisher and PubSubServer. This is part of a broader
paradigm shift towards Dependency Injection and Factory Pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Use interfaces to decouple services:
  - Decouple ManagedSubscriber from ISubscriberCore impl
  - Decouple PubTableBroker from IPublisherCore impl
  - Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl

* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
* Model config profiles after dotnet appsettings
  - Rename Configurations/<name>.json -> appsettings.<name>.json
  - Make json fields match corresponding csharp props verbatim
    * The Dependency Injection framework doesn't support JsonPropertyName
      attributes. Config class props must match json fields verbatim.

* Move reusable startup code into new Bts.Extensions.Hosting proj
* Add pubsub and md-publisher services to compose.yaml
2026-01-04 21:08:11 +00:00
John Colvin 716bc92394 refactor: Only support 1 tag type in IPubSubServerTagHandler 2025-12-30 11:42:23 -05:00
John Colvin 3a6141f3ae refactor: Change Configuration.initialize to be async
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-12-30 11:42:20 -05:00
John Colvin be2368f95a refactor: Add Spark service modules for Dependency Injection 2025-12-30 11:38:21 -05:00
John Colvin 7e32ac9e06 cleanup: Remove unused pubsub tag handler code
* Removed 'IPubSubServerTagHandler.register'
  - All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
  - This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
  - No reason these interfaces have to be tied together
2025-12-30 11:38:21 -05:00
John Colvin 8fc4df1427 refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
This matches the file name
2025-12-30 11:38:21 -05:00
John Colvin 2020ebed2d cleanup: Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
In config json files (not actual code change)
2025-12-30 11:38:20 -05:00
John Colvin cc483ca4b5 diag: Fix typos in log msgs
* Bad string interopolation syntax
2025-12-30 11:38:20 -05:00
John Colvin aae7356dc7 vscode: Add DataStreamPublisher debug cfg to skip build 2025-12-30 11:38:20 -05:00
John Colvin 8d6e94384c cleanup: Fix benign typos in build scripts
Change BUILD_SPARK_PY var name to BUILD_PY (these scripts build other
projects than 'spark').
2025-12-30 11:38:20 -05:00
John Colvin 7d6e4efa4f cleanup: Fix numerous build warnings in DataStreamSuite
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.

* Compression related files
  - BZip2InputStream  (disable nullability warnings)
  - BZip2OutputStream (disable nullability warnings)
  - DeflaterHuffman   (disable nullability warnings)
  - Inflater          (disable nullability warnings)
  - InflaterDynHeader (disable nullability warnings)
  - ParallelJoinTest  (disable nullability warnings)
  - TarHeader
  - TarInputStream    (disable nullability warnings)
  - ZipEntry
  - ZipFile
  - ZipInputStream

* AsyncTCPConnection_APM
* AsyncTCPConnection_APM
* InflaterHuffmanTree
* ManagedSubscriber
* ManagedTCPMessageHub
* MapStream
* PubSubClientMessage
  - Normalize indentation/whitespace
  - Fix some typos
  - Address nullability issues to fix build warnings
  - Modernize code
* PubSubMessageReactor
* PubSubServerManagedTCPConnection
* PubSubSingleTagStateEvent
* PubSubState
* PublisherCore
* PubsubDeserializationController
* RemoteMessage
  - make 'message' nullable to resolve warnings elsewhere (in calling code)
* SharedStreamConfig
* SingleTagData
* SubscriberCore
* SuperMapStream
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SystemSubscriptionMapPublisher
* ThreadSafeQueue
* TradeBarsStream

Only use Registry on windows.
2025-12-30 11:38:20 -05:00