Commit Graph
12726 Commits
Author SHA1 Message Date
John Colvin d5911a7657 build: Clean up data_files cmake proj (support --target clean) 2025-12-05 15:53:32 +00:00
John Colvin 96e7c5102c Add api server to docker compose 2025-12-05 15:53:32 +00:00
John Colvin ea21c3e7ff build: Resolve app data dir and shared mem prefix at runtime
We can't define them at compile time anymore since we build multiple exes that
depend on libspark in the same dir (e.g., api-server, pubsub, sparkd, etc.),
and each one needs its own shared memory and app data filespaces.

For shared memory:
  * Prefix all shm files with the proc name and an environment variable,
    BTS_SHARED_MEMORY_PREFIX (if defined). The env var is only necessary
    when 2 exe's with the same name run simultaneously.

For app data:
  * On linux, the default is:
    /home/<user_name>/.local/share/bts/<proc_name>
  * On windows, the default is:
    /User/<user_name>/AppData/Roaming/bts/<proc_name>
  * This path can be overridden in the environment variable:
    <proc_name>__APPDATA
    - For example, SPARKD__APPDATA, PUBSUB__APPDATA, etc.
2025-12-05 15:53:32 +00:00
John Colvin 26853d597a Don't log "initializing spark config" if it has already been initialized 2025-12-05 15:53:32 +00:00
John Colvin 87d26c076b refactor: Use dependency injection in spark api server 2025-12-05 15:53:32 +00:00
John Colvin f98208a8a6 refactor: Add handy WebsocketExtensions
* IsSocketStateOpen
* IsClosedProperly
2025-12-05 15:53:32 +00:00
John Colvin afd107402a fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2025-12-05 15:53:32 +00:00
John Colvin d628f38bb1 diag: Assert that local ip address is successfully obtained 2025-12-05 15:53:32 +00:00
John Colvin 4ade38ef9a improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2025-12-05 15:53:32 +00:00
John Colvin 0f9ea5139c diag: Assert TcpConnection endpoint matches expected 2025-12-05 15:53:32 +00:00
John Colvin 789f74e3f6 diag: Catch exceptions on AsyncTCPConnector_AsyncWrapper bg thread
Since it spawns a new thread on each attempt, uncaught exceptions were
previously undetected, and caused silent failures down the pipe. This
should make it easier to track such bugs down.
2025-12-05 15:53:32 +00:00
John Colvin 75d95080b7 fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2025-12-05 15:53:32 +00:00
John Colvin 9287a57be3 build: Upgrade packages (deps) in DataStreamTests 2025-12-05 15:53:32 +00:00
John Colvin cb20975417 cleanup: Remove unused ViewStreamDatabase config 2025-12-05 15:53:32 +00:00
John Colvin 25b13d9dff cleanup: Remove reference to spark protobuf Config in ViewStream
In general, we should keep BlueOrb dependencies on spark/sparkd to a minimum.
The dotnet dependency injection architecture provides a robust config framework
we can use instead, based on json files, environment variables, and command-line
arguments.
2025-12-05 15:53:32 +00:00
John Colvin d7b0296fbe refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-12-05 15:53:32 +00:00
John Colvin 69b1ac243e fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-12-05 15:53:31 +00:00
John Colvin f3f2e5834c diag: Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-12-05 15:53:31 +00:00
John Colvin d93946ce61 cleanup: Remove unused code from ViewStreamDatabase 2025-12-05 15:53:31 +00:00
John Colvin 99e50f539f fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2025-12-05 15:53:31 +00:00
John Colvin 504d87ae14 cleanup: Remove unnecessary deps from csprojs 2025-12-05 15:53:31 +00:00
John Colvin a8fc2a6179 fix: Catch exception when creating TcpClient 2025-12-05 15:53:31 +00:00
John Colvin 061f323026 fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-12-05 15:53:31 +00:00
John Colvin 60488a4770 fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-12-05 15:53:31 +00:00
John Colvin 5f517e4a7c improvement: Reduce task wait/timeout code in AsyncTCPConnector 2025-12-05 15:53:31 +00:00
John Colvin a04fb84656 diag: Assert System publishers 'doWork' is always called on same thread 2025-12-05 15:53:31 +00:00
John Colvin 8bd4a5131d deploy: Add pubsub and md-publisher services to compose.yaml 2025-12-05 15:53:31 +00:00
John Colvin 95c4b6b036 refactor: Model config profiles after dotnet appsettings
Namely, in DataStreamPublisher and PubSubServer. This is part of a
broader paradigm shift towards dotnet factory pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Rename Configurations/<name>.json -> appsettings.<name>.json
* Move reusable startup code into new Bts.Extensions.Hosting proj
2025-12-05 15:53:31 +00:00
John Colvin d2300fbbb9 improvement: Propagate DataStream exceptions via ErrorChanged event 2025-11-30 16:44:31 +00:00
John Colvin 12c21f12fe refactor: Change Configuration.initialize to be async
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-11-30 16:44:31 +00:00
John Colvin 5061a1bbd7 cleanup: Remove unused PubSubConfig props 2025-11-30 16:44:31 +00:00
John Colvin f6aec08919 fix: Always call 'handle_internal_pubs' in PubSubMessageReactor main loop 2025-11-30 16:44:31 +00:00
John Colvin 652f0e0043 fix: Fix a few column types in in PubSubServerTagStatsStream
* total_bytes      (int      -> long)
* last_update_time (DateTime -> long)
2025-11-30 16:44:31 +00:00
John Colvin 6889a971ff refactor: Add DataStreamPublisher service modules for Dependency Injection
* Change DataStreamPublisher json fields to match cs class property names verbatim
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
2025-11-30 16:44:31 +00:00
John Colvin c5f4d146f3 refactor: Add PubSub service modules for Dependency Injection
* Get rid of JsonPropertyName attributes in PubSubConfig
  - The Dependency Injection framework (based around dotnet ApplicationBuilder)
    does not support this attribute, and maps property names verbatim to json fields.

* Change PubSubServer json fields to match cs class property names verbatim
2025-11-30 16:44:31 +00:00
John Colvin 414ced01ee refactor: Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl 2025-11-30 16:44:20 +00:00
John Colvin 1fb8cf6220 refactor: Decouple ManagedSubscriber from ISubscriberCore impl 2025-11-30 16:44:20 +00:00
John Colvin 79b17579c3 refactor: Only support 1 tag type in IPubSubServerTagHandler 2025-11-30 16:44:20 +00:00
John Colvin 8fe24fd022 refactor: Decouple PubTableBroker from IPublisherCore impl
Reference interface rather than the PublisherCore impl.
2025-11-30 16:44:20 +00:00
John Colvin f0b89ffa23 refactor: Add Spark service modules for Dependency Injection 2025-11-30 16:44:18 +00:00
John Colvin 632f2c588f fix: Dispose market data sub in various md streams 2025-11-21 14:55:01 -05:00
John Colvin bbf1e87766 cleanup: Remove unused pubsub tag handler code
* Removed 'IPubSubServerTagHandler.register'
  - All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
  - This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
  - No reason these interfaces have to be tied together
2025-11-21 14:55:01 -05:00
John Colvin 137be2b861 refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
This matches the file name
2025-11-21 14:55:01 -05:00
John Colvin ab6db8f2da cleanup: Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
In config json files (not actual code change)
2025-11-21 14:55:01 -05:00
John Colvin 366769a778 chore: Fix typos in log msgs
* Bad string interopolation syntax
2025-11-21 14:55:01 -05:00
John Colvin a76cbafdbd cleanup: Fix numerous build warnings in DataStreamSuite
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.

* Compression related files
  - BZip2InputStream  (disable nullability warnings)
  - BZip2OutputStream (disable nullability warnings)
  - DeflaterHuffman   (disable nullability warnings)
  - Inflater          (disable nullability warnings)
  - InflaterDynHeader (disable nullability warnings)
  - ParallelJoinTest  (disable nullability warnings)
  - TarHeader
  - TarInputStream    (disable nullability warnings)
  - ZipEntry
  - ZipFile
  - ZipInputStream

* AsyncTCPConnection_APM
* AsyncTCPConnection_APM
* InflaterHuffmanTree
* ManagedSubscriber
* ManagedTCPMessageHub
* MapStream
* PubSubClientMessage
  - Normalize indentation/whitespace
  - Fix some typos
  - Address nullability issues to fix build warnings
  - Modernize code
* PubSubMessageReactor
* PubSubServerManagedTCPConnection
* PubSubSingleTagStateEvent
* PubSubState
* PublisherCore
* PubsubDeserializationController
* RemoteMessage
  - make 'message' nullable to resolve warnings elsewhere (in calling code)
* SharedStreamConfig
* SingleTagData
* SubscriberCore
* SuperMapStream
* SystemHeartbeatPublisher
* SystemStatsPublisher
* ThreadSafeQueue
* TradeBarsStream

fix build (warn): Only use Registry on windows
2025-11-21 14:54:58 -05:00
John Colvin a442e27d3d cleanup: Modernize PubSubRoute code, json parsing 2025-11-21 14:20:49 -05:00
John Colvin c3da5a5d0c cleanup: Remove unused DateTime grammar from exp parser 2025-11-21 14:20:48 -05:00
John Colvin 13fc7e0455 cleanup: Remove empty file (clearly added by accident) 2025-11-21 14:20:48 -05:00
John Colvin 0a8f002a6e fix: Remove query from literal error stream
The parser can't handle colons in the expression (it throws/crashes)
2025-11-21 14:09:28 -05:00