Commit Graph
12725 Commits
Author SHA1 Message Date
John Colvin f81dc08621 cleanup: Remove unused ViewStreamDatabase config 2025-11-19 13:23:43 -05:00
John Colvin d603adcf7a Remove reference to spark protobuf Config in ViewStream
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-19 13:23:43 -05:00
John Colvin 7932ede1e8 refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-11-19 13:23:42 -05:00
John Colvin 3e2bf37cc3 fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-11-19 13:23:42 -05:00
John Colvin 95683de1c6 Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-19 13:23:42 -05:00
John Colvin 00c8081516 cleanup: Remove unused code from ViewStreamDatabase 2025-11-19 13:23:40 -05:00
John Colvin 9b1aef6f22 fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2025-11-19 13:21:51 -05:00
John Colvin 7c5459291e cleanup: Remove unnecessary deps from csprojs 2025-11-19 13:21:51 -05:00
John Colvin 0871a15961 fix: Catch exception when creating TcpClient 2025-11-19 13:21:51 -05:00
John Colvin 1fab109987 fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-11-19 13:21:51 -05:00
John Colvin 12ea9a060a fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-11-19 13:21:51 -05:00
John Colvin a3794fdac4 Reduce task wait/timeout code in AsyncTCPConnector 2025-11-19 13:21:51 -05:00
John Colvin 4e7eab8e47 fix build warnings: Make props nullable strings in SharedStreamConfig 2025-11-19 13:21:51 -05:00
John Colvin 754dfa87c7 Assert that System publishers 'doWork' is always called on same thread 2025-11-19 13:21:51 -05:00
John Colvin 50a1dfae14 refactor: Model config profiles after dotnet appsettings
Namely, in DataStreamPublisher and PubSubServer. This is part of a
broader paradigm shift towards dotnet factory pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Rename Configurations/<name>.json -> appsettings.<name>.json
* Move reusable startup code into new Bts.Extensions.Hosting proj
2025-11-19 13:21:50 -05:00
John Colvin 6e09854337 Add ErrorChanged event to propagate DataStream exceptions 2025-11-19 13:21:50 -05:00
John Colvin df049622cb fix build: Fix Spark build after making config init async 2025-11-19 13:21:48 -05:00
John Colvin 0f0420ba4c refactor: Change Configuration.initialize to be async
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-11-19 13:18:10 -05:00
John Colvin 6c65cabda8 cleanup: Remove unused PubSubConfig props 2025-11-19 13:17:12 -05:00
John Colvin 64ad720ed6 Capture OperationCanceledException (instead of TaskCanceledException)
The former is the base class for the latter, so this will widen the umbrella to catch a few corner cases
that would otherwise been handled by the generic "Exception" catch block. One corner case I know
of is when an op is canceled via "CancellationToken.ThrowIfCancellationRequested". In that case,
an OperationCanceledException is thrown (not a TaskCanceledException).

https://stackoverflow.com/questions/34359129/operationcanceledexception-vs-taskcanceledexception-when-task-is-canceled
2025-11-19 13:17:12 -05:00
John Colvin 550bf5dea5 Fix dsqa config json 2025-11-19 13:17:10 -05:00
John Colvin c4cbaff350 fix: Always call 'handle_internal_pubs' in PubSubMessageReactor main loop 2025-11-19 13:16:06 -05:00
John Colvin 82c99f0eb3 Fix mismatched types in PubSubServerTagStatsStream schema 2025-11-19 13:16:06 -05:00
John Colvin cbf1ef3d89 Add DataStreamPublisher service modules designed for Dependency Injection
* Change DataStreamPublisher json fields to match cs class property names verbatim
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
2025-11-19 13:16:04 -05:00
John Colvin a86ddff4be refactor: Decouple ViewPublisher from IPubTableBroker impl 2025-11-19 12:27:23 -05:00
John Colvin f51d0ae973 Add PubSub service modules designed for Dependency Injection
* Get rid of JsonPropertyName attributes in PubSubConfig
  - The Dependency Injection framework (based around dotnet ApplicationBuilder)
    does not support this attribute, and maps property names verbatim to json fields.

* Change PubSubServer json fields to match cs class property names verbatim
2025-11-19 12:27:20 -05:00
John Colvin b9125fb44b refactor: Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl 2025-11-19 12:24:39 -05:00
John Colvin d51e09719d refactor: Decouple ManagedSubscriber from ISubscriberCore impl 2025-11-19 12:24:39 -05:00
John Colvin 520f6311c7 refactor: Only support 1 tag type in IPubSubServerTagHandler 2025-11-19 12:24:39 -05:00
John Colvin a19658c1e3 refactor: Decouple PubTableBroker from IPublisherCore impl
Reference interface rather than the PublisherCore impl.
2025-11-19 12:24:39 -05:00
John Colvin 22b46b38ee fix: Dispose market data sub in various md streams 2025-11-19 12:24:38 -05:00
John Colvin c66511a089 cleanup: Remove unused pubsub tag handler code
* Removed 'IPubSubServerTagHandler.register'
  - All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
  - This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
  - No reason these interfaces have to be tied together
2025-11-19 12:24:38 -05:00
John Colvin b2e94db451 refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
This matches the file name
2025-11-19 12:24:38 -05:00
John Colvin 010f46a2a0 Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
In config json files (not actual code change)
2025-11-19 12:24:38 -05:00
John Colvin b4f59fd13b chore: Fix typos in log msgs
* Bad string interopolation syntax
2025-11-19 12:24:38 -05:00
John Colvin d5840bf0cc Add spark service modules designed for Dependency Injection 2025-11-19 12:24:38 -05:00
John Colvin a569fec6cc fix build (warn): Only use Registry on windows 2025-11-19 12:24:38 -05:00
John Colvin 77c616af3e cleanup: Fix numerous build warnings in DataStreamSuite
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.

* PubSubClientMessage
  - Normalize indentation/whitespace
  - Fix some typos
  - Address nullability issues to fix build warnings
  - Modernize code

* Compression related files
  - ZipEntry
  - ZipFile
  - ZipInputStream
  - TarHeader

* RemoteMessage
  - make 'message' nullable to resolve warnings elsewhere (in calling code)

* PubSubSingleTagStateEvent.cs
* PubSubMessageReactor
* PubSubState
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SingleTagData
* ManagedTCPMessageHub
* PubsubDeserializationController
* PublisherCore
* InflaterHuffmanTree
* AsyncTCPConnection_APM
* ThreadSafeQueue
* ManagedSubscriber
* SubscriberCore
* PubSubServerManagedTCPConnection
* AsyncTCPConnection_APM
* SuperMapStream
* MapStream
* TradeBarsStream
2025-11-19 12:24:38 -05:00
John Colvin 4d0d7cf279 cleanup: Disable nullability warnings
* BZip2InputStream
* TarInputStream
* Inflater
* InflaterDynHeader
* BZip2OutputStream
* DeflaterHuffman
* ParallelJoinTest
2025-11-19 12:24:37 -05:00
John Colvin a81a4772d3 cleanup: Modernize PubSubRoute code, json parsing 2025-11-19 12:24:37 -05:00
John Colvin 24114b067b cleanup: Remove unused DateTime grammar from exp parser 2025-11-19 12:24:37 -05:00
John Colvin 57e0f0e8bd refactor: Add handy WebsocketExtensions
* IsSocketStateOpen
* IsClosedProperly
2025-11-19 12:24:37 -05:00
John Colvin 8fed31c86d chore: Add JsonRpc types for websocket paradigm 2025-11-19 12:24:37 -05:00
John Colvin b04b9db005 cleanup: Remove empty file (clearly added by accident) 2025-11-19 12:24:37 -05:00
John Colvin f571b4992f Add generic ViewStream, BlueRow types/interfaces 2025-11-19 12:24:37 -05:00
John Colvin 1d70eecb44 Add BlueRowColumnMask to hide columns w/ minimal overhead
Less overhead than ProjectStream
2025-11-19 12:24:36 -05:00
John Colvin ca82b8f001 fix: Remove query from literal error stream
The parser can't handle colons in the expression (it throws/crashes)
2025-11-19 12:24:36 -05:00
John Colvin c5df486cf4 fix: Remove erroneous "DSL: " prefix in ai prompt stream
This causes parse errors since we extended the dsl syntax to support
named params in the form of "name: arg".
2025-11-19 12:24:36 -05:00
John Colvin 69632d7327 fix Impl IDictionary<int, IBlueRow> in BlueTable 2025-11-19 12:24:36 -05:00
John Colvin ae21ae02d2 feat: Extend dsl to support explicit param names
When using explicit names, the argument order doesn't matter, and args
with defaults are automatically populated if omitted.
2025-11-19 12:24:36 -05:00