Commit Graph
12727 Commits
Author SHA1 Message Date
John Colvin cc00ff2a0a Add SparkLoggerAdapter for Msft ILogger 2025-11-03 16:19:48 -05:00
John Colvin 28545b5099 fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2025-11-03 16:19:48 -05:00
John Colvin 04c1ad7be8 Assert that local ip address is successfully obtained 2025-11-03 16:19:48 -05:00
John Colvin 2441a5267d improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2025-11-03 16:19:48 -05:00
John Colvin 12f44f63aa Assert TcpConnection endpoint matches expected 2025-11-03 16:19:48 -05:00
John Colvin c2fe578ccb Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2025-11-03 16:19:48 -05:00
John Colvin a01fc6cc15 fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2025-11-03 16:19:48 -05:00
John Colvin d80cd9c0fd build: Upgrade packages (deps) in DataStreamTests 2025-11-03 16:19:48 -05:00
John Colvin d707847079 cleanup: Remove unused ViewStreamDatabase config 2025-11-03 16:19:48 -05:00
John Colvin 118e2a9f2e Remove reference to spark protobuf Config in ViewStream
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-03 16:19:48 -05:00
John Colvin 1481a3fd07 refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-11-03 16:19:45 -05:00
John Colvin a5e58c26ce fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-11-03 16:13:44 -05:00
John Colvin 8637bb9254 Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-03 16:13:44 -05:00
John Colvin 7c61c2ef6d cleanup: Remove unused code from ViewStreamDatabase 2025-11-03 16:13:44 -05:00
John Colvin 0668e0cd8c fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2025-11-03 16:13:44 -05:00
John Colvin 3ae88248cc cleanup: Remove unnecessary deps from csprojs 2025-11-03 16:13:43 -05:00
John Colvin 1367cea9ca fix: Catch exception when creating TcpClient 2025-11-03 16:13:43 -05:00
John Colvin b384d89820 fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-11-03 16:13:43 -05:00
John Colvin d669d24aea fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-11-03 16:13:43 -05:00
John Colvin e5bc6667e7 Reduce task wait/timeout code in AsyncTCPConnector 2025-11-03 16:13:43 -05:00
John Colvin 41512cb811 fix build warnings: Make props nullable strings in SharedStreamConfig 2025-11-03 16:13:43 -05:00
John Colvin 01d600b821 Assert that System publishers 'doWork' is always called on same thread 2025-11-03 16:13:43 -05:00
John Colvin 9e1cb14523 refactor: Model config profiles after dotnet appsettings
Namely, in DataStreamPublisher and PubSubServer. This is part of a
broader paradigm shift towards dotnet factory pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Rename Configurations/<name>.json -> appsettings.<name>.json
* Move reusable startup code into new Bts.Extensions.Hosting proj
2025-11-03 16:13:41 -05:00
John Colvin cd7c04dce5 Add ErrorChanged event to propagate DataStream exceptions 2025-11-03 16:13:19 -05:00
John Colvin 6aee76590f fix build: Fix Spark build after making config init async 2025-11-03 16:13:19 -05:00
John Colvin fc79c451b2 refactor: Change Configuration.initialize to be async
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-11-03 16:13:19 -05:00
John Colvin 826e03af94 cleanup: Remove unused PubSubConfig props 2025-11-03 16:13:19 -05:00
John Colvin 57217c9adf Capture OperationCanceledException (instead of TaskCanceledException)
The former is the base class for the latter, so this will widen the umbrella to catch a few corner cases
that would otherwise been handled by the generic "Exception" catch block. One corner case I know
of is when an op is canceled via "CancellationToken.ThrowIfCancellationRequested". In that case,
an OperationCanceledException is thrown (not a TaskCanceledException).

https://stackoverflow.com/questions/34359129/operationcanceledexception-vs-taskcanceledexception-when-task-is-canceled
2025-11-03 16:13:19 -05:00
John Colvin deae8c7fb0 Fix dsqa config json 2025-11-03 16:13:18 -05:00
John Colvin 81d3b10c2b fix: Always call 'handle_internal_pubs' in PubSubMessageReactor main loop 2025-11-03 16:13:18 -05:00
John Colvin f2cc576c51 Fix mismatched types in PubSubServerTagStatsStream schema 2025-11-03 16:13:18 -05:00
John Colvin b25038fac6 Add DataStreamPublisher service modules designed for Dependency Injection
* Change DataStreamPublisher json fields to match cs class property names verbatim
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
2025-11-03 16:13:18 -05:00
John Colvin 9b95caa8e6 refactor: Decouple ViewPublisher from IPubTableBroker impl 2025-11-03 16:13:18 -05:00
John Colvin b4cb6ac856 Add PubSub service modules designed for Dependency Injection
* Get rid of JsonPropertyName attributes in PubSubConfig
  - The Dependency Injection framework (based around dotnet ApplicationBuilder)
    does not support this attribute, and maps property names verbatim to json fields.

* Change PubSubServer json fields to match cs class property names verbatim
2025-11-03 16:13:18 -05:00
John Colvin 83cbb0f47f refactor: Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl 2025-11-03 16:13:18 -05:00
John Colvin 4e6d1a2335 refactor: Decouple ManagedSubscriber from ISubscriberCore impl 2025-11-03 16:13:18 -05:00
John Colvin 41cd59e6f9 refactor: Only support 1 tag type in IPubSubServerTagHandler 2025-11-03 16:13:18 -05:00
John Colvin f2ace4ae32 refactor: Decouple PubTableBroker from IPublisherCore impl
Reference interface rather than the PublisherCore impl.
2025-11-03 16:13:18 -05:00
John Colvin 8db5655151 fix: Dispose market data sub in various md streams 2025-11-03 16:13:18 -05:00
John Colvin 135db27733 cleanup: Remove unused pubsub tag handler code
* Removed 'IPubSubServerTagHandler.register'
  - All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
  - This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
  - No reason these interfaces have to be tied together
2025-11-03 16:13:18 -05:00
John Colvin f240f005f4 refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
This matches the file name
2025-11-03 16:13:18 -05:00
John Colvin 67b1a213b2 Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
In config json files (not actual code change)
2025-11-03 16:13:17 -05:00
John Colvin e8a16b11bb chore: Fix typos in log msgs
* Bad string interopolation syntax
2025-11-03 16:13:17 -05:00
John Colvin 8e42eb5d18 Add spark service modules designed for Dependency Injection 2025-11-03 16:13:17 -05:00
John Colvin f289c3d2b0 fix build (warn): Only use Registry on windows 2025-11-03 16:13:17 -05:00
John Colvin 6e77eda16c cleanup: Fix numerous build warnings in DataStreamSuite
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.

* PubSubClientMessage
  - Normalize indentation/whitespace
  - Fix some typos
  - Address nullability issues to fix build warnings
  - Modernize code

* Compression related files
  - ZipEntry
  - ZipFile
  - ZipInputStream
  - TarHeader

* RemoteMessage
  - make 'message' nullable to resolve warnings elsewhere (in calling code)

* PubSubSingleTagStateEvent.cs
* PubSubMessageReactor
* PubSubState
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SingleTagData
* ManagedTCPMessageHub
* PubsubDeserializationController
* PublisherCore
* InflaterHuffmanTree
* AsyncTCPConnection_APM
* ThreadSafeQueue
* ManagedSubscriber
* SubscriberCore
* PubSubServerManagedTCPConnection
* AsyncTCPConnection_APM
* SuperMapStream
* MapStream
* TradeBarsStream
2025-11-03 16:13:17 -05:00
John Colvin 41be31a7b0 cleanup: Disable nullability warnings
* BZip2InputStream
* TarInputStream
* Inflater
* InflaterDynHeader
* BZip2OutputStream
* DeflaterHuffman
* ParallelJoinTest
2025-11-03 16:13:17 -05:00
John Colvin 96976b3fcd cleanup: Modernize PubSubRoute code, json parsing 2025-11-03 16:13:17 -05:00
John Colvin 58d6a4534e cleanup: Remove unused DateTime grammar from exp parser 2025-11-03 16:13:17 -05:00
John Colvin bf913f2b99 chore: Add json facilities for DataStream websocket api 2025-11-03 16:13:17 -05:00