John Colvin
f81dc08621
cleanup: Remove unused ViewStreamDatabase config
2025-11-19 13:23:43 -05:00
John Colvin
d603adcf7a
Remove reference to spark protobuf Config in ViewStream
...
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-19 13:23:43 -05:00
John Colvin
7932ede1e8
refactor: Run RemoteQueryManager as an IHostedService
...
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-11-19 13:23:42 -05:00
John Colvin
3e2bf37cc3
fix: Ensure spark config init and login are only performed once
...
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.
It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-11-19 13:23:42 -05:00
John Colvin
95683de1c6
Assert ViewStreamDatabase.instance is only created once
...
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-19 13:23:42 -05:00
John Colvin
00c8081516
cleanup: Remove unused code from ViewStreamDatabase
2025-11-19 13:23:40 -05:00
John Colvin
9b1aef6f22
fix: Define BoooleanConstant before StringConstant
...
Otherwise the former masks the latter and prevents it from matching
2025-11-19 13:21:51 -05:00
John Colvin
7c5459291e
cleanup: Remove unnecessary deps from csprojs
2025-11-19 13:21:51 -05:00
John Colvin
0871a15961
fix: Catch exception when creating TcpClient
2025-11-19 13:21:51 -05:00
John Colvin
1fab109987
fix: Make SystemSubscriptionMapPublisher threadsafe
...
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-11-19 13:21:51 -05:00
John Colvin
12ea9a060a
fix: Add a few null checks
...
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-11-19 13:21:51 -05:00
John Colvin
a3794fdac4
Reduce task wait/timeout code in AsyncTCPConnector
2025-11-19 13:21:51 -05:00
John Colvin
4e7eab8e47
fix build warnings: Make props nullable strings in SharedStreamConfig
2025-11-19 13:21:51 -05:00
John Colvin
754dfa87c7
Assert that System publishers 'doWork' is always called on same thread
2025-11-19 13:21:51 -05:00
John Colvin
50a1dfae14
refactor: Model config profiles after dotnet appsettings
...
Namely, in DataStreamPublisher and PubSubServer. This is part of a
broader paradigm shift towards dotnet factory pattern.
https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines
* Rename Configurations/<name>.json -> appsettings.<name>.json
* Move reusable startup code into new Bts.Extensions.Hosting proj
2025-11-19 13:21:50 -05:00
John Colvin
6e09854337
Add ErrorChanged event to propagate DataStream exceptions
2025-11-19 13:21:50 -05:00
John Colvin
df049622cb
fix build: Fix Spark build after making config init async
2025-11-19 13:21:48 -05:00
John Colvin
0f0420ba4c
refactor: Change Configuration.initialize to be async
...
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-11-19 13:18:10 -05:00
John Colvin
6c65cabda8
cleanup: Remove unused PubSubConfig props
2025-11-19 13:17:12 -05:00
John Colvin
64ad720ed6
Capture OperationCanceledException (instead of TaskCanceledException)
...
The former is the base class for the latter, so this will widen the umbrella to catch a few corner cases
that would otherwise been handled by the generic "Exception" catch block. One corner case I know
of is when an op is canceled via "CancellationToken.ThrowIfCancellationRequested". In that case,
an OperationCanceledException is thrown (not a TaskCanceledException).
https://stackoverflow.com/questions/34359129/operationcanceledexception-vs-taskcanceledexception-when-task-is-canceled
2025-11-19 13:17:12 -05:00
John Colvin
550bf5dea5
Fix dsqa config json
2025-11-19 13:17:10 -05:00
John Colvin
c4cbaff350
fix: Always call 'handle_internal_pubs' in PubSubMessageReactor main loop
2025-11-19 13:16:06 -05:00
John Colvin
82c99f0eb3
Fix mismatched types in PubSubServerTagStatsStream schema
2025-11-19 13:16:06 -05:00
John Colvin
cbf1ef3d89
Add DataStreamPublisher service modules designed for Dependency Injection
...
* Change DataStreamPublisher json fields to match cs class property names verbatim
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
2025-11-19 13:16:04 -05:00
John Colvin
a86ddff4be
refactor: Decouple ViewPublisher from IPubTableBroker impl
2025-11-19 12:27:23 -05:00
John Colvin
f51d0ae973
Add PubSub service modules designed for Dependency Injection
...
* Get rid of JsonPropertyName attributes in PubSubConfig
- The Dependency Injection framework (based around dotnet ApplicationBuilder)
does not support this attribute, and maps property names verbatim to json fields.
* Change PubSubServer json fields to match cs class property names verbatim
2025-11-19 12:27:20 -05:00
John Colvin
b9125fb44b
refactor: Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl
2025-11-19 12:24:39 -05:00
John Colvin
d51e09719d
refactor: Decouple ManagedSubscriber from ISubscriberCore impl
2025-11-19 12:24:39 -05:00
John Colvin
520f6311c7
refactor: Only support 1 tag type in IPubSubServerTagHandler
2025-11-19 12:24:39 -05:00
John Colvin
a19658c1e3
refactor: Decouple PubTableBroker from IPublisherCore impl
...
Reference interface rather than the PublisherCore impl.
2025-11-19 12:24:39 -05:00
John Colvin
22b46b38ee
fix: Dispose market data sub in various md streams
2025-11-19 12:24:38 -05:00
John Colvin
c66511a089
cleanup: Remove unused pubsub tag handler code
...
* Removed 'IPubSubServerTagHandler.register'
- All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
- This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
- No reason these interfaces have to be tied together
2025-11-19 12:24:38 -05:00
John Colvin
b2e94db451
refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
...
This matches the file name
2025-11-19 12:24:38 -05:00
John Colvin
010f46a2a0
Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
...
In config json files (not actual code change)
2025-11-19 12:24:38 -05:00
John Colvin
b4f59fd13b
chore: Fix typos in log msgs
...
* Bad string interopolation syntax
2025-11-19 12:24:38 -05:00
John Colvin
d5840bf0cc
Add spark service modules designed for Dependency Injection
2025-11-19 12:24:38 -05:00
John Colvin
a569fec6cc
fix build (warn): Only use Registry on windows
2025-11-19 12:24:38 -05:00
John Colvin
77c616af3e
cleanup: Fix numerous build warnings in DataStreamSuite
...
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.
* PubSubClientMessage
- Normalize indentation/whitespace
- Fix some typos
- Address nullability issues to fix build warnings
- Modernize code
* Compression related files
- ZipEntry
- ZipFile
- ZipInputStream
- TarHeader
* RemoteMessage
- make 'message' nullable to resolve warnings elsewhere (in calling code)
* PubSubSingleTagStateEvent.cs
* PubSubMessageReactor
* PubSubState
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SingleTagData
* ManagedTCPMessageHub
* PubsubDeserializationController
* PublisherCore
* InflaterHuffmanTree
* AsyncTCPConnection_APM
* ThreadSafeQueue
* ManagedSubscriber
* SubscriberCore
* PubSubServerManagedTCPConnection
* AsyncTCPConnection_APM
* SuperMapStream
* MapStream
* TradeBarsStream
2025-11-19 12:24:38 -05:00
John Colvin
4d0d7cf279
cleanup: Disable nullability warnings
...
* BZip2InputStream
* TarInputStream
* Inflater
* InflaterDynHeader
* BZip2OutputStream
* DeflaterHuffman
* ParallelJoinTest
2025-11-19 12:24:37 -05:00
John Colvin
a81a4772d3
cleanup: Modernize PubSubRoute code, json parsing
2025-11-19 12:24:37 -05:00
John Colvin
24114b067b
cleanup: Remove unused DateTime grammar from exp parser
2025-11-19 12:24:37 -05:00
John Colvin
57e0f0e8bd
refactor: Add handy WebsocketExtensions
...
* IsSocketStateOpen
* IsClosedProperly
2025-11-19 12:24:37 -05:00
John Colvin
8fed31c86d
chore: Add JsonRpc types for websocket paradigm
2025-11-19 12:24:37 -05:00
John Colvin
b04b9db005
cleanup: Remove empty file (clearly added by accident)
2025-11-19 12:24:37 -05:00
John Colvin
f571b4992f
Add generic ViewStream, BlueRow types/interfaces
2025-11-19 12:24:37 -05:00
John Colvin
1d70eecb44
Add BlueRowColumnMask to hide columns w/ minimal overhead
...
Less overhead than ProjectStream
2025-11-19 12:24:36 -05:00
John Colvin
ca82b8f001
fix: Remove query from literal error stream
...
The parser can't handle colons in the expression (it throws/crashes)
2025-11-19 12:24:36 -05:00
John Colvin
c5df486cf4
fix: Remove erroneous "DSL: " prefix in ai prompt stream
...
This causes parse errors since we extended the dsl syntax to support
named params in the form of "name: arg".
2025-11-19 12:24:36 -05:00
John Colvin
69632d7327
fix Impl IDictionary<int, IBlueRow> in BlueTable
2025-11-19 12:24:36 -05:00
John Colvin
ae21ae02d2
feat: Extend dsl to support explicit param names
...
When using explicit names, the argument order doesn't matter, and args
with defaults are automatically populated if omitted.
2025-11-19 12:24:36 -05:00