John Colvin
cc00ff2a0a
Add SparkLoggerAdapter for Msft ILogger
2025-11-03 16:19:48 -05:00
John Colvin
28545b5099
fix: Don't assert on exception in BooleanExpression.evaluate
...
Bugs (fixed):
* DataStream WebApp Server sometimes crashes when the user enters a
bad query.
2025-11-03 16:19:48 -05:00
John Colvin
04c1ad7be8
Assert that local ip address is successfully obtained
2025-11-03 16:19:48 -05:00
John Colvin
2441a5267d
improvement: Resolve hostnames when comparing ip addrs
...
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2025-11-03 16:19:48 -05:00
John Colvin
12f44f63aa
Assert TcpConnection endpoint matches expected
2025-11-03 16:19:48 -05:00
John Colvin
c2fe578ccb
Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
...
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2025-11-03 16:19:48 -05:00
John Colvin
a01fc6cc15
fix: Format PubSubRoute as json in log msgs
...
It was previously using the default ToString, which simply returns the type name
2025-11-03 16:19:48 -05:00
John Colvin
d80cd9c0fd
build: Upgrade packages (deps) in DataStreamTests
2025-11-03 16:19:48 -05:00
John Colvin
d707847079
cleanup: Remove unused ViewStreamDatabase config
2025-11-03 16:19:48 -05:00
John Colvin
118e2a9f2e
Remove reference to spark protobuf Config in ViewStream
...
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-03 16:19:48 -05:00
John Colvin
1481a3fd07
refactor: Run RemoteQueryManager as an IHostedService
...
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-11-03 16:19:45 -05:00
John Colvin
a5e58c26ce
fix: Ensure spark config init and login are only performed once
...
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.
It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-11-03 16:13:44 -05:00
John Colvin
8637bb9254
Assert ViewStreamDatabase.instance is only created once
...
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-03 16:13:44 -05:00
John Colvin
7c61c2ef6d
cleanup: Remove unused code from ViewStreamDatabase
2025-11-03 16:13:44 -05:00
John Colvin
0668e0cd8c
fix: Define BoooleanConstant before StringConstant
...
Otherwise the former masks the latter and prevents it from matching
2025-11-03 16:13:44 -05:00
John Colvin
3ae88248cc
cleanup: Remove unnecessary deps from csprojs
2025-11-03 16:13:43 -05:00
John Colvin
1367cea9ca
fix: Catch exception when creating TcpClient
2025-11-03 16:13:43 -05:00
John Colvin
b384d89820
fix: Make SystemSubscriptionMapPublisher threadsafe
...
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-11-03 16:13:43 -05:00
John Colvin
d669d24aea
fix: Add a few null checks
...
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-11-03 16:13:43 -05:00
John Colvin
e5bc6667e7
Reduce task wait/timeout code in AsyncTCPConnector
2025-11-03 16:13:43 -05:00
John Colvin
41512cb811
fix build warnings: Make props nullable strings in SharedStreamConfig
2025-11-03 16:13:43 -05:00
John Colvin
01d600b821
Assert that System publishers 'doWork' is always called on same thread
2025-11-03 16:13:43 -05:00
John Colvin
9e1cb14523
refactor: Model config profiles after dotnet appsettings
...
Namely, in DataStreamPublisher and PubSubServer. This is part of a
broader paradigm shift towards dotnet factory pattern.
https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines
* Rename Configurations/<name>.json -> appsettings.<name>.json
* Move reusable startup code into new Bts.Extensions.Hosting proj
2025-11-03 16:13:41 -05:00
John Colvin
cd7c04dce5
Add ErrorChanged event to propagate DataStream exceptions
2025-11-03 16:13:19 -05:00
John Colvin
6aee76590f
fix build: Fix Spark build after making config init async
2025-11-03 16:13:19 -05:00
John Colvin
fc79c451b2
refactor: Change Configuration.initialize to be async
...
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-11-03 16:13:19 -05:00
John Colvin
826e03af94
cleanup: Remove unused PubSubConfig props
2025-11-03 16:13:19 -05:00
John Colvin
57217c9adf
Capture OperationCanceledException (instead of TaskCanceledException)
...
The former is the base class for the latter, so this will widen the umbrella to catch a few corner cases
that would otherwise been handled by the generic "Exception" catch block. One corner case I know
of is when an op is canceled via "CancellationToken.ThrowIfCancellationRequested". In that case,
an OperationCanceledException is thrown (not a TaskCanceledException).
https://stackoverflow.com/questions/34359129/operationcanceledexception-vs-taskcanceledexception-when-task-is-canceled
2025-11-03 16:13:19 -05:00
John Colvin
deae8c7fb0
Fix dsqa config json
2025-11-03 16:13:18 -05:00
John Colvin
81d3b10c2b
fix: Always call 'handle_internal_pubs' in PubSubMessageReactor main loop
2025-11-03 16:13:18 -05:00
John Colvin
f2cc576c51
Fix mismatched types in PubSubServerTagStatsStream schema
2025-11-03 16:13:18 -05:00
John Colvin
b25038fac6
Add DataStreamPublisher service modules designed for Dependency Injection
...
* Change DataStreamPublisher json fields to match cs class property names verbatim
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
2025-11-03 16:13:18 -05:00
John Colvin
9b95caa8e6
refactor: Decouple ViewPublisher from IPubTableBroker impl
2025-11-03 16:13:18 -05:00
John Colvin
b4cb6ac856
Add PubSub service modules designed for Dependency Injection
...
* Get rid of JsonPropertyName attributes in PubSubConfig
- The Dependency Injection framework (based around dotnet ApplicationBuilder)
does not support this attribute, and maps property names verbatim to json fields.
* Change PubSubServer json fields to match cs class property names verbatim
2025-11-03 16:13:18 -05:00
John Colvin
83cbb0f47f
refactor: Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl
2025-11-03 16:13:18 -05:00
John Colvin
4e6d1a2335
refactor: Decouple ManagedSubscriber from ISubscriberCore impl
2025-11-03 16:13:18 -05:00
John Colvin
41cd59e6f9
refactor: Only support 1 tag type in IPubSubServerTagHandler
2025-11-03 16:13:18 -05:00
John Colvin
f2ace4ae32
refactor: Decouple PubTableBroker from IPublisherCore impl
...
Reference interface rather than the PublisherCore impl.
2025-11-03 16:13:18 -05:00
John Colvin
8db5655151
fix: Dispose market data sub in various md streams
2025-11-03 16:13:18 -05:00
John Colvin
135db27733
cleanup: Remove unused pubsub tag handler code
...
* Removed 'IPubSubServerTagHandler.register'
- All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
- This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
- No reason these interfaces have to be tied together
2025-11-03 16:13:18 -05:00
John Colvin
f240f005f4
refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
...
This matches the file name
2025-11-03 16:13:18 -05:00
John Colvin
67b1a213b2
Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
...
In config json files (not actual code change)
2025-11-03 16:13:17 -05:00
John Colvin
e8a16b11bb
chore: Fix typos in log msgs
...
* Bad string interopolation syntax
2025-11-03 16:13:17 -05:00
John Colvin
8e42eb5d18
Add spark service modules designed for Dependency Injection
2025-11-03 16:13:17 -05:00
John Colvin
f289c3d2b0
fix build (warn): Only use Registry on windows
2025-11-03 16:13:17 -05:00
John Colvin
6e77eda16c
cleanup: Fix numerous build warnings in DataStreamSuite
...
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.
* PubSubClientMessage
- Normalize indentation/whitespace
- Fix some typos
- Address nullability issues to fix build warnings
- Modernize code
* Compression related files
- ZipEntry
- ZipFile
- ZipInputStream
- TarHeader
* RemoteMessage
- make 'message' nullable to resolve warnings elsewhere (in calling code)
* PubSubSingleTagStateEvent.cs
* PubSubMessageReactor
* PubSubState
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SingleTagData
* ManagedTCPMessageHub
* PubsubDeserializationController
* PublisherCore
* InflaterHuffmanTree
* AsyncTCPConnection_APM
* ThreadSafeQueue
* ManagedSubscriber
* SubscriberCore
* PubSubServerManagedTCPConnection
* AsyncTCPConnection_APM
* SuperMapStream
* MapStream
* TradeBarsStream
2025-11-03 16:13:17 -05:00
John Colvin
41be31a7b0
cleanup: Disable nullability warnings
...
* BZip2InputStream
* TarInputStream
* Inflater
* InflaterDynHeader
* BZip2OutputStream
* DeflaterHuffman
* ParallelJoinTest
2025-11-03 16:13:17 -05:00
John Colvin
96976b3fcd
cleanup: Modernize PubSubRoute code, json parsing
2025-11-03 16:13:17 -05:00
John Colvin
58d6a4534e
cleanup: Remove unused DateTime grammar from exp parser
2025-11-03 16:13:17 -05:00
John Colvin
bf913f2b99
chore: Add json facilities for DataStream websocket api
2025-11-03 16:13:17 -05:00