John Colvin
69b1ac243e
fix: Ensure spark config init and login are only performed once
...
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.
It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-12-05 15:53:31 +00:00
John Colvin
f3f2e5834c
diag: Assert ViewStreamDatabase.instance is only created once
...
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-12-05 15:53:31 +00:00
John Colvin
d93946ce61
cleanup: Remove unused code from ViewStreamDatabase
2025-12-05 15:53:31 +00:00
John Colvin
99e50f539f
fix: Define BoooleanConstant before StringConstant
...
Otherwise the former masks the latter and prevents it from matching
2025-12-05 15:53:31 +00:00
John Colvin
504d87ae14
cleanup: Remove unnecessary deps from csprojs
2025-12-05 15:53:31 +00:00
John Colvin
a8fc2a6179
fix: Catch exception when creating TcpClient
2025-12-05 15:53:31 +00:00
John Colvin
061f323026
fix: Make SystemSubscriptionMapPublisher threadsafe
...
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-12-05 15:53:31 +00:00
John Colvin
60488a4770
fix: Add a few null checks
...
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-12-05 15:53:31 +00:00
John Colvin
5f517e4a7c
improvement: Reduce task wait/timeout code in AsyncTCPConnector
2025-12-05 15:53:31 +00:00
John Colvin
a04fb84656
diag: Assert System publishers 'doWork' is always called on same thread
2025-12-05 15:53:31 +00:00
John Colvin
8bd4a5131d
deploy: Add pubsub and md-publisher services to compose.yaml
2025-12-05 15:53:31 +00:00
John Colvin
95c4b6b036
refactor: Model config profiles after dotnet appsettings
...
Namely, in DataStreamPublisher and PubSubServer. This is part of a
broader paradigm shift towards dotnet factory pattern.
https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines
* Rename Configurations/<name>.json -> appsettings.<name>.json
* Move reusable startup code into new Bts.Extensions.Hosting proj
2025-12-05 15:53:31 +00:00
John Colvin
d2300fbbb9
improvement: Propagate DataStream exceptions via ErrorChanged event
2025-11-30 16:44:31 +00:00
John Colvin
12c21f12fe
refactor: Change Configuration.initialize to be async
...
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-11-30 16:44:31 +00:00
John Colvin
5061a1bbd7
cleanup: Remove unused PubSubConfig props
2025-11-30 16:44:31 +00:00
John Colvin
f6aec08919
fix: Always call 'handle_internal_pubs' in PubSubMessageReactor main loop
2025-11-30 16:44:31 +00:00
John Colvin
652f0e0043
fix: Fix a few column types in in PubSubServerTagStatsStream
...
* total_bytes (int -> long)
* last_update_time (DateTime -> long)
2025-11-30 16:44:31 +00:00
John Colvin
6889a971ff
refactor: Add DataStreamPublisher service modules for Dependency Injection
...
* Change DataStreamPublisher json fields to match cs class property names verbatim
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
2025-11-30 16:44:31 +00:00
John Colvin
c5f4d146f3
refactor: Add PubSub service modules for Dependency Injection
...
* Get rid of JsonPropertyName attributes in PubSubConfig
- The Dependency Injection framework (based around dotnet ApplicationBuilder)
does not support this attribute, and maps property names verbatim to json fields.
* Change PubSubServer json fields to match cs class property names verbatim
2025-11-30 16:44:31 +00:00
John Colvin
414ced01ee
refactor: Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl
2025-11-30 16:44:20 +00:00
John Colvin
1fb8cf6220
refactor: Decouple ManagedSubscriber from ISubscriberCore impl
2025-11-30 16:44:20 +00:00
John Colvin
79b17579c3
refactor: Only support 1 tag type in IPubSubServerTagHandler
2025-11-30 16:44:20 +00:00
John Colvin
8fe24fd022
refactor: Decouple PubTableBroker from IPublisherCore impl
...
Reference interface rather than the PublisherCore impl.
2025-11-30 16:44:20 +00:00
John Colvin
f0b89ffa23
refactor: Add Spark service modules for Dependency Injection
2025-11-30 16:44:18 +00:00
John Colvin
632f2c588f
fix: Dispose market data sub in various md streams
2025-11-21 14:55:01 -05:00
John Colvin
bbf1e87766
cleanup: Remove unused pubsub tag handler code
...
* Removed 'IPubSubServerTagHandler.register'
- All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
- This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
- No reason these interfaces have to be tied together
2025-11-21 14:55:01 -05:00
John Colvin
137be2b861
refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
...
This matches the file name
2025-11-21 14:55:01 -05:00
John Colvin
ab6db8f2da
cleanup: Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
...
In config json files (not actual code change)
2025-11-21 14:55:01 -05:00
John Colvin
366769a778
chore: Fix typos in log msgs
...
* Bad string interopolation syntax
2025-11-21 14:55:01 -05:00
John Colvin
741c80dfd9
vscode: Add DataStreamPublisher debug cfg to skip build
2025-11-21 14:55:01 -05:00
John Colvin
16b50e96c6
cleanup: Fix benign typos in build scripts
...
Change BUILD_SPARK_PY var name to BUILD_PY (these scripts build other
projects than 'spark').
2025-11-21 14:55:01 -05:00
John Colvin
a76cbafdbd
cleanup: Fix numerous build warnings in DataStreamSuite
...
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.
* Compression related files
- BZip2InputStream (disable nullability warnings)
- BZip2OutputStream (disable nullability warnings)
- DeflaterHuffman (disable nullability warnings)
- Inflater (disable nullability warnings)
- InflaterDynHeader (disable nullability warnings)
- ParallelJoinTest (disable nullability warnings)
- TarHeader
- TarInputStream (disable nullability warnings)
- ZipEntry
- ZipFile
- ZipInputStream
* AsyncTCPConnection_APM
* AsyncTCPConnection_APM
* InflaterHuffmanTree
* ManagedSubscriber
* ManagedTCPMessageHub
* MapStream
* PubSubClientMessage
- Normalize indentation/whitespace
- Fix some typos
- Address nullability issues to fix build warnings
- Modernize code
* PubSubMessageReactor
* PubSubServerManagedTCPConnection
* PubSubSingleTagStateEvent
* PubSubState
* PublisherCore
* PubsubDeserializationController
* RemoteMessage
- make 'message' nullable to resolve warnings elsewhere (in calling code)
* SharedStreamConfig
* SingleTagData
* SubscriberCore
* SuperMapStream
* SystemHeartbeatPublisher
* SystemStatsPublisher
* ThreadSafeQueue
* TradeBarsStream
fix build (warn): Only use Registry on windows
2025-11-21 14:54:58 -05:00
John Colvin
a442e27d3d
cleanup: Modernize PubSubRoute code, json parsing
2025-11-21 14:20:49 -05:00
John Colvin
c3da5a5d0c
cleanup: Remove unused DateTime grammar from exp parser
2025-11-21 14:20:48 -05:00
John Colvin
13fc7e0455
cleanup: Remove empty file (clearly added by accident)
2025-11-21 14:20:48 -05:00
John Colvin
0a8f002a6e
fix: Remove query from literal error stream
...
The parser can't handle colons in the expression (it throws/crashes)
2025-11-21 14:09:28 -05:00
John Colvin
e2d45ae8e1
feat: Extend dsl to support explicit param names
...
When using explicit names, the argument order doesn't matter, and args
with defaults are automatically populated if omitted.
2025-11-21 14:09:28 -05:00
John Colvin
cd2a61420d
chore: Add async SQLiteEnumerator to read table as IBlueRowCollection
2025-11-21 14:04:47 -05:00
John Colvin
c2559603a8
chore: Add various collection classes
...
These will be useful in stream operators, row indexing, etc:
* AggregateList
* CircularBuffer
* DeltaList
* DictionarySetWrapper
* ImmutableOrderedDictionary
* ImmutableOrderedSet
* IMutableCollectionFacade
- inherits IReadOnlyCollection<T> and ICollection<T>
- throws NotSupportedException in default impls for mutation ops
* IMutableListFacade
- inherits IReadOnlyList<T> and IList<T>
- throws NotSupportedException in default impls for mutation ops
* IndexDependentSelectorList
* MergedSortedList
* OffsetList
* OrderedDictionary
* ReverseArrayList
* SliceList
* StepList
More notes:
* Renames ExceptionAssert.is_valid ExceptionAssert.Index.IsValid.
- Like the 'ExceptionAssert.Argument' class, which provides methods to throw
ArgumentException if a condition isn't satisfied, this class throws
IndexOutOfRange exceptions.
2025-11-21 13:56:36 -05:00
John Colvin
8ca354e5d7
chore: Support mutations in AggregateDictionary (optionally)
2025-11-21 13:56:36 -05:00
John Colvin
0a4a7693ac
style: Use standard c# capitalization rules
...
* Use 'PascalCase' for public members (instead of 'snake_case')
* Use 'camelCase' for private members (instead of '_snake_case')
2025-11-21 13:56:26 -05:00
John Colvin
44be99ef7a
fix: Support DateTime/TimeSpan columns in DataPacketJsonSerializer
2025-11-21 12:56:10 -05:00
John Colvin
10bb90c731
fix: Use IDictionary.GetEnumerator explicitly to iterate keyvals
...
We really need to replace these old non-generic collections with generic ones.
BlueRow's IDictionaryEnumerator implementation iterates over values when passed
to a foreach loop, not key value pairs.
2025-11-19 23:55:35 -05:00
John Colvin
0746b27a40
cleanup: Remove docker submodules again
...
I lost count of the number of times they have been deleted long ago.
Taha, please remove them from your repo so they don't come back :)
2025-11-19 10:33:31 -05:00
John Colvin
3a3c3d45d3
cleanup: Change default DSQA sparkd config server to 127.0.0.1
2025-11-19 10:33:31 -05:00
John Colvin
2014931653
feat: Add live bbo stream
2025-11-19 10:33:31 -05:00
John Colvin
aa740e786c
refactor: Automatically add id column in BlueTableSchemaBuilder
2025-11-19 00:36:56 -05:00
John Colvin
12c6b367f4
fix (preemptive): Avoid StackOverflow in md client securities fn
...
One of the 'securities_incremental' fns was calling back into itself instead of
the other overload. This is a 'preemptive' fix because the problematic code path
wasn't running in practice (though it was a ticking time bomb).
2025-11-19 00:36:56 -05:00
John Colvin
5080960b97
refactor: Replace DataPacket w/ IDataPacket in stream handlers
...
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-19 00:36:56 -05:00
John Colvin
770e14752a
chore: Add json facilities for DataStream websocket api
2025-11-19 00:36:56 -05:00