Commit Graph
14314 Commits
Author SHA1 Message Date
John Colvin 69b1ac243e fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-12-05 15:53:31 +00:00
John Colvin f3f2e5834c diag: Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-12-05 15:53:31 +00:00
John Colvin d93946ce61 cleanup: Remove unused code from ViewStreamDatabase 2025-12-05 15:53:31 +00:00
John Colvin 99e50f539f fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2025-12-05 15:53:31 +00:00
John Colvin 504d87ae14 cleanup: Remove unnecessary deps from csprojs 2025-12-05 15:53:31 +00:00
John Colvin a8fc2a6179 fix: Catch exception when creating TcpClient 2025-12-05 15:53:31 +00:00
John Colvin 061f323026 fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-12-05 15:53:31 +00:00
John Colvin 60488a4770 fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-12-05 15:53:31 +00:00
John Colvin 5f517e4a7c improvement: Reduce task wait/timeout code in AsyncTCPConnector 2025-12-05 15:53:31 +00:00
John Colvin a04fb84656 diag: Assert System publishers 'doWork' is always called on same thread 2025-12-05 15:53:31 +00:00
John Colvin 8bd4a5131d deploy: Add pubsub and md-publisher services to compose.yaml 2025-12-05 15:53:31 +00:00
John Colvin 95c4b6b036 refactor: Model config profiles after dotnet appsettings
Namely, in DataStreamPublisher and PubSubServer. This is part of a
broader paradigm shift towards dotnet factory pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Rename Configurations/<name>.json -> appsettings.<name>.json
* Move reusable startup code into new Bts.Extensions.Hosting proj
2025-12-05 15:53:31 +00:00
John Colvin d2300fbbb9 improvement: Propagate DataStream exceptions via ErrorChanged event 2025-11-30 16:44:31 +00:00
John Colvin 12c21f12fe refactor: Change Configuration.initialize to be async
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-11-30 16:44:31 +00:00
John Colvin 5061a1bbd7 cleanup: Remove unused PubSubConfig props 2025-11-30 16:44:31 +00:00
John Colvin f6aec08919 fix: Always call 'handle_internal_pubs' in PubSubMessageReactor main loop 2025-11-30 16:44:31 +00:00
John Colvin 652f0e0043 fix: Fix a few column types in in PubSubServerTagStatsStream
* total_bytes      (int      -> long)
* last_update_time (DateTime -> long)
2025-11-30 16:44:31 +00:00
John Colvin 6889a971ff refactor: Add DataStreamPublisher service modules for Dependency Injection
* Change DataStreamPublisher json fields to match cs class property names verbatim
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
2025-11-30 16:44:31 +00:00
John Colvin c5f4d146f3 refactor: Add PubSub service modules for Dependency Injection
* Get rid of JsonPropertyName attributes in PubSubConfig
  - The Dependency Injection framework (based around dotnet ApplicationBuilder)
    does not support this attribute, and maps property names verbatim to json fields.

* Change PubSubServer json fields to match cs class property names verbatim
2025-11-30 16:44:31 +00:00
John Colvin 414ced01ee refactor: Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl 2025-11-30 16:44:20 +00:00
John Colvin 1fb8cf6220 refactor: Decouple ManagedSubscriber from ISubscriberCore impl 2025-11-30 16:44:20 +00:00
John Colvin 79b17579c3 refactor: Only support 1 tag type in IPubSubServerTagHandler 2025-11-30 16:44:20 +00:00
John Colvin 8fe24fd022 refactor: Decouple PubTableBroker from IPublisherCore impl
Reference interface rather than the PublisherCore impl.
2025-11-30 16:44:20 +00:00
John Colvin f0b89ffa23 refactor: Add Spark service modules for Dependency Injection 2025-11-30 16:44:18 +00:00
John Colvin 632f2c588f fix: Dispose market data sub in various md streams 2025-11-21 14:55:01 -05:00
John Colvin bbf1e87766 cleanup: Remove unused pubsub tag handler code
* Removed 'IPubSubServerTagHandler.register'
  - All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
  - This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
  - No reason these interfaces have to be tied together
2025-11-21 14:55:01 -05:00
John Colvin 137be2b861 refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
This matches the file name
2025-11-21 14:55:01 -05:00
John Colvin ab6db8f2da cleanup: Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
In config json files (not actual code change)
2025-11-21 14:55:01 -05:00
John Colvin 366769a778 chore: Fix typos in log msgs
* Bad string interopolation syntax
2025-11-21 14:55:01 -05:00
John Colvin 741c80dfd9 vscode: Add DataStreamPublisher debug cfg to skip build 2025-11-21 14:55:01 -05:00
John Colvin 16b50e96c6 cleanup: Fix benign typos in build scripts
Change BUILD_SPARK_PY var name to BUILD_PY (these scripts build other
projects than 'spark').
2025-11-21 14:55:01 -05:00
John Colvin a76cbafdbd cleanup: Fix numerous build warnings in DataStreamSuite
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.

* Compression related files
  - BZip2InputStream  (disable nullability warnings)
  - BZip2OutputStream (disable nullability warnings)
  - DeflaterHuffman   (disable nullability warnings)
  - Inflater          (disable nullability warnings)
  - InflaterDynHeader (disable nullability warnings)
  - ParallelJoinTest  (disable nullability warnings)
  - TarHeader
  - TarInputStream    (disable nullability warnings)
  - ZipEntry
  - ZipFile
  - ZipInputStream

* AsyncTCPConnection_APM
* AsyncTCPConnection_APM
* InflaterHuffmanTree
* ManagedSubscriber
* ManagedTCPMessageHub
* MapStream
* PubSubClientMessage
  - Normalize indentation/whitespace
  - Fix some typos
  - Address nullability issues to fix build warnings
  - Modernize code
* PubSubMessageReactor
* PubSubServerManagedTCPConnection
* PubSubSingleTagStateEvent
* PubSubState
* PublisherCore
* PubsubDeserializationController
* RemoteMessage
  - make 'message' nullable to resolve warnings elsewhere (in calling code)
* SharedStreamConfig
* SingleTagData
* SubscriberCore
* SuperMapStream
* SystemHeartbeatPublisher
* SystemStatsPublisher
* ThreadSafeQueue
* TradeBarsStream

fix build (warn): Only use Registry on windows
2025-11-21 14:54:58 -05:00
John Colvin a442e27d3d cleanup: Modernize PubSubRoute code, json parsing 2025-11-21 14:20:49 -05:00
John Colvin c3da5a5d0c cleanup: Remove unused DateTime grammar from exp parser 2025-11-21 14:20:48 -05:00
John Colvin 13fc7e0455 cleanup: Remove empty file (clearly added by accident) 2025-11-21 14:20:48 -05:00
John Colvin 0a8f002a6e fix: Remove query from literal error stream
The parser can't handle colons in the expression (it throws/crashes)
2025-11-21 14:09:28 -05:00
John Colvin e2d45ae8e1 feat: Extend dsl to support explicit param names
When using explicit names, the argument order doesn't matter, and args
with defaults are automatically populated if omitted.
2025-11-21 14:09:28 -05:00
John Colvin cd2a61420d chore: Add async SQLiteEnumerator to read table as IBlueRowCollection 2025-11-21 14:04:47 -05:00
John Colvin c2559603a8 chore: Add various collection classes
These will be useful in stream operators, row indexing, etc:
  * AggregateList
  * CircularBuffer
  * DeltaList
  * DictionarySetWrapper
  * ImmutableOrderedDictionary
  * ImmutableOrderedSet
  * IMutableCollectionFacade
    - inherits IReadOnlyCollection<T> and ICollection<T>
    - throws NotSupportedException in default impls for mutation ops
  * IMutableListFacade
    - inherits IReadOnlyList<T> and IList<T>
    - throws NotSupportedException in default impls for mutation ops
  * IndexDependentSelectorList
  * MergedSortedList
  * OffsetList
  * OrderedDictionary
  * ReverseArrayList
  * SliceList
  * StepList

More notes:
  * Renames ExceptionAssert.is_valid ExceptionAssert.Index.IsValid.
    - Like the 'ExceptionAssert.Argument' class, which provides methods to throw
      ArgumentException if a condition isn't satisfied, this class throws
      IndexOutOfRange exceptions.
2025-11-21 13:56:36 -05:00
John Colvin 8ca354e5d7 chore: Support mutations in AggregateDictionary (optionally) 2025-11-21 13:56:36 -05:00
John Colvin 0a4a7693ac style: Use standard c# capitalization rules
* Use 'PascalCase' for public members (instead of 'snake_case')
* Use 'camelCase' for private members (instead of '_snake_case')
2025-11-21 13:56:26 -05:00
John Colvin 44be99ef7a fix: Support DateTime/TimeSpan columns in DataPacketJsonSerializer 2025-11-21 12:56:10 -05:00
John Colvin 10bb90c731 fix: Use IDictionary.GetEnumerator explicitly to iterate keyvals
We really need to replace these old non-generic collections with generic ones.
BlueRow's IDictionaryEnumerator implementation iterates over values when passed
to a foreach loop, not key value pairs.
2025-11-19 23:55:35 -05:00
John Colvin 0746b27a40 cleanup: Remove docker submodules again
I lost count of the number of times they have been deleted long ago.
Taha, please remove them from your repo so they don't come back :)
2025-11-19 10:33:31 -05:00
John Colvin 3a3c3d45d3 cleanup: Change default DSQA sparkd config server to 127.0.0.1 2025-11-19 10:33:31 -05:00
John Colvin 2014931653 feat: Add live bbo stream 2025-11-19 10:33:31 -05:00
John Colvin aa740e786c refactor: Automatically add id column in BlueTableSchemaBuilder 2025-11-19 00:36:56 -05:00
John Colvin 12c6b367f4 fix (preemptive): Avoid StackOverflow in md client securities fn
One of the 'securities_incremental' fns was calling back into itself instead of
the other overload. This is a 'preemptive' fix because the problematic code path
wasn't running in practice (though it was a ticking time bomb).
2025-11-19 00:36:56 -05:00
John Colvin 5080960b97 refactor: Replace DataPacket w/ IDataPacket in stream handlers
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-19 00:36:56 -05:00
John Colvin 770e14752a chore: Add json facilities for DataStream websocket api 2025-11-19 00:36:56 -05:00