Commit Graph
12711 Commits
Author SHA1 Message Date
John Colvin 6986cb5e8e fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2026-01-04 21:08:12 +00:00
John Colvin 35a45c35ba refactor: Move position model to spark_api
So we can use it in the new sdk (c# server-side api) for Ophil
2026-01-04 21:08:12 +00:00
John Colvin 970555d2a1 diag: Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2026-01-04 21:08:12 +00:00
John Colvin 1d46ea061a cleanup: Remove unused code from ViewStreamDatabase 2026-01-04 21:08:12 +00:00
John Colvin 4bd12dc465 fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2026-01-04 21:08:12 +00:00
John Colvin ad1d404314 cleanup: Remove unnecessary deps from csprojs 2026-01-04 21:08:12 +00:00
John Colvin a7b6da27ae fix: Catch exception when creating TcpClient 2026-01-04 21:08:12 +00:00
John Colvin de4a55b34c fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2026-01-04 21:08:12 +00:00
John Colvin 2e14598667 fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2026-01-04 21:08:12 +00:00
John Colvin 3038d9154f improve: Reduce task wait/timeout code in AsyncTCPConnector 2026-01-04 21:08:12 +00:00
John Colvin 5805758187 diag: Assert System publishers 'doWork' is always called on same thread 2026-01-04 21:08:12 +00:00
John Colvin 2c1b7b95a4 improve: Propagate DataStream exceptions via ErrorChanged event 2026-01-04 21:08:12 +00:00
John Colvin 60af47c4eb cleanup: Remove unused PubSubConfig props 2026-01-04 21:08:12 +00:00
John Colvin b0d0594e3d fix: Always call 'handle_internal_pubs' in PubSubMessageReactor main loop 2026-01-04 21:08:12 +00:00
John Colvin b29b2a0adb fix: Fix a few column types in in PubSubServerTagStatsStream
* total_bytes      (int      -> long)
* last_update_time (DateTime -> long)
2026-01-04 21:08:11 +00:00
John Colvin 76fa35382e refactor: Add DataStream service modules for Dependency Injection
Namely, in DataStreamPublisher and PubSubServer. This is part of a broader
paradigm shift towards Dependency Injection and Factory Pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Use interfaces to decouple services:
  - Decouple ManagedSubscriber from ISubscriberCore impl
  - Decouple PubTableBroker from IPublisherCore impl
  - Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl

* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
* Model config profiles after dotnet appsettings
  - Rename Configurations/<name>.json -> appsettings.<name>.json
  - Make json fields match corresponding csharp props verbatim
    * The Dependency Injection framework doesn't support JsonPropertyName
      attributes. Config class props must match json fields verbatim.

* Move reusable startup code into new Bts.Extensions.Hosting proj
* Add pubsub and md-publisher services to compose.yaml
2026-01-04 21:08:11 +00:00
John Colvin 716bc92394 refactor: Only support 1 tag type in IPubSubServerTagHandler 2025-12-30 11:42:23 -05:00
John Colvin 3a6141f3ae refactor: Change Configuration.initialize to be async
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-12-30 11:42:20 -05:00
John Colvin be2368f95a refactor: Add Spark service modules for Dependency Injection 2025-12-30 11:38:21 -05:00
John Colvin 7e32ac9e06 cleanup: Remove unused pubsub tag handler code
* Removed 'IPubSubServerTagHandler.register'
  - All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
  - This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
  - No reason these interfaces have to be tied together
2025-12-30 11:38:21 -05:00
John Colvin 8fc4df1427 refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
This matches the file name
2025-12-30 11:38:21 -05:00
John Colvin 2020ebed2d cleanup: Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
In config json files (not actual code change)
2025-12-30 11:38:20 -05:00
John Colvin cc483ca4b5 diag: Fix typos in log msgs
* Bad string interopolation syntax
2025-12-30 11:38:20 -05:00
John Colvin 7d6e4efa4f cleanup: Fix numerous build warnings in DataStreamSuite
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.

* Compression related files
  - BZip2InputStream  (disable nullability warnings)
  - BZip2OutputStream (disable nullability warnings)
  - DeflaterHuffman   (disable nullability warnings)
  - Inflater          (disable nullability warnings)
  - InflaterDynHeader (disable nullability warnings)
  - ParallelJoinTest  (disable nullability warnings)
  - TarHeader
  - TarInputStream    (disable nullability warnings)
  - ZipEntry
  - ZipFile
  - ZipInputStream

* AsyncTCPConnection_APM
* AsyncTCPConnection_APM
* InflaterHuffmanTree
* ManagedSubscriber
* ManagedTCPMessageHub
* MapStream
* PubSubClientMessage
  - Normalize indentation/whitespace
  - Fix some typos
  - Address nullability issues to fix build warnings
  - Modernize code
* PubSubMessageReactor
* PubSubServerManagedTCPConnection
* PubSubSingleTagStateEvent
* PubSubState
* PublisherCore
* PubsubDeserializationController
* RemoteMessage
  - make 'message' nullable to resolve warnings elsewhere (in calling code)
* SharedStreamConfig
* SingleTagData
* SubscriberCore
* SuperMapStream
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SystemSubscriptionMapPublisher
* ThreadSafeQueue
* TradeBarsStream

Only use Registry on windows.
2025-12-30 11:38:20 -05:00
John Colvin 02f566a970 cleanup: Modernize PubSubRoute code, json parsing 2025-12-30 11:38:20 -05:00
John Colvin d0d05a5c18 cleanup: Remove unused DateTime grammar from exp parser 2025-12-30 11:38:20 -05:00
John Colvin 3fafeca9ad cleanup: Remove empty file (clearly added by accident) 2025-12-30 11:38:20 -05:00
John Colvin f5a4e7c410 fix: Remove query from literal error stream
The parser can't handle colons in the expression (it throws/crashes)
2025-12-30 11:38:17 -05:00
John Colvin a0b2a8b186 chore: Add async SQLiteEnumerator to read table as IBlueRowCollection 2025-12-30 11:36:35 -05:00
John Colvin 35a207e234 chore: Add various collection classes
These will be useful in stream operators, row indexing, etc:
  * AggregateList
  * CircularBuffer
  * DeltaList
  * DictionarySetWrapper
  * ImmutableOrderedDictionary
  * ImmutableOrderedSet
  * IMutableCollectionFacade
    - inherits IReadOnlyCollection<T> and ICollection<T>
    - throws NotSupportedException in default impls for mutation ops
  * IMutableListFacade
    - inherits IReadOnlyList<T> and IList<T>
    - throws NotSupportedException in default impls for mutation ops
  * IndexDependentSelectorList
  * MergedSortedList
  * OffsetList
  * OrderedDictionary
  * ReverseArrayList
  * SliceList
  * StepList

More notes:
  * Renames ExceptionAssert.is_valid ExceptionAssert.Index.IsValid.
    - Like the 'ExceptionAssert.Argument' class, which provides methods to throw
      ArgumentException if a condition isn't satisfied, this class throws
      IndexOutOfRange exceptions.
2025-12-30 11:36:35 -05:00
John Colvin 2069c64864 chore: Support mutations in AggregateDictionary (optionally) 2025-12-30 11:36:35 -05:00
arham e69012a5ad fix: sanitized dsl in websocket 2025-12-03 13:35:08 +00:00
arham 4237d58f76 Add: metadata to dsl callback 2025-12-03 13:34:37 +00:00
arham 1c75f4a53f Add: get last n lines of log for websocketserver 2025-12-03 12:24:16 +00:00
arham aabf1490eb Fix: Trycatch on base program.cs to prevent crash 2025-12-03 12:21:03 +00:00
arham 14a79b6352 Add:: cross exchange arb stream 2025-12-02 15:21:06 -05:00
arham f69b517f83 add: f10 to stop dsqa 2025-12-02 13:05:39 -05:00
arham 2d0d33028a fix: bug fix to rowbasedappendstream 2025-12-02 12:58:22 -05:00
arham f953b3bb8a change: change UnionStream name to AppendStream 2025-12-02 12:47:36 -05:00
arham 73672276f3 Add: add year to datetimexpander stream 2025-12-02 12:46:49 -05:00
John Colvin 10bb90c731 fix: Use IDictionary.GetEnumerator explicitly to iterate keyvals
We really need to replace these old non-generic collections with generic ones.
BlueRow's IDictionaryEnumerator implementation iterates over values when passed
to a foreach loop, not key value pairs.
2025-11-19 23:55:35 -05:00
John Colvin 3a3c3d45d3 cleanup: Change default DSQA sparkd config server to 127.0.0.1 2025-11-19 10:33:31 -05:00
John Colvin 2014931653 feat: Add live bbo stream 2025-11-19 10:33:31 -05:00
John Colvin aa740e786c refactor: Automatically add id column in BlueTableSchemaBuilder 2025-11-19 00:36:56 -05:00
John Colvin 12c6b367f4 fix (preemptive): Avoid StackOverflow in md client securities fn
One of the 'securities_incremental' fns was calling back into itself instead of
the other overload. This is a 'preemptive' fix because the problematic code path
wasn't running in practice (though it was a ticking time bomb).
2025-11-19 00:36:56 -05:00
John Colvin 5080960b97 refactor: Replace DataPacket w/ IDataPacket in stream handlers
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-19 00:36:56 -05:00
John Colvin 770e14752a chore: Add json facilities for DataStream websocket api 2025-11-19 00:36:56 -05:00
John Colvin e24b877e58 Add gen purpose blue row/table utils for data streams
* Add BlueRowReflection<T> so ViewStream implementations can use
  strong row types without needing to implement IBlueRow
* Add default implementations for IDictionary members in IBlueRow to make
  implementing IBlueRow less tedious
2025-11-19 00:31:53 -05:00
Taha Afzal 09abe4a432 Added RowBasedAppendStream 2025-11-18 03:10:30 +00:00
Taha Afzal 66c0baa150 Adding meta data to DataPacket, NLStream adds nl -> dsl as a name value pair in the meta data. Also removing windows ref from DataStreamStudio.sln 2025-09-24 13:16:28 +00:00