Commit Graph
63 Commits
Author SHA1 Message Date
John Colvin 8f3fa18bca refactor: Rename spark_api csproj Bts.Spark.Api
This follows the conventional naming practices, where the project shares its
name with the namespace. Doing this in anticipation of the csharp sdk we are
about to release (for Ophir).
2026-01-04 21:08:13 +00:00
John Colvin 3565f78be8 cleanup (build): Remove redundant csproj refs
For example, projects that reference spark_api don't need an explicit reference
to spark_core, since spark_api already references it. The same goes for projects
that reference DataStream and DataStreamCore.
2026-01-04 21:08:13 +00:00
John Colvin 856cd552eb refactor: Add Api Server service modules for Dependency Injection
* Part of a broader paradigm shift towards Dependency Injection and
  Factory Pattern.
* Adds "api-server" service to docker compose
* Replaces MarketDataClient w/ IMarketDataClient in a few streams
* Replaces ViewStreamDatabase w/ IWorkScheduler in a few streams
2026-01-04 21:08:12 +00:00
John Colvin b29b2a0adb fix: Fix a few column types in in PubSubServerTagStatsStream
* total_bytes      (int      -> long)
* last_update_time (DateTime -> long)
2026-01-04 21:08:11 +00:00
John Colvin 76fa35382e refactor: Add DataStream service modules for Dependency Injection
Namely, in DataStreamPublisher and PubSubServer. This is part of a broader
paradigm shift towards Dependency Injection and Factory Pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Use interfaces to decouple services:
  - Decouple ManagedSubscriber from ISubscriberCore impl
  - Decouple PubTableBroker from IPublisherCore impl
  - Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl

* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
* Model config profiles after dotnet appsettings
  - Rename Configurations/<name>.json -> appsettings.<name>.json
  - Make json fields match corresponding csharp props verbatim
    * The Dependency Injection framework doesn't support JsonPropertyName
      attributes. Config class props must match json fields verbatim.

* Move reusable startup code into new Bts.Extensions.Hosting proj
* Add pubsub and md-publisher services to compose.yaml
2026-01-04 21:08:11 +00:00
John Colvin 7d6e4efa4f cleanup: Fix numerous build warnings in DataStreamSuite
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.

* Compression related files
  - BZip2InputStream  (disable nullability warnings)
  - BZip2OutputStream (disable nullability warnings)
  - DeflaterHuffman   (disable nullability warnings)
  - Inflater          (disable nullability warnings)
  - InflaterDynHeader (disable nullability warnings)
  - ParallelJoinTest  (disable nullability warnings)
  - TarHeader
  - TarInputStream    (disable nullability warnings)
  - ZipEntry
  - ZipFile
  - ZipInputStream

* AsyncTCPConnection_APM
* AsyncTCPConnection_APM
* InflaterHuffmanTree
* ManagedSubscriber
* ManagedTCPMessageHub
* MapStream
* PubSubClientMessage
  - Normalize indentation/whitespace
  - Fix some typos
  - Address nullability issues to fix build warnings
  - Modernize code
* PubSubMessageReactor
* PubSubServerManagedTCPConnection
* PubSubSingleTagStateEvent
* PubSubState
* PublisherCore
* PubsubDeserializationController
* RemoteMessage
  - make 'message' nullable to resolve warnings elsewhere (in calling code)
* SharedStreamConfig
* SingleTagData
* SubscriberCore
* SuperMapStream
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SystemSubscriptionMapPublisher
* ThreadSafeQueue
* TradeBarsStream

Only use Registry on windows.
2025-12-30 11:38:20 -05:00
John Colvin 02f566a970 cleanup: Modernize PubSubRoute code, json parsing 2025-12-30 11:38:20 -05:00
John Colvin a0b2a8b186 chore: Add async SQLiteEnumerator to read table as IBlueRowCollection 2025-12-30 11:36:35 -05:00
arham 4237d58f76 Add: metadata to dsl callback 2025-12-03 13:34:37 +00:00
arham 14a79b6352 Add:: cross exchange arb stream 2025-12-02 15:21:06 -05:00
John Colvin 2014931653 feat: Add live bbo stream 2025-11-19 10:33:31 -05:00
Taha Afzal 66c0baa150 Adding meta data to DataPacket, NLStream adds nl -> dsl as a name value pair in the meta data. Also removing windows ref from DataStreamStudio.sln 2025-09-24 13:16:28 +00:00
arham 5305b32c04 Fix file path didnt work in linux 2025-09-17 04:28:25 +00:00
arham e6304609f8 changes to make streams run on linux 2025-09-16 15:38:21 +00:00
arham 21d84d811d simple changes to make stream work with wbesite 2025-09-02 17:17:43 +00:00
Taha 6b28fc4779 Lot of BlueOrb stream changes 2025-09-02 17:17:43 +00:00
Taha 0858eaac23 Polishing BlueOrb streams 2025-07-13 21:21:10 -04:00
Taha 70fae06a30 BlueOrb streams and MLP 2025-06-08 21:52:38 -04:00
Taha a130e47218 Arb Scoring stream 2025-05-20 00:24:17 -04:00
Taha 0a6733180d Added SpreadStatsStream 2025-05-18 01:10:33 -04:00
Taha f2169c6bd4 Adding more data to SpreadPerformanceStream 2025-05-16 01:42:32 -04:00
Taha 23ef478391 Spread analytics streams 2025-05-15 15:01:27 -04:00
John Colvin a6e8853cb9 fix build (warns): Account for nullable ref types 2025-04-28 17:12:58 +00:00
John Colvin 043efab5a2 build: Upgrade c# to net8.0 2025-04-28 17:12:58 +00:00
Taha 058b131760 Cleaning up some streams,new prompt etc 2025-04-04 15:36:52 -04:00
Taha d13ba2968a DateTimeExpanstionStream col name changes 2025-01-20 15:34:11 -05:00
Taha 461cbad338 Adding FakeBBO stream 2024-11-27 13:36:14 -05:00
Taha aab8e83543 Some rewrite of streams that fetch bars, and making the other business streams thin wrapper over stream primitives 2024-11-26 22:33:26 -05:00
Taha 6163887cd3 Bunch of CME queries are now working 2024-10-30 13:36:23 -04:00
Taha f9aa9dd982 Merge branch 'development' of bitbucket.org:bluetradesys/spark into development 2024-10-15 23:48:34 -04:00
Taha 22f1ab44f9 Basic set of streams that can talks ot the LLM and get DSL back from few-shot prompt 2024-10-15 23:38:49 -04:00
Taha 288b7712db Added and modified more streams to address teh CME NL queries 2024-10-09 13:59:01 -04:00
Taha 31df1a3827 Added benzinga streamd to MDPub's test config 2024-07-17 14:32:57 -04:00
Taha 7e219aec7c Merging 2024-07-17 02:06:17 -04:00
Taha 37f8b98df9 Icomplete SharedStreamsConfig was breaking build, fixing 2024-07-17 02:03:43 -04:00
Taha f06beff22f Added new benzinga streams 2024-07-17 02:02:01 -04:00
Taha Afzal 233e2576af Merging 2024-07-05 21:21:27 +00:00
Taha 050e21e22b Adding TableMetaDataStream 2024-07-04 23:39:04 -04:00
Taha Afzal 672ad6d237 Cleaning Config 1.0 (not completely done) 2024-06-30 23:17:20 -04:00
Taha 0a8f6e866f Added pubsub routes to the Title of DSQA 2024-06-27 14:04:07 -04:00
Taha 393f304be4 some buf gixes + cleanup 2024-06-26 00:23:19 -04:00
Taha 87b6a2044c PubSubServer's linux issues fixed 2024-06-23 21:13:37 -04:00
Taha fac0a9c25b RemoteQueryStream 2024-06-06 09:20:05 -05:00
John Colvin 6240d89a08 Map PubSubServerConfig json to historic PubSub3Config 2024-06-03 13:46:51 +00:00
John Colvin 9050318ceb refactor: Replace HighPerformanceTimer w/ Stopwatch
This is needed to build DataStreamSuite on Linux.

'Stopwatch' is a platform independent timer class included with the
System.Runtime library in modern dotnet. 'HighPerformanceTimer' is a BTS
class that predates Stopwatch, and is NOT platform independent as it
relies on Windows kernel32.dll QueryPerformanceCounter/Frequency fns.
2024-06-03 13:46:51 +00:00
Taha Afzal 7104e90fc9 Adding tests for stream serialization 2024-05-25 15:37:33 +00:00
Taha Afzal 07274f897a Pubsub stats 2024-05-25 15:37:33 +00:00
Taha Afzal 03e80dabcb CHanging member names from old style 2024-05-25 15:37:33 +00:00
Taha Afzal 54184d0875 Cleaning up ISubscriberCore to use ne PubSubClientMssage class 2024-05-25 15:37:33 +00:00
Taha Afzal 8c1342230b RowDiff added 2024-02-08 14:11:10 -05:00