Commit Graph
91 Commits
Author SHA1 Message Date
John Colvin 85ff7f412a cleanup: Fix a few more build warnings 2026-03-20 17:02:44 +00:00
John Colvin 8a3ad08456 refactor: Replace DataPacket with IDataPacket in more places 2026-03-20 17:02:44 +00:00
John Colvin 6c060f6171 cleanup: Fix indentation in TopStream
Whitespace changes
2026-03-08 16:00:50 +00:00
John Colvin dc5da87a65 cleanup: Fix cs build warnings 2026-03-08 16:00:49 +00:00
John Colvin 54c97b5610 fix: Set buffer Position to 0 when deserializing pubsub echo payload 2026-02-21 16:36:53 -05:00
John Colvin 5c05afd9b3 fix: Only fire stream update if there are actually changes
In other words, don't fire if addedRows and deletedRows are both empty.
2026-02-21 16:36:53 -05:00
John Colvin d09f22007e fix: Ensure PubSubTagHandlerFactory uses correct config
* Add this service as part of the PubSubClient services for dep
  injection (it is needed by PublisherCore)

* Get rid of PublisherCore's default constructor, so dep injection
  treats the tag handler factory as a required param (fails if it isn't
  available)
  - Replace the default ctr with a static "Create" fn, so code that
    hasn't been updated to use dependency injection can still use that.
  - Use PubSubConfig.Instance (runtime) instead of compile time Default

* Fix compile time Default config
  - In fallback handler config, change Tag from "" to ".*" and move it
    from the beginning of the list to the end (it should be the last
    resort, not the other way around).
2026-02-21 16:36:53 -05:00
John Colvin 05c7a32fc5 refactor: Add parameterless ctr to PubsubEchoTestStream
So it's quick/easy to check in DSQA (less typing required)
2026-02-21 14:23:51 -05:00
John Colvin 0bc021912a fix: Replace more DataPacket references with IDataPacket 2026-02-20 02:13:17 -05:00
John Colvin 88beb0f148 fix: Avoid null ref ex in DiffStream 2026-01-11 15:15:13 -05:00
John Colvin bcd207f89e cleanup: Fix more build warnings in DataStreamSuite
Alot more
2026-01-11 15:15:13 -05:00
John Colvin aacd22d496 refactor: Move Exception utils to Spark.Core.Diagnostics ns 2026-01-11 15:15:12 -05:00
John Colvin cd34f9b1cb cleanup: Tidy up SecondStream, move ThrottleStream to own file
There actually already was a file called "ThrottleStream", but since it
didn't have the ".cs" extension it wasn't getting built.
2026-01-11 11:20:40 -05:00
John Colvin 5f689dbd8e cleanup: Tidy whitespace and using statements 2026-01-11 11:20:40 -05:00
John Colvin 03a8f4e9c0 refactor: Replace DataPacket w/ IDataPacket in many places 2026-01-11 11:20:40 -05:00
John Colvin 9de4b27003 refactor: Ensure stream subscriptions only disposed once
In the debugger, I noticed a few streams were unhooking their internal
onChange callbacks unnecessarily (when they were already unhooked).
2026-01-10 12:59:14 -05:00
John Colvin 849c6c28e5 fix: Don't fire empty updates in MutableAppendStream 2026-01-10 12:59:14 -05:00
John Colvin bb5ea19f69 refactor: Get rid of ViewStream.rows prop
It was causing weird side effects in debugger
2026-01-10 12:59:14 -05:00
John Colvin 327665b4c3 fix: Add null check in ProximityJoinStream 2026-01-08 14:24:27 -05:00
John Colvin 2843638386 fix: Only start Stopwatch when actually doing work 2026-01-08 14:24:27 -05:00
John Colvin 59b1aead62 refactor: Use HostBuilder to launch spark w/ streams 2026-01-04 21:19:00 +00:00
John Colvin c4a6b4591d cleanup: Remove last vestiges of spark proto config from DataStreamCore 2026-01-04 21:08:13 +00:00
John Colvin 1c579ee063 refactor: Change namespace to Bts.DataStreamCore for a few types
Since they're in the DataStreamCore csproj
2026-01-04 21:08:12 +00:00
John Colvin 2e14598667 fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2026-01-04 21:08:12 +00:00
John Colvin 2c1b7b95a4 improve: Propagate DataStream exceptions via ErrorChanged event 2026-01-04 21:08:12 +00:00
John Colvin 76fa35382e refactor: Add DataStream service modules for Dependency Injection
Namely, in DataStreamPublisher and PubSubServer. This is part of a broader
paradigm shift towards Dependency Injection and Factory Pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Use interfaces to decouple services:
  - Decouple ManagedSubscriber from ISubscriberCore impl
  - Decouple PubTableBroker from IPublisherCore impl
  - Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl

* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
* Model config profiles after dotnet appsettings
  - Rename Configurations/<name>.json -> appsettings.<name>.json
  - Make json fields match corresponding csharp props verbatim
    * The Dependency Injection framework doesn't support JsonPropertyName
      attributes. Config class props must match json fields verbatim.

* Move reusable startup code into new Bts.Extensions.Hosting proj
* Add pubsub and md-publisher services to compose.yaml
2026-01-04 21:08:11 +00:00
John Colvin 7d6e4efa4f cleanup: Fix numerous build warnings in DataStreamSuite
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.

* Compression related files
  - BZip2InputStream  (disable nullability warnings)
  - BZip2OutputStream (disable nullability warnings)
  - DeflaterHuffman   (disable nullability warnings)
  - Inflater          (disable nullability warnings)
  - InflaterDynHeader (disable nullability warnings)
  - ParallelJoinTest  (disable nullability warnings)
  - TarHeader
  - TarInputStream    (disable nullability warnings)
  - ZipEntry
  - ZipFile
  - ZipInputStream

* AsyncTCPConnection_APM
* AsyncTCPConnection_APM
* InflaterHuffmanTree
* ManagedSubscriber
* ManagedTCPMessageHub
* MapStream
* PubSubClientMessage
  - Normalize indentation/whitespace
  - Fix some typos
  - Address nullability issues to fix build warnings
  - Modernize code
* PubSubMessageReactor
* PubSubServerManagedTCPConnection
* PubSubSingleTagStateEvent
* PubSubState
* PublisherCore
* PubsubDeserializationController
* RemoteMessage
  - make 'message' nullable to resolve warnings elsewhere (in calling code)
* SharedStreamConfig
* SingleTagData
* SubscriberCore
* SuperMapStream
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SystemSubscriptionMapPublisher
* ThreadSafeQueue
* TradeBarsStream

Only use Registry on windows.
2025-12-30 11:38:20 -05:00
John Colvin a0b2a8b186 chore: Add async SQLiteEnumerator to read table as IBlueRowCollection 2025-12-30 11:36:35 -05:00
arham 2d0d33028a fix: bug fix to rowbasedappendstream 2025-12-02 12:58:22 -05:00
John Colvin 5080960b97 refactor: Replace DataPacket w/ IDataPacket in stream handlers
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-19 00:36:56 -05:00
Taha Afzal 09abe4a432 Added RowBasedAppendStream 2025-11-18 03:10:30 +00:00
arham 5305b32c04 Fix file path didnt work in linux 2025-09-17 04:28:25 +00:00
arham e6304609f8 changes to make streams run on linux 2025-09-16 15:38:21 +00:00
Taha 6b28fc4779 Lot of BlueOrb stream changes 2025-09-02 17:17:43 +00:00
Taha 0858eaac23 Polishing BlueOrb streams 2025-07-13 21:21:10 -04:00
Taha 70fae06a30 BlueOrb streams and MLP 2025-06-08 21:52:38 -04:00
Taha a130e47218 Arb Scoring stream 2025-05-20 00:24:17 -04:00
Taha 0a6733180d Added SpreadStatsStream 2025-05-18 01:10:33 -04:00
Taha 23ef478391 Spread analytics streams 2025-05-15 15:01:27 -04:00
Taha 058b131760 Cleaning up some streams,new prompt etc 2025-04-04 15:36:52 -04:00
Taha d13ba2968a DateTimeExpanstionStream col name changes 2025-01-20 15:34:11 -05:00
Taha 461cbad338 Adding FakeBBO stream 2024-11-27 13:36:14 -05:00
Taha aab8e83543 Some rewrite of streams that fetch bars, and making the other business streams thin wrapper over stream primitives 2024-11-26 22:33:26 -05:00
Taha 6163887cd3 Bunch of CME queries are now working 2024-10-30 13:36:23 -04:00
Taha f9aa9dd982 Merge branch 'development' of bitbucket.org:bluetradesys/spark into development 2024-10-15 23:48:34 -04:00
Taha 22f1ab44f9 Basic set of streams that can talks ot the LLM and get DSL back from few-shot prompt 2024-10-15 23:38:49 -04:00
Taha 288b7712db Added and modified more streams to address teh CME NL queries 2024-10-09 13:59:01 -04:00
Taha 322c06bfbe Fixing build relted to BCG vs bts namespace 2024-08-21 16:24:22 -04:00
Taha b68fceb362 Cleaning up FileStreams 2024-08-21 16:11:57 -04:00
Taha f06beff22f Added new benzinga streams 2024-07-17 02:02:01 -04:00