Commit Graph
123 Commits
Author SHA1 Message Date
John Colvin 5d752f3b73 refactor: Replace DataPacket with IDataPacket in more places 2026-03-26 16:20:40 -04:00
John Colvin c3341ec29f refactor: Replace a few more DataPacket refs with IDataPacket 2026-02-23 12:56:27 -05:00
John Colvin 54c97b5610 fix: Set buffer Position to 0 when deserializing pubsub echo payload 2026-02-21 16:36:53 -05:00
John Colvin 5c05afd9b3 fix: Only fire stream update if there are actually changes
In other words, don't fire if addedRows and deletedRows are both empty.
2026-02-21 16:36:53 -05:00
John Colvin d09f22007e fix: Ensure PubSubTagHandlerFactory uses correct config
* Add this service as part of the PubSubClient services for dep
  injection (it is needed by PublisherCore)

* Get rid of PublisherCore's default constructor, so dep injection
  treats the tag handler factory as a required param (fails if it isn't
  available)
  - Replace the default ctr with a static "Create" fn, so code that
    hasn't been updated to use dependency injection can still use that.
  - Use PubSubConfig.Instance (runtime) instead of compile time Default

* Fix compile time Default config
  - In fallback handler config, change Tag from "" to ".*" and move it
    from the beginning of the list to the end (it should be the last
    resort, not the other way around).
2026-02-21 16:36:53 -05:00
John Colvin 05c7a32fc5 refactor: Add parameterless ctr to PubsubEchoTestStream
So it's quick/easy to check in DSQA (less typing required)
2026-02-21 14:23:51 -05:00
John Colvin 0bc021912a fix: Replace more DataPacket references with IDataPacket 2026-02-20 02:13:17 -05:00
John Colvin 88beb0f148 fix: Avoid null ref ex in DiffStream 2026-01-11 15:15:13 -05:00
John Colvin bcd207f89e cleanup: Fix more build warnings in DataStreamSuite
Alot more
2026-01-11 15:15:13 -05:00
John Colvin aacd22d496 refactor: Move Exception utils to Spark.Core.Diagnostics ns 2026-01-11 15:15:12 -05:00
John Colvin d89164c0ff refactor: Move CollectionExtensions to Spark.Core.Collections namespace 2026-01-11 15:12:33 -05:00
John Colvin cd34f9b1cb cleanup: Tidy up SecondStream, move ThrottleStream to own file
There actually already was a file called "ThrottleStream", but since it
didn't have the ".cs" extension it wasn't getting built.
2026-01-11 11:20:40 -05:00
John Colvin 5f689dbd8e cleanup: Tidy whitespace and using statements 2026-01-11 11:20:40 -05:00
John Colvin 03a8f4e9c0 refactor: Replace DataPacket w/ IDataPacket in many places 2026-01-11 11:20:40 -05:00
John Colvin 9de4b27003 refactor: Ensure stream subscriptions only disposed once
In the debugger, I noticed a few streams were unhooking their internal
onChange callbacks unnecessarily (when they were already unhooked).
2026-01-10 12:59:14 -05:00
John Colvin 849c6c28e5 fix: Don't fire empty updates in MutableAppendStream 2026-01-10 12:59:14 -05:00
John Colvin 44749739b9 build: Define centralized SparkVersion prop
All csprojs should be on the same version. At the very least, it
is a sensible default.
2026-01-10 12:59:14 -05:00
John Colvin bb5ea19f69 refactor: Get rid of ViewStream.rows prop
It was causing weird side effects in debugger
2026-01-10 12:59:14 -05:00
John Colvin 327665b4c3 fix: Add null check in ProximityJoinStream 2026-01-08 14:24:27 -05:00
John Colvin 2843638386 fix: Only start Stopwatch when actually doing work 2026-01-08 14:24:27 -05:00
John Colvin bb1e7c30c8 build: Add msbuild flag to reference projs or local nugets
So we can use project refs for dev builds, and package refs for releases.
2026-01-04 21:19:00 +00:00
John Colvin 59b1aead62 refactor: Use HostBuilder to launch spark w/ streams 2026-01-04 21:19:00 +00:00
John Colvin a515468e37 refactor: Rename TableUpdate props to PascalCase
Makes the sdk look more polished, professional
2026-01-04 21:19:00 +00:00
John Colvin c4a6b4591d cleanup: Remove last vestiges of spark proto config from DataStreamCore 2026-01-04 21:08:13 +00:00
John Colvin 3565f78be8 cleanup (build): Remove redundant csproj refs
For example, projects that reference spark_api don't need an explicit reference
to spark_core, since spark_api already references it. The same goes for projects
that reference DataStream and DataStreamCore.
2026-01-04 21:08:13 +00:00
John Colvin 1c579ee063 refactor: Change namespace to Bts.DataStreamCore for a few types
Since they're in the DataStreamCore csproj
2026-01-04 21:08:12 +00:00
John Colvin 0253a0f60f refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2026-01-04 21:08:12 +00:00
John Colvin 2e14598667 fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2026-01-04 21:08:12 +00:00
John Colvin 2c1b7b95a4 improve: Propagate DataStream exceptions via ErrorChanged event 2026-01-04 21:08:12 +00:00
John Colvin 76fa35382e refactor: Add DataStream service modules for Dependency Injection
Namely, in DataStreamPublisher and PubSubServer. This is part of a broader
paradigm shift towards Dependency Injection and Factory Pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Use interfaces to decouple services:
  - Decouple ManagedSubscriber from ISubscriberCore impl
  - Decouple PubTableBroker from IPublisherCore impl
  - Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl

* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
* Model config profiles after dotnet appsettings
  - Rename Configurations/<name>.json -> appsettings.<name>.json
  - Make json fields match corresponding csharp props verbatim
    * The Dependency Injection framework doesn't support JsonPropertyName
      attributes. Config class props must match json fields verbatim.

* Move reusable startup code into new Bts.Extensions.Hosting proj
* Add pubsub and md-publisher services to compose.yaml
2026-01-04 21:08:11 +00:00
John Colvin 8fc4df1427 refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
This matches the file name
2025-12-30 11:38:21 -05:00
John Colvin 7d6e4efa4f cleanup: Fix numerous build warnings in DataStreamSuite
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.

* Compression related files
  - BZip2InputStream  (disable nullability warnings)
  - BZip2OutputStream (disable nullability warnings)
  - DeflaterHuffman   (disable nullability warnings)
  - Inflater          (disable nullability warnings)
  - InflaterDynHeader (disable nullability warnings)
  - ParallelJoinTest  (disable nullability warnings)
  - TarHeader
  - TarInputStream    (disable nullability warnings)
  - ZipEntry
  - ZipFile
  - ZipInputStream

* AsyncTCPConnection_APM
* AsyncTCPConnection_APM
* InflaterHuffmanTree
* ManagedSubscriber
* ManagedTCPMessageHub
* MapStream
* PubSubClientMessage
  - Normalize indentation/whitespace
  - Fix some typos
  - Address nullability issues to fix build warnings
  - Modernize code
* PubSubMessageReactor
* PubSubServerManagedTCPConnection
* PubSubSingleTagStateEvent
* PubSubState
* PublisherCore
* PubsubDeserializationController
* RemoteMessage
  - make 'message' nullable to resolve warnings elsewhere (in calling code)
* SharedStreamConfig
* SingleTagData
* SubscriberCore
* SuperMapStream
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SystemSubscriptionMapPublisher
* ThreadSafeQueue
* TradeBarsStream

Only use Registry on windows.
2025-12-30 11:38:20 -05:00
John Colvin f5a4e7c410 fix: Remove query from literal error stream
The parser can't handle colons in the expression (it throws/crashes)
2025-12-30 11:38:17 -05:00
John Colvin a0b2a8b186 chore: Add async SQLiteEnumerator to read table as IBlueRowCollection 2025-12-30 11:36:35 -05:00
arham e69012a5ad fix: sanitized dsl in websocket 2025-12-03 13:35:08 +00:00
arham 2d0d33028a fix: bug fix to rowbasedappendstream 2025-12-02 12:58:22 -05:00
arham f953b3bb8a change: change UnionStream name to AppendStream 2025-12-02 12:47:36 -05:00
arham 73672276f3 Add: add year to datetimexpander stream 2025-12-02 12:46:49 -05:00
John Colvin 5080960b97 refactor: Replace DataPacket w/ IDataPacket in stream handlers
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-19 00:36:56 -05:00
Taha Afzal 09abe4a432 Added RowBasedAppendStream 2025-11-18 03:10:30 +00:00
arham 5305b32c04 Fix file path didnt work in linux 2025-09-17 04:28:25 +00:00
arham e6304609f8 changes to make streams run on linux 2025-09-16 15:38:21 +00:00
Taha 6b28fc4779 Lot of BlueOrb stream changes 2025-09-02 17:17:43 +00:00
Taha afd229b7c7 Merging 2025-07-15 17:51:28 -04:00
John Colvin 7e79a21cd4 Add some nifty DataPacket/TableUpdate IObservable utils 2025-07-15 10:13:53 -04:00
John Colvin 77caef2640 fix build: Add missing name to fn arg 2025-07-15 10:13:53 -04:00
Taha 767fa25d61 Adding comments 2025-07-14 15:18:03 -04:00
Taha 0858eaac23 Polishing BlueOrb streams 2025-07-13 21:21:10 -04:00
Taha 70fae06a30 BlueOrb streams and MLP 2025-06-08 21:52:38 -04:00
Taha a130e47218 Arb Scoring stream 2025-05-20 00:24:17 -04:00