Commit Graph
71 Commits
Author SHA1 Message Date
John Colvin 59b1aead62 refactor: Use HostBuilder to launch spark w/ streams 2026-01-04 21:19:00 +00:00
John Colvin c4a6b4591d cleanup: Remove last vestiges of spark proto config from DataStreamCore 2026-01-04 21:08:13 +00:00
John Colvin 1c579ee063 refactor: Change namespace to Bts.DataStreamCore for a few types
Since they're in the DataStreamCore csproj
2026-01-04 21:08:12 +00:00
John Colvin 2e14598667 fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2026-01-04 21:08:12 +00:00
John Colvin 2c1b7b95a4 improve: Propagate DataStream exceptions via ErrorChanged event 2026-01-04 21:08:12 +00:00
John Colvin 76fa35382e refactor: Add DataStream service modules for Dependency Injection
Namely, in DataStreamPublisher and PubSubServer. This is part of a broader
paradigm shift towards Dependency Injection and Factory Pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Use interfaces to decouple services:
  - Decouple ManagedSubscriber from ISubscriberCore impl
  - Decouple PubTableBroker from IPublisherCore impl
  - Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl

* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
* Model config profiles after dotnet appsettings
  - Rename Configurations/<name>.json -> appsettings.<name>.json
  - Make json fields match corresponding csharp props verbatim
    * The Dependency Injection framework doesn't support JsonPropertyName
      attributes. Config class props must match json fields verbatim.

* Move reusable startup code into new Bts.Extensions.Hosting proj
* Add pubsub and md-publisher services to compose.yaml
2026-01-04 21:08:11 +00:00
John Colvin 7d6e4efa4f cleanup: Fix numerous build warnings in DataStreamSuite
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.

* Compression related files
  - BZip2InputStream  (disable nullability warnings)
  - BZip2OutputStream (disable nullability warnings)
  - DeflaterHuffman   (disable nullability warnings)
  - Inflater          (disable nullability warnings)
  - InflaterDynHeader (disable nullability warnings)
  - ParallelJoinTest  (disable nullability warnings)
  - TarHeader
  - TarInputStream    (disable nullability warnings)
  - ZipEntry
  - ZipFile
  - ZipInputStream

* AsyncTCPConnection_APM
* AsyncTCPConnection_APM
* InflaterHuffmanTree
* ManagedSubscriber
* ManagedTCPMessageHub
* MapStream
* PubSubClientMessage
  - Normalize indentation/whitespace
  - Fix some typos
  - Address nullability issues to fix build warnings
  - Modernize code
* PubSubMessageReactor
* PubSubServerManagedTCPConnection
* PubSubSingleTagStateEvent
* PubSubState
* PublisherCore
* PubsubDeserializationController
* RemoteMessage
  - make 'message' nullable to resolve warnings elsewhere (in calling code)
* SharedStreamConfig
* SingleTagData
* SubscriberCore
* SuperMapStream
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SystemSubscriptionMapPublisher
* ThreadSafeQueue
* TradeBarsStream

Only use Registry on windows.
2025-12-30 11:38:20 -05:00
John Colvin a0b2a8b186 chore: Add async SQLiteEnumerator to read table as IBlueRowCollection 2025-12-30 11:36:35 -05:00
arham 2d0d33028a fix: bug fix to rowbasedappendstream 2025-12-02 12:58:22 -05:00
John Colvin 5080960b97 refactor: Replace DataPacket w/ IDataPacket in stream handlers
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-19 00:36:56 -05:00
Taha Afzal 09abe4a432 Added RowBasedAppendStream 2025-11-18 03:10:30 +00:00
arham 5305b32c04 Fix file path didnt work in linux 2025-09-17 04:28:25 +00:00
arham e6304609f8 changes to make streams run on linux 2025-09-16 15:38:21 +00:00
Taha 6b28fc4779 Lot of BlueOrb stream changes 2025-09-02 17:17:43 +00:00
Taha 0858eaac23 Polishing BlueOrb streams 2025-07-13 21:21:10 -04:00
Taha 70fae06a30 BlueOrb streams and MLP 2025-06-08 21:52:38 -04:00
Taha a130e47218 Arb Scoring stream 2025-05-20 00:24:17 -04:00
Taha 0a6733180d Added SpreadStatsStream 2025-05-18 01:10:33 -04:00
Taha 23ef478391 Spread analytics streams 2025-05-15 15:01:27 -04:00
Taha 058b131760 Cleaning up some streams,new prompt etc 2025-04-04 15:36:52 -04:00
Taha d13ba2968a DateTimeExpanstionStream col name changes 2025-01-20 15:34:11 -05:00
Taha 461cbad338 Adding FakeBBO stream 2024-11-27 13:36:14 -05:00
Taha aab8e83543 Some rewrite of streams that fetch bars, and making the other business streams thin wrapper over stream primitives 2024-11-26 22:33:26 -05:00
Taha 6163887cd3 Bunch of CME queries are now working 2024-10-30 13:36:23 -04:00
Taha f9aa9dd982 Merge branch 'development' of bitbucket.org:bluetradesys/spark into development 2024-10-15 23:48:34 -04:00
Taha 22f1ab44f9 Basic set of streams that can talks ot the LLM and get DSL back from few-shot prompt 2024-10-15 23:38:49 -04:00
Taha 288b7712db Added and modified more streams to address teh CME NL queries 2024-10-09 13:59:01 -04:00
Taha 322c06bfbe Fixing build relted to BCG vs bts namespace 2024-08-21 16:24:22 -04:00
Taha b68fceb362 Cleaning up FileStreams 2024-08-21 16:11:57 -04:00
Taha f06beff22f Added new benzinga streams 2024-07-17 02:02:01 -04:00
Taha 050e21e22b Adding TableMetaDataStream 2024-07-04 23:39:04 -04:00
Taha Afzal 672ad6d237 Cleaning Config 1.0 (not completely done) 2024-06-30 23:17:20 -04:00
Taha 393f304be4 some buf gixes + cleanup 2024-06-26 00:23:19 -04:00
Taha 87b6a2044c PubSubServer's linux issues fixed 2024-06-23 21:13:37 -04:00
Taha fac0a9c25b RemoteQueryStream 2024-06-06 09:20:05 -05:00
John Colvin 1824d5f9a4 feat: Add [incremental] MongoCollectionStream 2024-06-03 13:46:51 +00:00
John Colvin 5741ec9ab2 cleanup: Remove unnecessary usings (namespaces) 2024-06-03 13:46:51 +00:00
John Colvin 9050318ceb refactor: Replace HighPerformanceTimer w/ Stopwatch
This is needed to build DataStreamSuite on Linux.

'Stopwatch' is a platform independent timer class included with the
System.Runtime library in modern dotnet. 'HighPerformanceTimer' is a BTS
class that predates Stopwatch, and is NOT platform independent as it
relies on Windows kernel32.dll QueryPerformanceCounter/Frequency fns.
2024-06-03 13:46:51 +00:00
John Colvin e95997ea17 cleanup: Chip away a few nullability warnings in DataStreamSuite
This is a start, but there're still > 700 of them!
2024-05-25 15:37:33 +00:00
Taha Afzal 90ec64d25c deleting old serializers 2024-05-25 15:37:33 +00:00
Taha Afzal 061481419e Streams can be published and subscribed now 2024-05-25 15:37:33 +00:00
Taha Afzal 8035255868 Cleaned up the cashe held by PublisherCore 2024-05-25 15:37:33 +00:00
Taha Afzal 7feaf869dc Stream serialization revamp, all tests pass (TestsStream.s) 2024-05-25 15:37:33 +00:00
Taha Afzal 7104e90fc9 Adding tests for stream serialization 2024-05-25 15:37:33 +00:00
Taha Afzal 07274f897a Pubsub stats 2024-05-25 15:37:33 +00:00
Taha Afzal f626e53415 CPublisherCore, SubscriberCore, PubsubMessageReactor claned 2024-05-25 15:37:33 +00:00
Taha Afzal 90bcf5a455 Adding testss for PubSubMessages 2024-05-25 15:37:33 +00:00
Taha Afzal 4510e120ab enabling some streams like max, that were dependent on AddCOlumns (thta John fixed in the last commit) 2024-02-12 13:21:28 -05:00
Taha Afzal da2af06ee6 Merging 2024-02-08 15:48:51 -05:00
John Colvin 8660dea85a build: Downgrade new DataStream code from net8 -> net7
The AddColumnsStream, SortStream, and requisite yacc/lex grammar were
originally added to a new branch, off of development, which is on net8.
I forgot to change it to net7 when I cherry-picked the commits for the
regular development branch.

It's wasn't quite as simple as flipping net8 -> net7 in csproj files,
because I was eager to use new language features only available in net8!
Namely, primary constructors and collection expressions.
2024-02-08 14:24:31 -06:00