John Colvin
59b1aead62
refactor: Use HostBuilder to launch spark w/ streams
2026-01-04 21:19:00 +00:00
John Colvin
c4a6b4591d
cleanup: Remove last vestiges of spark proto config from DataStreamCore
2026-01-04 21:08:13 +00:00
John Colvin
1c579ee063
refactor: Change namespace to Bts.DataStreamCore for a few types
...
Since they're in the DataStreamCore csproj
2026-01-04 21:08:12 +00:00
John Colvin
2e14598667
fix: Add a few null checks
...
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2026-01-04 21:08:12 +00:00
John Colvin
2c1b7b95a4
improve: Propagate DataStream exceptions via ErrorChanged event
2026-01-04 21:08:12 +00:00
John Colvin
76fa35382e
refactor: Add DataStream service modules for Dependency Injection
...
Namely, in DataStreamPublisher and PubSubServer. This is part of a broader
paradigm shift towards Dependency Injection and Factory Pattern.
https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines
* Use interfaces to decouple services:
- Decouple ManagedSubscriber from ISubscriberCore impl
- Decouple PubTableBroker from IPublisherCore impl
- Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
* Model config profiles after dotnet appsettings
- Rename Configurations/<name>.json -> appsettings.<name>.json
- Make json fields match corresponding csharp props verbatim
* The Dependency Injection framework doesn't support JsonPropertyName
attributes. Config class props must match json fields verbatim.
* Move reusable startup code into new Bts.Extensions.Hosting proj
* Add pubsub and md-publisher services to compose.yaml
2026-01-04 21:08:11 +00:00
John Colvin
7d6e4efa4f
cleanup: Fix numerous build warnings in DataStreamSuite
...
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.
* Compression related files
- BZip2InputStream (disable nullability warnings)
- BZip2OutputStream (disable nullability warnings)
- DeflaterHuffman (disable nullability warnings)
- Inflater (disable nullability warnings)
- InflaterDynHeader (disable nullability warnings)
- ParallelJoinTest (disable nullability warnings)
- TarHeader
- TarInputStream (disable nullability warnings)
- ZipEntry
- ZipFile
- ZipInputStream
* AsyncTCPConnection_APM
* AsyncTCPConnection_APM
* InflaterHuffmanTree
* ManagedSubscriber
* ManagedTCPMessageHub
* MapStream
* PubSubClientMessage
- Normalize indentation/whitespace
- Fix some typos
- Address nullability issues to fix build warnings
- Modernize code
* PubSubMessageReactor
* PubSubServerManagedTCPConnection
* PubSubSingleTagStateEvent
* PubSubState
* PublisherCore
* PubsubDeserializationController
* RemoteMessage
- make 'message' nullable to resolve warnings elsewhere (in calling code)
* SharedStreamConfig
* SingleTagData
* SubscriberCore
* SuperMapStream
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SystemSubscriptionMapPublisher
* ThreadSafeQueue
* TradeBarsStream
Only use Registry on windows.
2025-12-30 11:38:20 -05:00
John Colvin
a0b2a8b186
chore: Add async SQLiteEnumerator to read table as IBlueRowCollection
2025-12-30 11:36:35 -05:00
arham
2d0d33028a
fix: bug fix to rowbasedappendstream
2025-12-02 12:58:22 -05:00
John Colvin
5080960b97
refactor: Replace DataPacket w/ IDataPacket in stream handlers
...
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-19 00:36:56 -05:00
Taha Afzal
09abe4a432
Added RowBasedAppendStream
2025-11-18 03:10:30 +00:00
arham
5305b32c04
Fix file path didnt work in linux
2025-09-17 04:28:25 +00:00
arham
e6304609f8
changes to make streams run on linux
2025-09-16 15:38:21 +00:00
Taha
6b28fc4779
Lot of BlueOrb stream changes
2025-09-02 17:17:43 +00:00
Taha
0858eaac23
Polishing BlueOrb streams
2025-07-13 21:21:10 -04:00
Taha
70fae06a30
BlueOrb streams and MLP
2025-06-08 21:52:38 -04:00
Taha
a130e47218
Arb Scoring stream
2025-05-20 00:24:17 -04:00
Taha
0a6733180d
Added SpreadStatsStream
2025-05-18 01:10:33 -04:00
Taha
23ef478391
Spread analytics streams
2025-05-15 15:01:27 -04:00
Taha
058b131760
Cleaning up some streams,new prompt etc
2025-04-04 15:36:52 -04:00
Taha
d13ba2968a
DateTimeExpanstionStream col name changes
2025-01-20 15:34:11 -05:00
Taha
461cbad338
Adding FakeBBO stream
2024-11-27 13:36:14 -05:00
Taha
aab8e83543
Some rewrite of streams that fetch bars, and making the other business streams thin wrapper over stream primitives
2024-11-26 22:33:26 -05:00
Taha
6163887cd3
Bunch of CME queries are now working
2024-10-30 13:36:23 -04:00
Taha
f9aa9dd982
Merge branch 'development' of bitbucket.org:bluetradesys/spark into development
2024-10-15 23:48:34 -04:00
Taha
22f1ab44f9
Basic set of streams that can talks ot the LLM and get DSL back from few-shot prompt
2024-10-15 23:38:49 -04:00
Taha
288b7712db
Added and modified more streams to address teh CME NL queries
2024-10-09 13:59:01 -04:00
Taha
322c06bfbe
Fixing build relted to BCG vs bts namespace
2024-08-21 16:24:22 -04:00
Taha
b68fceb362
Cleaning up FileStreams
2024-08-21 16:11:57 -04:00
Taha
f06beff22f
Added new benzinga streams
2024-07-17 02:02:01 -04:00
Taha
050e21e22b
Adding TableMetaDataStream
2024-07-04 23:39:04 -04:00
Taha Afzal
672ad6d237
Cleaning Config 1.0 (not completely done)
2024-06-30 23:17:20 -04:00
Taha
393f304be4
some buf gixes + cleanup
2024-06-26 00:23:19 -04:00
Taha
87b6a2044c
PubSubServer's linux issues fixed
2024-06-23 21:13:37 -04:00
Taha
fac0a9c25b
RemoteQueryStream
2024-06-06 09:20:05 -05:00
John Colvin
1824d5f9a4
feat: Add [incremental] MongoCollectionStream
2024-06-03 13:46:51 +00:00
John Colvin
5741ec9ab2
cleanup: Remove unnecessary usings (namespaces)
2024-06-03 13:46:51 +00:00
John Colvin
9050318ceb
refactor: Replace HighPerformanceTimer w/ Stopwatch
...
This is needed to build DataStreamSuite on Linux.
'Stopwatch' is a platform independent timer class included with the
System.Runtime library in modern dotnet. 'HighPerformanceTimer' is a BTS
class that predates Stopwatch, and is NOT platform independent as it
relies on Windows kernel32.dll QueryPerformanceCounter/Frequency fns.
2024-06-03 13:46:51 +00:00
John Colvin
e95997ea17
cleanup: Chip away a few nullability warnings in DataStreamSuite
...
This is a start, but there're still > 700 of them!
2024-05-25 15:37:33 +00:00
Taha Afzal
90ec64d25c
deleting old serializers
2024-05-25 15:37:33 +00:00
Taha Afzal
061481419e
Streams can be published and subscribed now
2024-05-25 15:37:33 +00:00
Taha Afzal
8035255868
Cleaned up the cashe held by PublisherCore
2024-05-25 15:37:33 +00:00
Taha Afzal
7feaf869dc
Stream serialization revamp, all tests pass (TestsStream.s)
2024-05-25 15:37:33 +00:00
Taha Afzal
7104e90fc9
Adding tests for stream serialization
2024-05-25 15:37:33 +00:00
Taha Afzal
07274f897a
Pubsub stats
2024-05-25 15:37:33 +00:00
Taha Afzal
f626e53415
CPublisherCore, SubscriberCore, PubsubMessageReactor claned
2024-05-25 15:37:33 +00:00
Taha Afzal
90bcf5a455
Adding testss for PubSubMessages
2024-05-25 15:37:33 +00:00
Taha Afzal
4510e120ab
enabling some streams like max, that were dependent on AddCOlumns (thta John fixed in the last commit)
2024-02-12 13:21:28 -05:00
Taha Afzal
da2af06ee6
Merging
2024-02-08 15:48:51 -05:00
John Colvin
8660dea85a
build: Downgrade new DataStream code from net8 -> net7
...
The AddColumnsStream, SortStream, and requisite yacc/lex grammar were
originally added to a new branch, off of development, which is on net8.
I forgot to change it to net7 when I cherry-picked the commits for the
regular development branch.
It's wasn't quite as simple as flipping net8 -> net7 in csproj files,
because I was eager to use new language features only available in net8!
Namely, primary constructors and collection expressions.
2024-02-08 14:24:31 -06:00