John Colvin
5d752f3b73
refactor: Replace DataPacket with IDataPacket in more places
2026-03-26 16:20:40 -04:00
John Colvin
c3341ec29f
refactor: Replace a few more DataPacket refs with IDataPacket
2026-02-23 12:56:27 -05:00
John Colvin
54c97b5610
fix: Set buffer Position to 0 when deserializing pubsub echo payload
2026-02-21 16:36:53 -05:00
John Colvin
5c05afd9b3
fix: Only fire stream update if there are actually changes
...
In other words, don't fire if addedRows and deletedRows are both empty.
2026-02-21 16:36:53 -05:00
John Colvin
d09f22007e
fix: Ensure PubSubTagHandlerFactory uses correct config
...
* Add this service as part of the PubSubClient services for dep
injection (it is needed by PublisherCore)
* Get rid of PublisherCore's default constructor, so dep injection
treats the tag handler factory as a required param (fails if it isn't
available)
- Replace the default ctr with a static "Create" fn, so code that
hasn't been updated to use dependency injection can still use that.
- Use PubSubConfig.Instance (runtime) instead of compile time Default
* Fix compile time Default config
- In fallback handler config, change Tag from "" to ".*" and move it
from the beginning of the list to the end (it should be the last
resort, not the other way around).
2026-02-21 16:36:53 -05:00
John Colvin
05c7a32fc5
refactor: Add parameterless ctr to PubsubEchoTestStream
...
So it's quick/easy to check in DSQA (less typing required)
2026-02-21 14:23:51 -05:00
John Colvin
0bc021912a
fix: Replace more DataPacket references with IDataPacket
2026-02-20 02:13:17 -05:00
John Colvin
88beb0f148
fix: Avoid null ref ex in DiffStream
2026-01-11 15:15:13 -05:00
John Colvin
bcd207f89e
cleanup: Fix more build warnings in DataStreamSuite
...
Alot more
2026-01-11 15:15:13 -05:00
John Colvin
aacd22d496
refactor: Move Exception utils to Spark.Core.Diagnostics ns
2026-01-11 15:15:12 -05:00
John Colvin
d89164c0ff
refactor: Move CollectionExtensions to Spark.Core.Collections namespace
2026-01-11 15:12:33 -05:00
John Colvin
cd34f9b1cb
cleanup: Tidy up SecondStream, move ThrottleStream to own file
...
There actually already was a file called "ThrottleStream", but since it
didn't have the ".cs" extension it wasn't getting built.
2026-01-11 11:20:40 -05:00
John Colvin
5f689dbd8e
cleanup: Tidy whitespace and using statements
2026-01-11 11:20:40 -05:00
John Colvin
03a8f4e9c0
refactor: Replace DataPacket w/ IDataPacket in many places
2026-01-11 11:20:40 -05:00
John Colvin
9de4b27003
refactor: Ensure stream subscriptions only disposed once
...
In the debugger, I noticed a few streams were unhooking their internal
onChange callbacks unnecessarily (when they were already unhooked).
2026-01-10 12:59:14 -05:00
John Colvin
849c6c28e5
fix: Don't fire empty updates in MutableAppendStream
2026-01-10 12:59:14 -05:00
John Colvin
44749739b9
build: Define centralized SparkVersion prop
...
All csprojs should be on the same version. At the very least, it
is a sensible default.
2026-01-10 12:59:14 -05:00
John Colvin
bb5ea19f69
refactor: Get rid of ViewStream.rows prop
...
It was causing weird side effects in debugger
2026-01-10 12:59:14 -05:00
John Colvin
327665b4c3
fix: Add null check in ProximityJoinStream
2026-01-08 14:24:27 -05:00
John Colvin
2843638386
fix: Only start Stopwatch when actually doing work
2026-01-08 14:24:27 -05:00
John Colvin
bb1e7c30c8
build: Add msbuild flag to reference projs or local nugets
...
So we can use project refs for dev builds, and package refs for releases.
2026-01-04 21:19:00 +00:00
John Colvin
59b1aead62
refactor: Use HostBuilder to launch spark w/ streams
2026-01-04 21:19:00 +00:00
John Colvin
a515468e37
refactor: Rename TableUpdate props to PascalCase
...
Makes the sdk look more polished, professional
2026-01-04 21:19:00 +00:00
John Colvin
c4a6b4591d
cleanup: Remove last vestiges of spark proto config from DataStreamCore
2026-01-04 21:08:13 +00:00
John Colvin
3565f78be8
cleanup (build): Remove redundant csproj refs
...
For example, projects that reference spark_api don't need an explicit reference
to spark_core, since spark_api already references it. The same goes for projects
that reference DataStream and DataStreamCore.
2026-01-04 21:08:13 +00:00
John Colvin
1c579ee063
refactor: Change namespace to Bts.DataStreamCore for a few types
...
Since they're in the DataStreamCore csproj
2026-01-04 21:08:12 +00:00
John Colvin
0253a0f60f
refactor: Run RemoteQueryManager as an IHostedService
...
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2026-01-04 21:08:12 +00:00
John Colvin
2e14598667
fix: Add a few null checks
...
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2026-01-04 21:08:12 +00:00
John Colvin
2c1b7b95a4
improve: Propagate DataStream exceptions via ErrorChanged event
2026-01-04 21:08:12 +00:00
John Colvin
76fa35382e
refactor: Add DataStream service modules for Dependency Injection
...
Namely, in DataStreamPublisher and PubSubServer. This is part of a broader
paradigm shift towards Dependency Injection and Factory Pattern.
https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines
* Use interfaces to decouple services:
- Decouple ManagedSubscriber from ISubscriberCore impl
- Decouple PubTableBroker from IPublisherCore impl
- Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
* Model config profiles after dotnet appsettings
- Rename Configurations/<name>.json -> appsettings.<name>.json
- Make json fields match corresponding csharp props verbatim
* The Dependency Injection framework doesn't support JsonPropertyName
attributes. Config class props must match json fields verbatim.
* Move reusable startup code into new Bts.Extensions.Hosting proj
* Add pubsub and md-publisher services to compose.yaml
2026-01-04 21:08:11 +00:00
John Colvin
8fc4df1427
refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
...
This matches the file name
2025-12-30 11:38:21 -05:00
John Colvin
7d6e4efa4f
cleanup: Fix numerous build warnings in DataStreamSuite
...
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.
* Compression related files
- BZip2InputStream (disable nullability warnings)
- BZip2OutputStream (disable nullability warnings)
- DeflaterHuffman (disable nullability warnings)
- Inflater (disable nullability warnings)
- InflaterDynHeader (disable nullability warnings)
- ParallelJoinTest (disable nullability warnings)
- TarHeader
- TarInputStream (disable nullability warnings)
- ZipEntry
- ZipFile
- ZipInputStream
* AsyncTCPConnection_APM
* AsyncTCPConnection_APM
* InflaterHuffmanTree
* ManagedSubscriber
* ManagedTCPMessageHub
* MapStream
* PubSubClientMessage
- Normalize indentation/whitespace
- Fix some typos
- Address nullability issues to fix build warnings
- Modernize code
* PubSubMessageReactor
* PubSubServerManagedTCPConnection
* PubSubSingleTagStateEvent
* PubSubState
* PublisherCore
* PubsubDeserializationController
* RemoteMessage
- make 'message' nullable to resolve warnings elsewhere (in calling code)
* SharedStreamConfig
* SingleTagData
* SubscriberCore
* SuperMapStream
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SystemSubscriptionMapPublisher
* ThreadSafeQueue
* TradeBarsStream
Only use Registry on windows.
2025-12-30 11:38:20 -05:00
John Colvin
f5a4e7c410
fix: Remove query from literal error stream
...
The parser can't handle colons in the expression (it throws/crashes)
2025-12-30 11:38:17 -05:00
John Colvin
a0b2a8b186
chore: Add async SQLiteEnumerator to read table as IBlueRowCollection
2025-12-30 11:36:35 -05:00
arham
e69012a5ad
fix: sanitized dsl in websocket
2025-12-03 13:35:08 +00:00
arham
2d0d33028a
fix: bug fix to rowbasedappendstream
2025-12-02 12:58:22 -05:00
arham
f953b3bb8a
change: change UnionStream name to AppendStream
2025-12-02 12:47:36 -05:00
arham
73672276f3
Add: add year to datetimexpander stream
2025-12-02 12:46:49 -05:00
John Colvin
5080960b97
refactor: Replace DataPacket w/ IDataPacket in stream handlers
...
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-19 00:36:56 -05:00
Taha Afzal
09abe4a432
Added RowBasedAppendStream
2025-11-18 03:10:30 +00:00
arham
5305b32c04
Fix file path didnt work in linux
2025-09-17 04:28:25 +00:00
arham
e6304609f8
changes to make streams run on linux
2025-09-16 15:38:21 +00:00
Taha
6b28fc4779
Lot of BlueOrb stream changes
2025-09-02 17:17:43 +00:00
Taha
afd229b7c7
Merging
2025-07-15 17:51:28 -04:00
John Colvin
7e79a21cd4
Add some nifty DataPacket/TableUpdate IObservable utils
2025-07-15 10:13:53 -04:00
John Colvin
77caef2640
fix build: Add missing name to fn arg
2025-07-15 10:13:53 -04:00
Taha
767fa25d61
Adding comments
2025-07-14 15:18:03 -04:00
Taha
0858eaac23
Polishing BlueOrb streams
2025-07-13 21:21:10 -04:00
Taha
70fae06a30
BlueOrb streams and MLP
2025-06-08 21:52:38 -04:00
Taha
a130e47218
Arb Scoring stream
2025-05-20 00:24:17 -04:00