John Colvin
b82cc31d2d
refactor: Launch LiveTradeStream via IDataStreamFactory
2025-11-15 12:38:51 -05:00
John Colvin
d2a1b67717
refactor: Make IBlueRowCollection inherit IList<IBlueRow>
2025-11-15 12:38:51 -05:00
John Colvin
e7625341e3
refactor: Remove hardcoded csv path from ReplayFutureLiveBBOStream
...
Add constructor overload to accept an input ViewStream
2025-11-15 12:38:51 -05:00
John Colvin
52f35cc421
refactor: Move IBlueTableSchema ext fns into DataStreamCore
...
The csproj in which IBlueTableSchema is defined.
2025-11-15 12:38:50 -05:00
John Colvin
eb28e04bf1
refactor: Replace old NLStreamFactory w/ new BlueOrb one
...
Replace hardcoded file paths with proper config while we're at it
2025-11-15 12:38:50 -05:00
John Colvin
4ef2dbe4d7
refactor: Pass composite stream factory back to recursive factories as service
2025-11-15 12:38:50 -05:00
John Colvin
0d10c2ba5d
new: Add GetColumn fn to IBlueRowCollection interface
...
This allows us to implement python/pandas functionality, much of which is based
on column operations.
2025-11-15 12:38:50 -05:00
John Colvin
75f2678df6
cleanup: Fix build warnings
...
* BlueOrbNLStream.cs
* ChangeTableNameStream.cs
* FilterStream.cs
* FilterStream.cs
* GroupByOutputTable.cs
* GroupByStream.cs
* JoinForeignTable.cs
* MultiRowRollingMathStream.cs
* MutableAppendStream.cs
* PersistWindowState.cs
* PubsubEchoTestStream.cs
* RowDictionaryTable.cs
* SortedTable.cs
* TimeSeriesStream.cs
Also add Metadata prop to IDataPacket
2025-11-15 12:38:49 -05:00
John Colvin
8bbe850549
chore: Add 'Rows' prop to BlueTable to avoid accidentally iterating over KeyValuePairs
2025-11-15 12:38:49 -05:00
John Colvin
57b82cf10d
fix: Use IDictionary.GetEnumerator explicitly to iterate keyvals
...
We really need to replace these old non-generic collections with generic ones.
BlueRow's IDictionaryEnumerator implementation iterates over values when passed
to a foreach loop, not key value pairs.
2025-11-15 12:38:49 -05:00
John Colvin
f07d83e7bb
refactor: Make CompositeDataStreamFactory class internal
2025-11-15 12:38:49 -05:00
John Colvin
fcd1b0e799
refactor: Use generic collection types in PrimitiveManager
2025-11-15 12:38:49 -05:00
John Colvin
eb102a5898
cleanup: Remove unused code in PrimitiveManager.cs
2025-11-15 12:38:48 -05:00
John Colvin
fb3262edea
cleanup: Use file-scoped namespace in PrimitiveManager.cs
...
Also sort using statements
2025-11-15 12:38:48 -05:00
John Colvin
6e1e5d4bbf
refactor: Add IDataStreamFactory(s) for spatial arb streams
2025-11-15 12:38:48 -05:00
John Colvin
c4f12be3e4
cleanup: Fix a ton of build warnings (cs)
...
All these warnings have been bothering me for a while. The work isn't done, but
this commit chops them down from ~1000 to ~600.
* AppendCollection.cs
* AppendStreamModified.cs
* BenzingaNewsStream.cs
* BenzingaNewsSymbolStateStream.cs
* BookStream.cs
* CSVSaverStream.cs
* ChartsUtils.cs
* ColumnInfo.cs
* ContractUtils.cs
* ContractUtils.cs
* DiffStream.cs
* FuturesActiveContractStream.cs
* FuzzyJoinOverTimeStream.cs
* HistoricalBarsStream.cs
* HistoricalBarsStream.cs
* InstrumentsStream.cs
* JoinOperandRowIndex.cs
* JoinRow.cs
* JoinStream.cs
* JoinTable.cs
* JoinTableSchema.cs
* LatencyStream.cs
* LeftInternalJoinStream.cs
* LightForeignJoinRow.cs
* LightForeignJoinRowList.cs
* LightLeftForeignJoinRow.cs
* LiteralStream.cs
* NLStream.cs
* NLToDSLStream.cs
* NLToIndexedFewShotPromptStream.cs
* NonIncGroupByStream.cs
* NonIncJoinBookkeeper.cs
* NonIncJoinSpec.cs
* OHLCVStream.cs
* OpenAIPromptStream.cs
* PriceStatsStream.cs
* PriceStatsStream.cs
* ProjectHelper.cs
* ProjectStream.cs
* PubSubServerStateStream.cs
* PubsubHeartbeatStream.cs
* SimpleNode.cs
* SortRow.cs
* SpatialArbStatsStream.cs
* SpatialFuturePairsStream.cs
* SpreadPerformanceEvaluator.cs
* SpreadPerformanceStream.cs
* StreamIOCommon.cs
* SuperJoinStream.cs
* TableMetaDataStream.cs
* TextWriterStream.cs
* TimeAverageStream.cs
2025-11-15 12:16:33 -05:00
John Colvin
03df578a81
chore: Add GetValue, TryGetValue ext fns to IBlueRow
2025-11-15 12:16:32 -05:00
John Colvin
7a50ad2cc5
chore: Add 'Empty' fn for BlueTable/Schema (for convenience)
2025-11-15 12:16:32 -05:00
John Colvin
e2816aaa33
refactor: Decouple ViewStreamDatabase from TableStream, ISubscriberCore
2025-11-15 12:16:32 -05:00
John Colvin
05644ed6cf
feat: Add DataStream WebApp
...
Included in DataStreamSuite.sln, but not DataStreamQueryAnalyzer.sln
2025-11-14 15:00:45 -05:00
John Colvin
1bb7d6eca2
Add ColumnIndex class to resolve column names to indices
...
Improves performance compared to accessing columns by name
2025-11-14 14:59:33 -05:00
John Colvin
6b93c07271
Serialize TimeSpan w/ microsecond precision in DataPacket json
2025-11-14 14:59:33 -05:00
John Colvin
c3288c721d
Format DateTimes w/ microsecond precision in DataPacketJsonConverter
2025-11-14 14:59:33 -05:00
John Colvin
2622626eed
Add BlueColumnTable for pandas integration
...
This will make it easier/more natural to support pandas column expressions
2025-11-14 14:59:33 -05:00
John Colvin
ada7d4bed1
refactor: Load IDataStreamFactory(s) with dependency injection
...
So we can create ViewStreams that depend on other services, config, etc.
Previously, the PrimitiveManager relied purely on static construction of
types through reflection.
2025-11-14 14:52:07 -05:00
John Colvin
f65b6b82f2
refactor: Replace DataPacket w/ IDataPacket in stream handlers
...
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-03 16:21:32 -05:00
John Colvin
1155fd6997
refactor: Change namespace to Bts.DataStreamCore for a few types
...
Since they're in the DataStreamCore csproj
2025-11-03 16:19:50 -05:00
John Colvin
d707847079
cleanup: Remove unused ViewStreamDatabase config
2025-11-03 16:19:48 -05:00
John Colvin
118e2a9f2e
Remove reference to spark protobuf Config in ViewStream
...
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-03 16:19:48 -05:00
John Colvin
8637bb9254
Assert ViewStreamDatabase.instance is only created once
...
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-03 16:13:44 -05:00
John Colvin
7c61c2ef6d
cleanup: Remove unused code from ViewStreamDatabase
2025-11-03 16:13:44 -05:00
John Colvin
d669d24aea
fix: Add a few null checks
...
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-11-03 16:13:43 -05:00
John Colvin
cd7c04dce5
Add ErrorChanged event to propagate DataStream exceptions
2025-11-03 16:13:19 -05:00
John Colvin
b4cb6ac856
Add PubSub service modules designed for Dependency Injection
...
* Get rid of JsonPropertyName attributes in PubSubConfig
- The Dependency Injection framework (based around dotnet ApplicationBuilder)
does not support this attribute, and maps property names verbatim to json fields.
* Change PubSubServer json fields to match cs class property names verbatim
2025-11-03 16:13:18 -05:00
John Colvin
41cd59e6f9
refactor: Only support 1 tag type in IPubSubServerTagHandler
2025-11-03 16:13:18 -05:00
John Colvin
f2ace4ae32
refactor: Decouple PubTableBroker from IPublisherCore impl
...
Reference interface rather than the PublisherCore impl.
2025-11-03 16:13:18 -05:00
John Colvin
135db27733
cleanup: Remove unused pubsub tag handler code
...
* Removed 'IPubSubServerTagHandler.register'
- All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
- This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
- No reason these interfaces have to be tied together
2025-11-03 16:13:18 -05:00
John Colvin
bf913f2b99
chore: Add json facilities for DataStream websocket api
2025-11-03 16:13:17 -05:00
John Colvin
bf1ce9467d
Add generic ViewStream, BlueRow types/interfaces
2025-11-03 16:13:16 -05:00
John Colvin
906e48743d
Add BlueRowColumnMask to hide columns w/ minimal overhead
...
Less overhead than ProjectStream
2025-11-03 16:13:16 -05:00
John Colvin
1df508a16c
fix Impl IDictionary<int, IBlueRow> in BlueTable
2025-11-03 16:13:16 -05:00
John Colvin
97efe8396e
Add gen purpose blue row/table utils for data streams
...
* Add BlueRowReflection<T> so ViewStream implementations can use
strong row types without needing to implement IBlueRow
* Add default implementations for IDictionary members in IBlueRow to make
implementing IBlueRow less tedious
2025-11-03 16:13:16 -05:00
Taha Afzal
66c0baa150
Adding meta data to DataPacket, NLStream adds nl -> dsl as a name value pair in the meta data. Also removing windows ref from DataStreamStudio.sln
2025-09-24 13:16:28 +00:00
John Colvin
7e79a21cd4
Add some nifty DataPacket/TableUpdate IObservable utils
2025-07-15 10:13:53 -04:00
Taha
0858eaac23
Polishing BlueOrb streams
2025-07-13 21:21:10 -04:00
John Colvin
5163ca4cae
refactor: Organize libspark proj w/ stuttering include
...
This new structure provides several benefits:
* Allows us to install multiple CMake projects in the same destination
dir (without include path conflicts).
- This makes shipping the trading algo sdk a trivial excercise. Simply
copy the lib and include dirs from the install dest. That's it!
* Puts us in a position to break up the monolithic libspark into smaller
modules, which are easier to manage.
Windows specific notes:
* Get rid of unnecessary script vs_filters.cmake
* Define WebsocketConnectionGateway directly in cs (instead of proto)
- The proto would no longer be in the include path passed to protoc.
Moving it to c# because there wasn't a compelling reason for it to
be a proto in the first place.
* Set spark output prefix to "lib" on windows
- So we get "libspark.dll" instead of "spark.dll", which conflicts
with the csharp lib, "Spark.dll".
* Copy libspark.dll to net8.0 & net8.0-windows in a single cmake build
target
fix build: Copy libspark to both dirs net8.0, net8.0-windows
2025-06-16 11:08:53 -04:00
Taha
70fae06a30
BlueOrb streams and MLP
2025-06-08 21:52:38 -04:00
John Colvin
043efab5a2
build: Upgrade c# to net8.0
2025-04-28 17:12:58 +00:00
Taha
aab8e83543
Some rewrite of streams that fetch bars, and making the other business streams thin wrapper over stream primitives
2024-11-26 22:33:26 -05:00
Taha
6163887cd3
Bunch of CME queries are now working
2024-10-30 13:36:23 -04:00