Commit Graph
100 Commits
Author SHA1 Message Date
John Colvin b82cc31d2d refactor: Launch LiveTradeStream via IDataStreamFactory 2025-11-15 12:38:51 -05:00
John Colvin d2a1b67717 refactor: Make IBlueRowCollection inherit IList<IBlueRow> 2025-11-15 12:38:51 -05:00
John Colvin e7625341e3 refactor: Remove hardcoded csv path from ReplayFutureLiveBBOStream
Add constructor overload to accept an input ViewStream
2025-11-15 12:38:51 -05:00
John Colvin 52f35cc421 refactor: Move IBlueTableSchema ext fns into DataStreamCore
The csproj in which IBlueTableSchema is defined.
2025-11-15 12:38:50 -05:00
John Colvin eb28e04bf1 refactor: Replace old NLStreamFactory w/ new BlueOrb one
Replace hardcoded file paths with proper config while we're at it
2025-11-15 12:38:50 -05:00
John Colvin 4ef2dbe4d7 refactor: Pass composite stream factory back to recursive factories as service 2025-11-15 12:38:50 -05:00
John Colvin 0d10c2ba5d new: Add GetColumn fn to IBlueRowCollection interface
This allows us to implement python/pandas functionality, much of which is based
on column operations.
2025-11-15 12:38:50 -05:00
John Colvin 75f2678df6 cleanup: Fix build warnings
* BlueOrbNLStream.cs
* ChangeTableNameStream.cs
* FilterStream.cs
* FilterStream.cs
* GroupByOutputTable.cs
* GroupByStream.cs
* JoinForeignTable.cs
* MultiRowRollingMathStream.cs
* MutableAppendStream.cs
* PersistWindowState.cs
* PubsubEchoTestStream.cs
* RowDictionaryTable.cs
* SortedTable.cs
* TimeSeriesStream.cs

Also add Metadata prop to IDataPacket
2025-11-15 12:38:49 -05:00
John Colvin 8bbe850549 chore: Add 'Rows' prop to BlueTable to avoid accidentally iterating over KeyValuePairs 2025-11-15 12:38:49 -05:00
John Colvin 57b82cf10d fix: Use IDictionary.GetEnumerator explicitly to iterate keyvals
We really need to replace these old non-generic collections with generic ones.
BlueRow's IDictionaryEnumerator implementation iterates over values when passed
to a foreach loop, not key value pairs.
2025-11-15 12:38:49 -05:00
John Colvin f07d83e7bb refactor: Make CompositeDataStreamFactory class internal 2025-11-15 12:38:49 -05:00
John Colvin fcd1b0e799 refactor: Use generic collection types in PrimitiveManager 2025-11-15 12:38:49 -05:00
John Colvin eb102a5898 cleanup: Remove unused code in PrimitiveManager.cs 2025-11-15 12:38:48 -05:00
John Colvin fb3262edea cleanup: Use file-scoped namespace in PrimitiveManager.cs
Also sort using statements
2025-11-15 12:38:48 -05:00
John Colvin 6e1e5d4bbf refactor: Add IDataStreamFactory(s) for spatial arb streams 2025-11-15 12:38:48 -05:00
John Colvin c4f12be3e4 cleanup: Fix a ton of build warnings (cs)
All these warnings have been bothering me for a while. The work isn't done, but
this commit chops them down from ~1000 to ~600.

* AppendCollection.cs
* AppendStreamModified.cs
* BenzingaNewsStream.cs
* BenzingaNewsSymbolStateStream.cs
* BookStream.cs
* CSVSaverStream.cs
* ChartsUtils.cs
* ColumnInfo.cs
* ContractUtils.cs
* ContractUtils.cs
* DiffStream.cs
* FuturesActiveContractStream.cs
* FuzzyJoinOverTimeStream.cs
* HistoricalBarsStream.cs
* HistoricalBarsStream.cs
* InstrumentsStream.cs
* JoinOperandRowIndex.cs
* JoinRow.cs
* JoinStream.cs
* JoinTable.cs
* JoinTableSchema.cs
* LatencyStream.cs
* LeftInternalJoinStream.cs
* LightForeignJoinRow.cs
* LightForeignJoinRowList.cs
* LightLeftForeignJoinRow.cs
* LiteralStream.cs
* NLStream.cs
* NLToDSLStream.cs
* NLToIndexedFewShotPromptStream.cs
* NonIncGroupByStream.cs
* NonIncJoinBookkeeper.cs
* NonIncJoinSpec.cs
* OHLCVStream.cs
* OpenAIPromptStream.cs
* PriceStatsStream.cs
* PriceStatsStream.cs
* ProjectHelper.cs
* ProjectStream.cs
* PubSubServerStateStream.cs
* PubsubHeartbeatStream.cs
* SimpleNode.cs
* SortRow.cs
* SpatialArbStatsStream.cs
* SpatialFuturePairsStream.cs
* SpreadPerformanceEvaluator.cs
* SpreadPerformanceStream.cs
* StreamIOCommon.cs
* SuperJoinStream.cs
* TableMetaDataStream.cs
* TextWriterStream.cs
* TimeAverageStream.cs
2025-11-15 12:16:33 -05:00
John Colvin 03df578a81 chore: Add GetValue, TryGetValue ext fns to IBlueRow 2025-11-15 12:16:32 -05:00
John Colvin 7a50ad2cc5 chore: Add 'Empty' fn for BlueTable/Schema (for convenience) 2025-11-15 12:16:32 -05:00
John Colvin e2816aaa33 refactor: Decouple ViewStreamDatabase from TableStream, ISubscriberCore 2025-11-15 12:16:32 -05:00
John Colvin 05644ed6cf feat: Add DataStream WebApp
Included in DataStreamSuite.sln, but not DataStreamQueryAnalyzer.sln
2025-11-14 15:00:45 -05:00
John Colvin 1bb7d6eca2 Add ColumnIndex class to resolve column names to indices
Improves performance compared to accessing columns by name
2025-11-14 14:59:33 -05:00
John Colvin 6b93c07271 Serialize TimeSpan w/ microsecond precision in DataPacket json 2025-11-14 14:59:33 -05:00
John Colvin c3288c721d Format DateTimes w/ microsecond precision in DataPacketJsonConverter 2025-11-14 14:59:33 -05:00
John Colvin 2622626eed Add BlueColumnTable for pandas integration
This will make it easier/more natural to support pandas column expressions
2025-11-14 14:59:33 -05:00
John Colvin ada7d4bed1 refactor: Load IDataStreamFactory(s) with dependency injection
So we can create ViewStreams that depend on other services, config, etc.
Previously, the PrimitiveManager relied purely on static construction of
types through reflection.
2025-11-14 14:52:07 -05:00
John Colvin f65b6b82f2 refactor: Replace DataPacket w/ IDataPacket in stream handlers
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-03 16:21:32 -05:00
John Colvin 1155fd6997 refactor: Change namespace to Bts.DataStreamCore for a few types
Since they're in the DataStreamCore csproj
2025-11-03 16:19:50 -05:00
John Colvin d707847079 cleanup: Remove unused ViewStreamDatabase config 2025-11-03 16:19:48 -05:00
John Colvin 118e2a9f2e Remove reference to spark protobuf Config in ViewStream
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-03 16:19:48 -05:00
John Colvin 8637bb9254 Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-03 16:13:44 -05:00
John Colvin 7c61c2ef6d cleanup: Remove unused code from ViewStreamDatabase 2025-11-03 16:13:44 -05:00
John Colvin d669d24aea fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-11-03 16:13:43 -05:00
John Colvin cd7c04dce5 Add ErrorChanged event to propagate DataStream exceptions 2025-11-03 16:13:19 -05:00
John Colvin b4cb6ac856 Add PubSub service modules designed for Dependency Injection
* Get rid of JsonPropertyName attributes in PubSubConfig
  - The Dependency Injection framework (based around dotnet ApplicationBuilder)
    does not support this attribute, and maps property names verbatim to json fields.

* Change PubSubServer json fields to match cs class property names verbatim
2025-11-03 16:13:18 -05:00
John Colvin 41cd59e6f9 refactor: Only support 1 tag type in IPubSubServerTagHandler 2025-11-03 16:13:18 -05:00
John Colvin f2ace4ae32 refactor: Decouple PubTableBroker from IPublisherCore impl
Reference interface rather than the PublisherCore impl.
2025-11-03 16:13:18 -05:00
John Colvin 135db27733 cleanup: Remove unused pubsub tag handler code
* Removed 'IPubSubServerTagHandler.register'
  - All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
  - This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
  - No reason these interfaces have to be tied together
2025-11-03 16:13:18 -05:00
John Colvin bf913f2b99 chore: Add json facilities for DataStream websocket api 2025-11-03 16:13:17 -05:00
John Colvin bf1ce9467d Add generic ViewStream, BlueRow types/interfaces 2025-11-03 16:13:16 -05:00
John Colvin 906e48743d Add BlueRowColumnMask to hide columns w/ minimal overhead
Less overhead than ProjectStream
2025-11-03 16:13:16 -05:00
John Colvin 1df508a16c fix Impl IDictionary<int, IBlueRow> in BlueTable 2025-11-03 16:13:16 -05:00
John Colvin 97efe8396e Add gen purpose blue row/table utils for data streams
* Add BlueRowReflection<T> so ViewStream implementations can use
  strong row types without needing to implement IBlueRow
* Add default implementations for IDictionary members in IBlueRow to make
  implementing IBlueRow less tedious
2025-11-03 16:13:16 -05:00
Taha Afzal 66c0baa150 Adding meta data to DataPacket, NLStream adds nl -> dsl as a name value pair in the meta data. Also removing windows ref from DataStreamStudio.sln 2025-09-24 13:16:28 +00:00
John Colvin 7e79a21cd4 Add some nifty DataPacket/TableUpdate IObservable utils 2025-07-15 10:13:53 -04:00
Taha 0858eaac23 Polishing BlueOrb streams 2025-07-13 21:21:10 -04:00
John Colvin 5163ca4cae refactor: Organize libspark proj w/ stuttering include
This new structure provides several benefits:

* Allows us to install multiple CMake projects in the same destination
  dir (without include path conflicts).
  - This makes shipping the trading algo sdk a trivial excercise. Simply
    copy the lib and include dirs from the install dest. That's it!
* Puts us in a position to break up the monolithic libspark into smaller
  modules, which are easier to manage.

Windows specific notes:

* Get rid of unnecessary script vs_filters.cmake
* Define WebsocketConnectionGateway directly in cs (instead of proto)
  - The proto would no longer be in the include path passed to protoc.
    Moving it to c# because there wasn't a compelling reason for it to
    be a proto in the first place.
* Set spark output prefix to "lib" on windows
  - So we get "libspark.dll" instead of "spark.dll", which conflicts
    with the csharp lib, "Spark.dll".
* Copy libspark.dll to net8.0 & net8.0-windows in a single cmake build
  target

fix build: Copy libspark to both dirs net8.0, net8.0-windows
2025-06-16 11:08:53 -04:00
Taha 70fae06a30 BlueOrb streams and MLP 2025-06-08 21:52:38 -04:00
John Colvin 043efab5a2 build: Upgrade c# to net8.0 2025-04-28 17:12:58 +00:00
Taha aab8e83543 Some rewrite of streams that fetch bars, and making the other business streams thin wrapper over stream primitives 2024-11-26 22:33:26 -05:00
Taha 6163887cd3 Bunch of CME queries are now working 2024-10-30 13:36:23 -04:00