Commit Graph
108 Commits
Author SHA1 Message Date
John Colvin 6045c602dc feat: Add pseudo member (extension) function calls to DSL 2025-11-15 12:45:07 -05:00
John Colvin 0cf44fcc40 feat: Add historic market data streams factory
Loads historic md from Databento files in structured paths. It is
designed to work with files downloaded via databento_client.py
2025-11-15 12:45:04 -05:00
John Colvin a79f5ff860 improve: Support all integer types in IBlueRow Id ext fn 2025-11-15 12:42:40 -05:00
John Colvin 4db9cdd357 feat: Add DiffStream to compute row diffs (like pandas) 2025-11-15 12:42:40 -05:00
John Colvin 18b19ac787 refactor: Rewrite SortStream now that IBlueRowCollection is IList 2025-11-15 12:42:40 -05:00
John Colvin 03294ffe37 feat: Add function defs to DataStream DSL 2025-11-15 12:42:39 -05:00
John Colvin 805693e170 feat: Add ParquetReader/Writer streams 2025-11-15 12:40:48 -05:00
John Colvin 3a7f3ac608 fix: Fire multiple rows at once in replay bbo/trade streams doWork fn
If necessary to keep pace with timestamps in the file.
2025-11-15 12:38:51 -05:00
John Colvin b82cc31d2d refactor: Launch LiveTradeStream via IDataStreamFactory 2025-11-15 12:38:51 -05:00
John Colvin d2a1b67717 refactor: Make IBlueRowCollection inherit IList<IBlueRow> 2025-11-15 12:38:51 -05:00
John Colvin e7625341e3 refactor: Remove hardcoded csv path from ReplayFutureLiveBBOStream
Add constructor overload to accept an input ViewStream
2025-11-15 12:38:51 -05:00
John Colvin 52f35cc421 refactor: Move IBlueTableSchema ext fns into DataStreamCore
The csproj in which IBlueTableSchema is defined.
2025-11-15 12:38:50 -05:00
John Colvin eb28e04bf1 refactor: Replace old NLStreamFactory w/ new BlueOrb one
Replace hardcoded file paths with proper config while we're at it
2025-11-15 12:38:50 -05:00
John Colvin 4ef2dbe4d7 refactor: Pass composite stream factory back to recursive factories as service 2025-11-15 12:38:50 -05:00
John Colvin 0d10c2ba5d new: Add GetColumn fn to IBlueRowCollection interface
This allows us to implement python/pandas functionality, much of which is based
on column operations.
2025-11-15 12:38:50 -05:00
John Colvin 75f2678df6 cleanup: Fix build warnings
* BlueOrbNLStream.cs
* ChangeTableNameStream.cs
* FilterStream.cs
* FilterStream.cs
* GroupByOutputTable.cs
* GroupByStream.cs
* JoinForeignTable.cs
* MultiRowRollingMathStream.cs
* MutableAppendStream.cs
* PersistWindowState.cs
* PubsubEchoTestStream.cs
* RowDictionaryTable.cs
* SortedTable.cs
* TimeSeriesStream.cs

Also add Metadata prop to IDataPacket
2025-11-15 12:38:49 -05:00
John Colvin 8bbe850549 chore: Add 'Rows' prop to BlueTable to avoid accidentally iterating over KeyValuePairs 2025-11-15 12:38:49 -05:00
John Colvin 57b82cf10d fix: Use IDictionary.GetEnumerator explicitly to iterate keyvals
We really need to replace these old non-generic collections with generic ones.
BlueRow's IDictionaryEnumerator implementation iterates over values when passed
to a foreach loop, not key value pairs.
2025-11-15 12:38:49 -05:00
John Colvin f07d83e7bb refactor: Make CompositeDataStreamFactory class internal 2025-11-15 12:38:49 -05:00
John Colvin fcd1b0e799 refactor: Use generic collection types in PrimitiveManager 2025-11-15 12:38:49 -05:00
John Colvin eb102a5898 cleanup: Remove unused code in PrimitiveManager.cs 2025-11-15 12:38:48 -05:00
John Colvin fb3262edea cleanup: Use file-scoped namespace in PrimitiveManager.cs
Also sort using statements
2025-11-15 12:38:48 -05:00
John Colvin 6e1e5d4bbf refactor: Add IDataStreamFactory(s) for spatial arb streams 2025-11-15 12:38:48 -05:00
John Colvin c4f12be3e4 cleanup: Fix a ton of build warnings (cs)
All these warnings have been bothering me for a while. The work isn't done, but
this commit chops them down from ~1000 to ~600.

* AppendCollection.cs
* AppendStreamModified.cs
* BenzingaNewsStream.cs
* BenzingaNewsSymbolStateStream.cs
* BookStream.cs
* CSVSaverStream.cs
* ChartsUtils.cs
* ColumnInfo.cs
* ContractUtils.cs
* ContractUtils.cs
* DiffStream.cs
* FuturesActiveContractStream.cs
* FuzzyJoinOverTimeStream.cs
* HistoricalBarsStream.cs
* HistoricalBarsStream.cs
* InstrumentsStream.cs
* JoinOperandRowIndex.cs
* JoinRow.cs
* JoinStream.cs
* JoinTable.cs
* JoinTableSchema.cs
* LatencyStream.cs
* LeftInternalJoinStream.cs
* LightForeignJoinRow.cs
* LightForeignJoinRowList.cs
* LightLeftForeignJoinRow.cs
* LiteralStream.cs
* NLStream.cs
* NLToDSLStream.cs
* NLToIndexedFewShotPromptStream.cs
* NonIncGroupByStream.cs
* NonIncJoinBookkeeper.cs
* NonIncJoinSpec.cs
* OHLCVStream.cs
* OpenAIPromptStream.cs
* PriceStatsStream.cs
* PriceStatsStream.cs
* ProjectHelper.cs
* ProjectStream.cs
* PubSubServerStateStream.cs
* PubsubHeartbeatStream.cs
* SimpleNode.cs
* SortRow.cs
* SpatialArbStatsStream.cs
* SpatialFuturePairsStream.cs
* SpreadPerformanceEvaluator.cs
* SpreadPerformanceStream.cs
* StreamIOCommon.cs
* SuperJoinStream.cs
* TableMetaDataStream.cs
* TextWriterStream.cs
* TimeAverageStream.cs
2025-11-15 12:16:33 -05:00
John Colvin 03df578a81 chore: Add GetValue, TryGetValue ext fns to IBlueRow 2025-11-15 12:16:32 -05:00
John Colvin 7a50ad2cc5 chore: Add 'Empty' fn for BlueTable/Schema (for convenience) 2025-11-15 12:16:32 -05:00
John Colvin e2816aaa33 refactor: Decouple ViewStreamDatabase from TableStream, ISubscriberCore 2025-11-15 12:16:32 -05:00
John Colvin 05644ed6cf feat: Add DataStream WebApp
Included in DataStreamSuite.sln, but not DataStreamQueryAnalyzer.sln
2025-11-14 15:00:45 -05:00
John Colvin 1bb7d6eca2 Add ColumnIndex class to resolve column names to indices
Improves performance compared to accessing columns by name
2025-11-14 14:59:33 -05:00
John Colvin 6b93c07271 Serialize TimeSpan w/ microsecond precision in DataPacket json 2025-11-14 14:59:33 -05:00
John Colvin c3288c721d Format DateTimes w/ microsecond precision in DataPacketJsonConverter 2025-11-14 14:59:33 -05:00
John Colvin 2622626eed Add BlueColumnTable for pandas integration
This will make it easier/more natural to support pandas column expressions
2025-11-14 14:59:33 -05:00
John Colvin ada7d4bed1 refactor: Load IDataStreamFactory(s) with dependency injection
So we can create ViewStreams that depend on other services, config, etc.
Previously, the PrimitiveManager relied purely on static construction of
types through reflection.
2025-11-14 14:52:07 -05:00
John Colvin f65b6b82f2 refactor: Replace DataPacket w/ IDataPacket in stream handlers
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-03 16:21:32 -05:00
John Colvin 1155fd6997 refactor: Change namespace to Bts.DataStreamCore for a few types
Since they're in the DataStreamCore csproj
2025-11-03 16:19:50 -05:00
John Colvin d707847079 cleanup: Remove unused ViewStreamDatabase config 2025-11-03 16:19:48 -05:00
John Colvin 118e2a9f2e Remove reference to spark protobuf Config in ViewStream
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-03 16:19:48 -05:00
John Colvin 8637bb9254 Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-03 16:13:44 -05:00
John Colvin 7c61c2ef6d cleanup: Remove unused code from ViewStreamDatabase 2025-11-03 16:13:44 -05:00
John Colvin d669d24aea fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-11-03 16:13:43 -05:00
John Colvin cd7c04dce5 Add ErrorChanged event to propagate DataStream exceptions 2025-11-03 16:13:19 -05:00
John Colvin b4cb6ac856 Add PubSub service modules designed for Dependency Injection
* Get rid of JsonPropertyName attributes in PubSubConfig
  - The Dependency Injection framework (based around dotnet ApplicationBuilder)
    does not support this attribute, and maps property names verbatim to json fields.

* Change PubSubServer json fields to match cs class property names verbatim
2025-11-03 16:13:18 -05:00
John Colvin 41cd59e6f9 refactor: Only support 1 tag type in IPubSubServerTagHandler 2025-11-03 16:13:18 -05:00
John Colvin f2ace4ae32 refactor: Decouple PubTableBroker from IPublisherCore impl
Reference interface rather than the PublisherCore impl.
2025-11-03 16:13:18 -05:00
John Colvin 135db27733 cleanup: Remove unused pubsub tag handler code
* Removed 'IPubSubServerTagHandler.register'
  - All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
  - This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
  - No reason these interfaces have to be tied together
2025-11-03 16:13:18 -05:00
John Colvin bf913f2b99 chore: Add json facilities for DataStream websocket api 2025-11-03 16:13:17 -05:00
John Colvin bf1ce9467d Add generic ViewStream, BlueRow types/interfaces 2025-11-03 16:13:16 -05:00
John Colvin 906e48743d Add BlueRowColumnMask to hide columns w/ minimal overhead
Less overhead than ProjectStream
2025-11-03 16:13:16 -05:00
John Colvin 1df508a16c fix Impl IDictionary<int, IBlueRow> in BlueTable 2025-11-03 16:13:16 -05:00
John Colvin 97efe8396e Add gen purpose blue row/table utils for data streams
* Add BlueRowReflection<T> so ViewStream implementations can use
  strong row types without needing to implement IBlueRow
* Add default implementations for IDictionary members in IBlueRow to make
  implementing IBlueRow less tedious
2025-11-03 16:13:16 -05:00