Commit Graph
321 Commits
Author SHA1 Message Date
John Colvin 0cf44fcc40 feat: Add historic market data streams factory
Loads historic md from Databento files in structured paths. It is
designed to work with files downloaded via databento_client.py
2025-11-15 12:45:04 -05:00
John Colvin 2549e34a17 Create LiveBBOStream via BookStreamFactory
Instead of PrimitiveManager
2025-11-15 12:42:40 -05:00
John Colvin 693f2deb67 fix: Don't modify input env in BoundBooleanExpression.Evaluate 2025-11-15 12:42:40 -05:00
John Colvin 63665e9b10 refactor: Pass IMarketDataClient to streams via dependency injection
In general, the streams should not instantiate services, etc. themselves.
2025-11-15 12:42:40 -05:00
John Colvin a79f5ff860 improve: Support all integer types in IBlueRow Id ext fn 2025-11-15 12:42:40 -05:00
John Colvin dae75c507e cleanup: Remove unused code CSharpString.cs 2025-11-15 12:42:40 -05:00
John Colvin 4db9cdd357 feat: Add DiffStream to compute row diffs (like pandas) 2025-11-15 12:42:40 -05:00
John Colvin 18b19ac787 refactor: Rewrite SortStream now that IBlueRowCollection is IList 2025-11-15 12:42:40 -05:00
John Colvin b5ea918378 refactor: Add param for TimeSeriesStream timestamp col name
Default it to 'ts_event' instead of 'timestamp' (this is consistent w/
Databento schema fwiw)
2025-11-15 12:42:40 -05:00
John Colvin 9df4ecf6c8 fix: Trim backticks from nl generated dsl 2025-11-15 12:42:40 -05:00
John Colvin 957f6cb33f feat: Add FeatherReader stream
We don't need this right now, but I already wrote it...it might be
useful later.
2025-11-15 12:42:39 -05:00
John Colvin 03294ffe37 feat: Add function defs to DataStream DSL 2025-11-15 12:42:39 -05:00
John Colvin 97eefb1a1e improve: Serialize to/from parquet in PythonDataframeStream 2025-11-15 12:42:39 -05:00
John Colvin 805693e170 feat: Add ParquetReader/Writer streams 2025-11-15 12:40:48 -05:00
John Colvin 3a7f3ac608 fix: Fire multiple rows at once in replay bbo/trade streams doWork fn
If necessary to keep pace with timestamps in the file.
2025-11-15 12:38:51 -05:00
John Colvin b82cc31d2d refactor: Launch LiveTradeStream via IDataStreamFactory 2025-11-15 12:38:51 -05:00
John Colvin d2a1b67717 refactor: Make IBlueRowCollection inherit IList<IBlueRow> 2025-11-15 12:38:51 -05:00
John Colvin 144b29c728 improvement: Lookup timestamp col by index rather than name 2025-11-15 12:38:51 -05:00
John Colvin e7625341e3 refactor: Remove hardcoded csv path from ReplayFutureLiveBBOStream
Add constructor overload to accept an input ViewStream
2025-11-15 12:38:51 -05:00
John Colvin fb15b3fb24 improvement: Don't mutate IBlueRowCollection after firing in FilterStream 2025-11-15 12:38:51 -05:00
John Colvin ea0246929a improvement: Make MultiRowRollingMathStream more performant
Avoid creating extra lists and o(n) enumeration
2025-11-15 12:38:50 -05:00
John Colvin 52f35cc421 refactor: Move IBlueTableSchema ext fns into DataStreamCore
The csproj in which IBlueTableSchema is defined.
2025-11-15 12:38:50 -05:00
John Colvin eb28e04bf1 refactor: Replace old NLStreamFactory w/ new BlueOrb one
Replace hardcoded file paths with proper config while we're at it
2025-11-15 12:38:50 -05:00
John Colvin 4ef2dbe4d7 refactor: Pass composite stream factory back to recursive factories as service 2025-11-15 12:38:50 -05:00
John Colvin 979051f147 build: Add appsettings.Local.json to DSQA 2025-11-15 12:38:50 -05:00
John Colvin 0d10c2ba5d new: Add GetColumn fn to IBlueRowCollection interface
This allows us to implement python/pandas functionality, much of which is based
on column operations.
2025-11-15 12:38:50 -05:00
John Colvin 75f2678df6 cleanup: Fix build warnings
* BlueOrbNLStream.cs
* ChangeTableNameStream.cs
* FilterStream.cs
* FilterStream.cs
* GroupByOutputTable.cs
* GroupByStream.cs
* JoinForeignTable.cs
* MultiRowRollingMathStream.cs
* MutableAppendStream.cs
* PersistWindowState.cs
* PubsubEchoTestStream.cs
* RowDictionaryTable.cs
* SortedTable.cs
* TimeSeriesStream.cs

Also add Metadata prop to IDataPacket
2025-11-15 12:38:49 -05:00
John Colvin 8bbe850549 chore: Add 'Rows' prop to BlueTable to avoid accidentally iterating over KeyValuePairs 2025-11-15 12:38:49 -05:00
John Colvin 25860af7bf refactor: Use dependency injection in Spark.Console 2025-11-15 12:38:49 -05:00
John Colvin 57b82cf10d fix: Use IDictionary.GetEnumerator explicitly to iterate keyvals
We really need to replace these old non-generic collections with generic ones.
BlueRow's IDictionaryEnumerator implementation iterates over values when passed
to a foreach loop, not key value pairs.
2025-11-15 12:38:49 -05:00
John Colvin f07d83e7bb refactor: Make CompositeDataStreamFactory class internal 2025-11-15 12:38:49 -05:00
John Colvin 760cdc0fba cleanup: Fix build warnings, remove unused/redundant code
* Make a few member vars and fns local to the constructor
* Replace a couple non-generic collections with generic ones
* Fix warnings and clean code in:
  - DictionaryExtensions.cs
    * Remove ext fns that are part of dotnet now
  - JoinNameResolver.cs
  - MainForm.cs

cleanup: Remove unused code in ladder

Some props in cs, and StaticResources in xaml were leftover after I combined
the panel price points into a single collection, bound to an ItemsControl.
2025-11-15 12:38:49 -05:00
John Colvin fcd1b0e799 refactor: Use generic collection types in PrimitiveManager 2025-11-15 12:38:49 -05:00
John Colvin eb102a5898 cleanup: Remove unused code in PrimitiveManager.cs 2025-11-15 12:38:48 -05:00
John Colvin fb3262edea cleanup: Use file-scoped namespace in PrimitiveManager.cs
Also sort using statements
2025-11-15 12:38:48 -05:00
John Colvin 6e1e5d4bbf refactor: Add IDataStreamFactory(s) for spatial arb streams 2025-11-15 12:38:48 -05:00
John Colvin 24892fb1a8 cleanup: Remove unused BooleanExpressionToken Negate 2025-11-15 12:38:48 -05:00
John Colvin 29080795f7 refactor: Rename AddRemoteQueryManager -> AddRemoteQueryStreams 2025-11-15 12:38:48 -05:00
John Colvin cd291b7a0d refactor: Hide md client behind interface, create via Dep Injection 2025-11-15 12:38:48 -05:00
John Colvin c4f12be3e4 cleanup: Fix a ton of build warnings (cs)
All these warnings have been bothering me for a while. The work isn't done, but
this commit chops them down from ~1000 to ~600.

* AppendCollection.cs
* AppendStreamModified.cs
* BenzingaNewsStream.cs
* BenzingaNewsSymbolStateStream.cs
* BookStream.cs
* CSVSaverStream.cs
* ChartsUtils.cs
* ColumnInfo.cs
* ContractUtils.cs
* ContractUtils.cs
* DiffStream.cs
* FuturesActiveContractStream.cs
* FuzzyJoinOverTimeStream.cs
* HistoricalBarsStream.cs
* HistoricalBarsStream.cs
* InstrumentsStream.cs
* JoinOperandRowIndex.cs
* JoinRow.cs
* JoinStream.cs
* JoinTable.cs
* JoinTableSchema.cs
* LatencyStream.cs
* LeftInternalJoinStream.cs
* LightForeignJoinRow.cs
* LightForeignJoinRowList.cs
* LightLeftForeignJoinRow.cs
* LiteralStream.cs
* NLStream.cs
* NLToDSLStream.cs
* NLToIndexedFewShotPromptStream.cs
* NonIncGroupByStream.cs
* NonIncJoinBookkeeper.cs
* NonIncJoinSpec.cs
* OHLCVStream.cs
* OpenAIPromptStream.cs
* PriceStatsStream.cs
* PriceStatsStream.cs
* ProjectHelper.cs
* ProjectStream.cs
* PubSubServerStateStream.cs
* PubsubHeartbeatStream.cs
* SimpleNode.cs
* SortRow.cs
* SpatialArbStatsStream.cs
* SpatialFuturePairsStream.cs
* SpreadPerformanceEvaluator.cs
* SpreadPerformanceStream.cs
* StreamIOCommon.cs
* SuperJoinStream.cs
* TableMetaDataStream.cs
* TextWriterStream.cs
* TimeAverageStream.cs
2025-11-15 12:16:33 -05:00
John Colvin 54d7279832 style: Normalize indentation in PubsubHeartbeatStream.cs 2025-11-15 12:16:33 -05:00
John Colvin 03df578a81 chore: Add GetValue, TryGetValue ext fns to IBlueRow 2025-11-15 12:16:32 -05:00
John Colvin 7a50ad2cc5 chore: Add 'Empty' fn for BlueTable/Schema (for convenience) 2025-11-15 12:16:32 -05:00
John Colvin 0cc4b83417 cleanup: Remove redundant code block in HistoricalBarsStream.cs 2025-11-15 12:16:32 -05:00
John Colvin 4f1efea066 cleanup: Fix indentation and build warnings in TestStream.cs 2025-11-15 12:16:32 -05:00
John Colvin e2816aaa33 refactor: Decouple ViewStreamDatabase from TableStream, ISubscriberCore 2025-11-15 12:16:32 -05:00
John Colvin e2a101eb20 feat: Add SpreadBook stream 2025-11-15 12:16:32 -05:00
John Colvin 45fcb2e20b refactor: Use HostBuilder to launch spark w/ streams 2025-11-15 12:16:32 -05:00
John Colvin 15883e42b6 fix (dsqa): Resolve name conflicts with new BlueOrb streams 2025-11-14 15:02:16 -05:00
John Colvin 34caf54015 fix: Fix typo in "DateTimeExpansionStream" type name
Previously, it was "DateTimeExpanstion" (erroneous "t")
2025-11-14 15:02:16 -05:00