John Colvin
0cf44fcc40
feat: Add historic market data streams factory
...
Loads historic md from Databento files in structured paths. It is
designed to work with files downloaded via databento_client.py
2025-11-15 12:45:04 -05:00
John Colvin
2549e34a17
Create LiveBBOStream via BookStreamFactory
...
Instead of PrimitiveManager
2025-11-15 12:42:40 -05:00
John Colvin
693f2deb67
fix: Don't modify input env in BoundBooleanExpression.Evaluate
2025-11-15 12:42:40 -05:00
John Colvin
63665e9b10
refactor: Pass IMarketDataClient to streams via dependency injection
...
In general, the streams should not instantiate services, etc. themselves.
2025-11-15 12:42:40 -05:00
John Colvin
a79f5ff860
improve: Support all integer types in IBlueRow Id ext fn
2025-11-15 12:42:40 -05:00
John Colvin
dae75c507e
cleanup: Remove unused code CSharpString.cs
2025-11-15 12:42:40 -05:00
John Colvin
4db9cdd357
feat: Add DiffStream to compute row diffs (like pandas)
2025-11-15 12:42:40 -05:00
John Colvin
18b19ac787
refactor: Rewrite SortStream now that IBlueRowCollection is IList
2025-11-15 12:42:40 -05:00
John Colvin
b5ea918378
refactor: Add param for TimeSeriesStream timestamp col name
...
Default it to 'ts_event' instead of 'timestamp' (this is consistent w/
Databento schema fwiw)
2025-11-15 12:42:40 -05:00
John Colvin
9df4ecf6c8
fix: Trim backticks from nl generated dsl
2025-11-15 12:42:40 -05:00
John Colvin
957f6cb33f
feat: Add FeatherReader stream
...
We don't need this right now, but I already wrote it...it might be
useful later.
2025-11-15 12:42:39 -05:00
John Colvin
03294ffe37
feat: Add function defs to DataStream DSL
2025-11-15 12:42:39 -05:00
John Colvin
97eefb1a1e
improve: Serialize to/from parquet in PythonDataframeStream
2025-11-15 12:42:39 -05:00
John Colvin
805693e170
feat: Add ParquetReader/Writer streams
2025-11-15 12:40:48 -05:00
John Colvin
3a7f3ac608
fix: Fire multiple rows at once in replay bbo/trade streams doWork fn
...
If necessary to keep pace with timestamps in the file.
2025-11-15 12:38:51 -05:00
John Colvin
b82cc31d2d
refactor: Launch LiveTradeStream via IDataStreamFactory
2025-11-15 12:38:51 -05:00
John Colvin
d2a1b67717
refactor: Make IBlueRowCollection inherit IList<IBlueRow>
2025-11-15 12:38:51 -05:00
John Colvin
144b29c728
improvement: Lookup timestamp col by index rather than name
2025-11-15 12:38:51 -05:00
John Colvin
e7625341e3
refactor: Remove hardcoded csv path from ReplayFutureLiveBBOStream
...
Add constructor overload to accept an input ViewStream
2025-11-15 12:38:51 -05:00
John Colvin
fb15b3fb24
improvement: Don't mutate IBlueRowCollection after firing in FilterStream
2025-11-15 12:38:51 -05:00
John Colvin
ea0246929a
improvement: Make MultiRowRollingMathStream more performant
...
Avoid creating extra lists and o(n) enumeration
2025-11-15 12:38:50 -05:00
John Colvin
52f35cc421
refactor: Move IBlueTableSchema ext fns into DataStreamCore
...
The csproj in which IBlueTableSchema is defined.
2025-11-15 12:38:50 -05:00
John Colvin
eb28e04bf1
refactor: Replace old NLStreamFactory w/ new BlueOrb one
...
Replace hardcoded file paths with proper config while we're at it
2025-11-15 12:38:50 -05:00
John Colvin
4ef2dbe4d7
refactor: Pass composite stream factory back to recursive factories as service
2025-11-15 12:38:50 -05:00
John Colvin
979051f147
build: Add appsettings.Local.json to DSQA
2025-11-15 12:38:50 -05:00
John Colvin
0d10c2ba5d
new: Add GetColumn fn to IBlueRowCollection interface
...
This allows us to implement python/pandas functionality, much of which is based
on column operations.
2025-11-15 12:38:50 -05:00
John Colvin
75f2678df6
cleanup: Fix build warnings
...
* BlueOrbNLStream.cs
* ChangeTableNameStream.cs
* FilterStream.cs
* FilterStream.cs
* GroupByOutputTable.cs
* GroupByStream.cs
* JoinForeignTable.cs
* MultiRowRollingMathStream.cs
* MutableAppendStream.cs
* PersistWindowState.cs
* PubsubEchoTestStream.cs
* RowDictionaryTable.cs
* SortedTable.cs
* TimeSeriesStream.cs
Also add Metadata prop to IDataPacket
2025-11-15 12:38:49 -05:00
John Colvin
8bbe850549
chore: Add 'Rows' prop to BlueTable to avoid accidentally iterating over KeyValuePairs
2025-11-15 12:38:49 -05:00
John Colvin
25860af7bf
refactor: Use dependency injection in Spark.Console
2025-11-15 12:38:49 -05:00
John Colvin
57b82cf10d
fix: Use IDictionary.GetEnumerator explicitly to iterate keyvals
...
We really need to replace these old non-generic collections with generic ones.
BlueRow's IDictionaryEnumerator implementation iterates over values when passed
to a foreach loop, not key value pairs.
2025-11-15 12:38:49 -05:00
John Colvin
f07d83e7bb
refactor: Make CompositeDataStreamFactory class internal
2025-11-15 12:38:49 -05:00
John Colvin
760cdc0fba
cleanup: Fix build warnings, remove unused/redundant code
...
* Make a few member vars and fns local to the constructor
* Replace a couple non-generic collections with generic ones
* Fix warnings and clean code in:
- DictionaryExtensions.cs
* Remove ext fns that are part of dotnet now
- JoinNameResolver.cs
- MainForm.cs
cleanup: Remove unused code in ladder
Some props in cs, and StaticResources in xaml were leftover after I combined
the panel price points into a single collection, bound to an ItemsControl.
2025-11-15 12:38:49 -05:00
John Colvin
fcd1b0e799
refactor: Use generic collection types in PrimitiveManager
2025-11-15 12:38:49 -05:00
John Colvin
eb102a5898
cleanup: Remove unused code in PrimitiveManager.cs
2025-11-15 12:38:48 -05:00
John Colvin
fb3262edea
cleanup: Use file-scoped namespace in PrimitiveManager.cs
...
Also sort using statements
2025-11-15 12:38:48 -05:00
John Colvin
6e1e5d4bbf
refactor: Add IDataStreamFactory(s) for spatial arb streams
2025-11-15 12:38:48 -05:00
John Colvin
24892fb1a8
cleanup: Remove unused BooleanExpressionToken Negate
2025-11-15 12:38:48 -05:00
John Colvin
29080795f7
refactor: Rename AddRemoteQueryManager -> AddRemoteQueryStreams
2025-11-15 12:38:48 -05:00
John Colvin
cd291b7a0d
refactor: Hide md client behind interface, create via Dep Injection
2025-11-15 12:38:48 -05:00
John Colvin
c4f12be3e4
cleanup: Fix a ton of build warnings (cs)
...
All these warnings have been bothering me for a while. The work isn't done, but
this commit chops them down from ~1000 to ~600.
* AppendCollection.cs
* AppendStreamModified.cs
* BenzingaNewsStream.cs
* BenzingaNewsSymbolStateStream.cs
* BookStream.cs
* CSVSaverStream.cs
* ChartsUtils.cs
* ColumnInfo.cs
* ContractUtils.cs
* ContractUtils.cs
* DiffStream.cs
* FuturesActiveContractStream.cs
* FuzzyJoinOverTimeStream.cs
* HistoricalBarsStream.cs
* HistoricalBarsStream.cs
* InstrumentsStream.cs
* JoinOperandRowIndex.cs
* JoinRow.cs
* JoinStream.cs
* JoinTable.cs
* JoinTableSchema.cs
* LatencyStream.cs
* LeftInternalJoinStream.cs
* LightForeignJoinRow.cs
* LightForeignJoinRowList.cs
* LightLeftForeignJoinRow.cs
* LiteralStream.cs
* NLStream.cs
* NLToDSLStream.cs
* NLToIndexedFewShotPromptStream.cs
* NonIncGroupByStream.cs
* NonIncJoinBookkeeper.cs
* NonIncJoinSpec.cs
* OHLCVStream.cs
* OpenAIPromptStream.cs
* PriceStatsStream.cs
* PriceStatsStream.cs
* ProjectHelper.cs
* ProjectStream.cs
* PubSubServerStateStream.cs
* PubsubHeartbeatStream.cs
* SimpleNode.cs
* SortRow.cs
* SpatialArbStatsStream.cs
* SpatialFuturePairsStream.cs
* SpreadPerformanceEvaluator.cs
* SpreadPerformanceStream.cs
* StreamIOCommon.cs
* SuperJoinStream.cs
* TableMetaDataStream.cs
* TextWriterStream.cs
* TimeAverageStream.cs
2025-11-15 12:16:33 -05:00
John Colvin
54d7279832
style: Normalize indentation in PubsubHeartbeatStream.cs
2025-11-15 12:16:33 -05:00
John Colvin
03df578a81
chore: Add GetValue, TryGetValue ext fns to IBlueRow
2025-11-15 12:16:32 -05:00
John Colvin
7a50ad2cc5
chore: Add 'Empty' fn for BlueTable/Schema (for convenience)
2025-11-15 12:16:32 -05:00
John Colvin
0cc4b83417
cleanup: Remove redundant code block in HistoricalBarsStream.cs
2025-11-15 12:16:32 -05:00
John Colvin
4f1efea066
cleanup: Fix indentation and build warnings in TestStream.cs
2025-11-15 12:16:32 -05:00
John Colvin
e2816aaa33
refactor: Decouple ViewStreamDatabase from TableStream, ISubscriberCore
2025-11-15 12:16:32 -05:00
John Colvin
e2a101eb20
feat: Add SpreadBook stream
2025-11-15 12:16:32 -05:00
John Colvin
45fcb2e20b
refactor: Use HostBuilder to launch spark w/ streams
2025-11-15 12:16:32 -05:00
John Colvin
15883e42b6
fix (dsqa): Resolve name conflicts with new BlueOrb streams
2025-11-14 15:02:16 -05:00
John Colvin
34caf54015
fix: Fix typo in "DateTimeExpansionStream" type name
...
Previously, it was "DateTimeExpanstion" (erroneous "t")
2025-11-14 15:02:16 -05:00