John Colvin
6045c602dc
feat: Add pseudo member (extension) function calls to DSL
2025-11-15 12:45:07 -05:00
John Colvin
4a2a814af0
refactor: Move StreamQueryMetadata into StreamFactories namespace
2025-11-15 12:45:07 -05:00
John Colvin
08a655b1d5
Change WellKnownColumn names to match those in mlp
...
This is easier than changing them everywhere in the master learning prompt.
2025-11-15 12:45:07 -05:00
John Colvin
2830af5644
improve: Add config for natural language streams
...
They're created by a stream factory now, which has the config handy.
Config includes api key, model name, mlp file, etc.
fix add config for natural language streams
2025-11-15 12:45:07 -05:00
John Colvin
659eb83f4d
refactor: Drop null type modifier in TryGetFirst ext
...
Replace w/ MaybeNullWhen attributes
2025-11-15 12:45:07 -05:00
John Colvin
5d9d364de3
cleanup: Remove unnecessary code
2025-11-15 12:45:07 -05:00
John Colvin
07c1ad3204
qol: Don't connect to sparkd by default in Spark.Console cli
2025-11-15 12:45:07 -05:00
John Colvin
e61829f8e0
qol: Change default pubsub url from 'pubsub' to 127.0.0.1
...
The hostname 'pubsub' works when running pubsub and other services
via docker compose. In practice, it seems we usually just run them on a
single host, without using docker compose.
2025-11-15 12:45:07 -05:00
John Colvin
0cf44fcc40
feat: Add historic market data streams factory
...
Loads historic md from Databento files in structured paths. It is
designed to work with files downloaded via databento_client.py
2025-11-15 12:45:04 -05:00
John Colvin
2549e34a17
Create LiveBBOStream via BookStreamFactory
...
Instead of PrimitiveManager
2025-11-15 12:42:40 -05:00
John Colvin
693f2deb67
fix: Don't modify input env in BoundBooleanExpression.Evaluate
2025-11-15 12:42:40 -05:00
John Colvin
63665e9b10
refactor: Pass IMarketDataClient to streams via dependency injection
...
In general, the streams should not instantiate services, etc. themselves.
2025-11-15 12:42:40 -05:00
John Colvin
a79f5ff860
improve: Support all integer types in IBlueRow Id ext fn
2025-11-15 12:42:40 -05:00
John Colvin
dae75c507e
cleanup: Remove unused code CSharpString.cs
2025-11-15 12:42:40 -05:00
John Colvin
4db9cdd357
feat: Add DiffStream to compute row diffs (like pandas)
2025-11-15 12:42:40 -05:00
John Colvin
18b19ac787
refactor: Rewrite SortStream now that IBlueRowCollection is IList
2025-11-15 12:42:40 -05:00
John Colvin
b5ea918378
refactor: Add param for TimeSeriesStream timestamp col name
...
Default it to 'ts_event' instead of 'timestamp' (this is consistent w/
Databento schema fwiw)
2025-11-15 12:42:40 -05:00
John Colvin
9df4ecf6c8
fix: Trim backticks from nl generated dsl
2025-11-15 12:42:40 -05:00
John Colvin
957f6cb33f
feat: Add FeatherReader stream
...
We don't need this right now, but I already wrote it...it might be
useful later.
2025-11-15 12:42:39 -05:00
John Colvin
03294ffe37
feat: Add function defs to DataStream DSL
2025-11-15 12:42:39 -05:00
John Colvin
0550078de0
chore: Support mutations in AggregateDictionary (optionally)
2025-11-15 12:42:39 -05:00
John Colvin
89129f726e
chore: Implement IComparable in BookTuple
2025-11-15 12:42:39 -05:00
John Colvin
97eefb1a1e
improve: Serialize to/from parquet in PythonDataframeStream
2025-11-15 12:42:39 -05:00
John Colvin
805693e170
feat: Add ParquetReader/Writer streams
2025-11-15 12:40:48 -05:00
John Colvin
3a7f3ac608
fix: Fire multiple rows at once in replay bbo/trade streams doWork fn
...
If necessary to keep pace with timestamps in the file.
2025-11-15 12:38:51 -05:00
John Colvin
b82cc31d2d
refactor: Launch LiveTradeStream via IDataStreamFactory
2025-11-15 12:38:51 -05:00
John Colvin
34859d7506
chore: Add id field to TradeRecord
...
Need this for streams
2025-11-15 12:38:51 -05:00
John Colvin
8a9bc1872e
fix: Actually use sec filter in MarketDataClient.Trades
2025-11-15 12:38:51 -05:00
John Colvin
6b4ff90a56
style: Rename dispose_thread_safe in PascalCase
2025-11-15 12:38:51 -05:00
John Colvin
d2a1b67717
refactor: Make IBlueRowCollection inherit IList<IBlueRow>
2025-11-15 12:38:51 -05:00
John Colvin
144b29c728
improvement: Lookup timestamp col by index rather than name
2025-11-15 12:38:51 -05:00
John Colvin
e7625341e3
refactor: Remove hardcoded csv path from ReplayFutureLiveBBOStream
...
Add constructor overload to accept an input ViewStream
2025-11-15 12:38:51 -05:00
John Colvin
fb15b3fb24
improvement: Don't mutate IBlueRowCollection after firing in FilterStream
2025-11-15 12:38:51 -05:00
John Colvin
ea0246929a
improvement: Make MultiRowRollingMathStream more performant
...
Avoid creating extra lists and o(n) enumeration
2025-11-15 12:38:50 -05:00
John Colvin
52f35cc421
refactor: Move IBlueTableSchema ext fns into DataStreamCore
...
The csproj in which IBlueTableSchema is defined.
2025-11-15 12:38:50 -05:00
John Colvin
7e968fec76
tool: Add databento_client.py to fetch (and store) databento data
2025-11-15 12:38:50 -05:00
John Colvin
188acc08ca
chore: Copy/paste some log utils from py interop dir to tools/py
2025-11-15 12:38:50 -05:00
John Colvin
29be8c3ff5
refactor: Support dict and list of tuples in py fmt_table
2025-11-15 12:38:50 -05:00
John Colvin
eb28e04bf1
refactor: Replace old NLStreamFactory w/ new BlueOrb one
...
Replace hardcoded file paths with proper config while we're at it
2025-11-15 12:38:50 -05:00
John Colvin
4ef2dbe4d7
refactor: Pass composite stream factory back to recursive factories as service
2025-11-15 12:38:50 -05:00
John Colvin
979051f147
build: Add appsettings.Local.json to DSQA
2025-11-15 12:38:50 -05:00
John Colvin
35a17b0fd0
refactor: Support random access (IList) to data in TableUpdate
...
We're doing the same thing to IBlueRowCollection. This will allow us to
implement column based operators, like python pandas.
2025-11-15 12:38:50 -05:00
John Colvin
d93ce336f6
Add TestCommand back to Spark.Console cli
2025-11-15 12:38:50 -05:00
John Colvin
0d10c2ba5d
new: Add GetColumn fn to IBlueRowCollection interface
...
This allows us to implement python/pandas functionality, much of which is based
on column operations.
2025-11-15 12:38:50 -05:00
John Colvin
2a4d62a1e9
chore: Add various collection classes
...
These will be useful in stream operators, row indexing, etc.
* AggregateList
* DeltaList
* DictionarySetWrapper
* ImmutableOrderedDictionary
* IMutableListFacade
- inherits IReadOnlyList<T> and IList<T>
- throws NotSupportedException in default impls for mutation ops
* IMutableCollectionFacade
- inherits IReadOnlyCollection<T> and ICollection<T>
- throws NotSupportedException in default impls for mutation ops
* MergedSortedList
* OrderedDictionary
* ReverseArrayList
* SliceList
* StepList
2025-11-15 12:38:50 -05:00
John Colvin
b9e191eeb9
refactor: Drop nullable modifier for TValue in GetValueOrDefault ext fn
2025-11-15 12:38:50 -05:00
John Colvin
75f2678df6
cleanup: Fix build warnings
...
* BlueOrbNLStream.cs
* ChangeTableNameStream.cs
* FilterStream.cs
* FilterStream.cs
* GroupByOutputTable.cs
* GroupByStream.cs
* JoinForeignTable.cs
* MultiRowRollingMathStream.cs
* MutableAppendStream.cs
* PersistWindowState.cs
* PubsubEchoTestStream.cs
* RowDictionaryTable.cs
* SortedTable.cs
* TimeSeriesStream.cs
Also add Metadata prop to IDataPacket
2025-11-15 12:38:49 -05:00
John Colvin
8bbe850549
chore: Add 'Rows' prop to BlueTable to avoid accidentally iterating over KeyValuePairs
2025-11-15 12:38:49 -05:00
John Colvin
25860af7bf
refactor: Use dependency injection in Spark.Console
2025-11-15 12:38:49 -05:00
John Colvin
68198a058e
cleanup: Remove broken settings cmd from Spark.Console
2025-11-15 12:38:49 -05:00