Commit Graph
14477 Commits
Author SHA1 Message Date
John Colvin 6045c602dc feat: Add pseudo member (extension) function calls to DSL 2025-11-15 12:45:07 -05:00
John Colvin 4a2a814af0 refactor: Move StreamQueryMetadata into StreamFactories namespace 2025-11-15 12:45:07 -05:00
John Colvin 08a655b1d5 Change WellKnownColumn names to match those in mlp
This is easier than changing them everywhere in the master learning prompt.
2025-11-15 12:45:07 -05:00
John Colvin 2830af5644 improve: Add config for natural language streams
They're created by a stream factory now, which has the config handy.
Config includes api key, model name, mlp file, etc.

fix add config for natural language streams
2025-11-15 12:45:07 -05:00
John Colvin 659eb83f4d refactor: Drop null type modifier in TryGetFirst ext
Replace w/ MaybeNullWhen attributes
2025-11-15 12:45:07 -05:00
John Colvin 5d9d364de3 cleanup: Remove unnecessary code 2025-11-15 12:45:07 -05:00
John Colvin 07c1ad3204 qol: Don't connect to sparkd by default in Spark.Console cli 2025-11-15 12:45:07 -05:00
John Colvin e61829f8e0 qol: Change default pubsub url from 'pubsub' to 127.0.0.1
The hostname 'pubsub' works when running pubsub and other services
via docker compose. In practice, it seems we usually just run them on a
single host, without using docker compose.
2025-11-15 12:45:07 -05:00
John Colvin 0cf44fcc40 feat: Add historic market data streams factory
Loads historic md from Databento files in structured paths. It is
designed to work with files downloaded via databento_client.py
2025-11-15 12:45:04 -05:00
John Colvin 2549e34a17 Create LiveBBOStream via BookStreamFactory
Instead of PrimitiveManager
2025-11-15 12:42:40 -05:00
John Colvin 693f2deb67 fix: Don't modify input env in BoundBooleanExpression.Evaluate 2025-11-15 12:42:40 -05:00
John Colvin 63665e9b10 refactor: Pass IMarketDataClient to streams via dependency injection
In general, the streams should not instantiate services, etc. themselves.
2025-11-15 12:42:40 -05:00
John Colvin a79f5ff860 improve: Support all integer types in IBlueRow Id ext fn 2025-11-15 12:42:40 -05:00
John Colvin dae75c507e cleanup: Remove unused code CSharpString.cs 2025-11-15 12:42:40 -05:00
John Colvin 4db9cdd357 feat: Add DiffStream to compute row diffs (like pandas) 2025-11-15 12:42:40 -05:00
John Colvin 18b19ac787 refactor: Rewrite SortStream now that IBlueRowCollection is IList 2025-11-15 12:42:40 -05:00
John Colvin b5ea918378 refactor: Add param for TimeSeriesStream timestamp col name
Default it to 'ts_event' instead of 'timestamp' (this is consistent w/
Databento schema fwiw)
2025-11-15 12:42:40 -05:00
John Colvin 9df4ecf6c8 fix: Trim backticks from nl generated dsl 2025-11-15 12:42:40 -05:00
John Colvin 957f6cb33f feat: Add FeatherReader stream
We don't need this right now, but I already wrote it...it might be
useful later.
2025-11-15 12:42:39 -05:00
John Colvin 03294ffe37 feat: Add function defs to DataStream DSL 2025-11-15 12:42:39 -05:00
John Colvin 0550078de0 chore: Support mutations in AggregateDictionary (optionally) 2025-11-15 12:42:39 -05:00
John Colvin 89129f726e chore: Implement IComparable in BookTuple 2025-11-15 12:42:39 -05:00
John Colvin 97eefb1a1e improve: Serialize to/from parquet in PythonDataframeStream 2025-11-15 12:42:39 -05:00
John Colvin 805693e170 feat: Add ParquetReader/Writer streams 2025-11-15 12:40:48 -05:00
John Colvin 3a7f3ac608 fix: Fire multiple rows at once in replay bbo/trade streams doWork fn
If necessary to keep pace with timestamps in the file.
2025-11-15 12:38:51 -05:00
John Colvin b82cc31d2d refactor: Launch LiveTradeStream via IDataStreamFactory 2025-11-15 12:38:51 -05:00
John Colvin 34859d7506 chore: Add id field to TradeRecord
Need this for streams
2025-11-15 12:38:51 -05:00
John Colvin 8a9bc1872e fix: Actually use sec filter in MarketDataClient.Trades 2025-11-15 12:38:51 -05:00
John Colvin 6b4ff90a56 style: Rename dispose_thread_safe in PascalCase 2025-11-15 12:38:51 -05:00
John Colvin d2a1b67717 refactor: Make IBlueRowCollection inherit IList<IBlueRow> 2025-11-15 12:38:51 -05:00
John Colvin 144b29c728 improvement: Lookup timestamp col by index rather than name 2025-11-15 12:38:51 -05:00
John Colvin e7625341e3 refactor: Remove hardcoded csv path from ReplayFutureLiveBBOStream
Add constructor overload to accept an input ViewStream
2025-11-15 12:38:51 -05:00
John Colvin fb15b3fb24 improvement: Don't mutate IBlueRowCollection after firing in FilterStream 2025-11-15 12:38:51 -05:00
John Colvin ea0246929a improvement: Make MultiRowRollingMathStream more performant
Avoid creating extra lists and o(n) enumeration
2025-11-15 12:38:50 -05:00
John Colvin 52f35cc421 refactor: Move IBlueTableSchema ext fns into DataStreamCore
The csproj in which IBlueTableSchema is defined.
2025-11-15 12:38:50 -05:00
John Colvin 7e968fec76 tool: Add databento_client.py to fetch (and store) databento data 2025-11-15 12:38:50 -05:00
John Colvin 188acc08ca chore: Copy/paste some log utils from py interop dir to tools/py 2025-11-15 12:38:50 -05:00
John Colvin 29be8c3ff5 refactor: Support dict and list of tuples in py fmt_table 2025-11-15 12:38:50 -05:00
John Colvin eb28e04bf1 refactor: Replace old NLStreamFactory w/ new BlueOrb one
Replace hardcoded file paths with proper config while we're at it
2025-11-15 12:38:50 -05:00
John Colvin 4ef2dbe4d7 refactor: Pass composite stream factory back to recursive factories as service 2025-11-15 12:38:50 -05:00
John Colvin 979051f147 build: Add appsettings.Local.json to DSQA 2025-11-15 12:38:50 -05:00
John Colvin 35a17b0fd0 refactor: Support random access (IList) to data in TableUpdate
We're doing the same thing to IBlueRowCollection. This will allow us to
implement column based operators, like python pandas.
2025-11-15 12:38:50 -05:00
John Colvin d93ce336f6 Add TestCommand back to Spark.Console cli 2025-11-15 12:38:50 -05:00
John Colvin 0d10c2ba5d new: Add GetColumn fn to IBlueRowCollection interface
This allows us to implement python/pandas functionality, much of which is based
on column operations.
2025-11-15 12:38:50 -05:00
John Colvin 2a4d62a1e9 chore: Add various collection classes
These will be useful in stream operators, row indexing, etc.

* AggregateList
* DeltaList
* DictionarySetWrapper
* ImmutableOrderedDictionary
* IMutableListFacade
  - inherits IReadOnlyList<T> and IList<T>
  - throws NotSupportedException in default impls for mutation ops
* IMutableCollectionFacade
  - inherits IReadOnlyCollection<T> and ICollection<T>
  - throws NotSupportedException in default impls for mutation ops
* MergedSortedList
* OrderedDictionary
* ReverseArrayList
* SliceList
* StepList
2025-11-15 12:38:50 -05:00
John Colvin b9e191eeb9 refactor: Drop nullable modifier for TValue in GetValueOrDefault ext fn 2025-11-15 12:38:50 -05:00
John Colvin 75f2678df6 cleanup: Fix build warnings
* BlueOrbNLStream.cs
* ChangeTableNameStream.cs
* FilterStream.cs
* FilterStream.cs
* GroupByOutputTable.cs
* GroupByStream.cs
* JoinForeignTable.cs
* MultiRowRollingMathStream.cs
* MutableAppendStream.cs
* PersistWindowState.cs
* PubsubEchoTestStream.cs
* RowDictionaryTable.cs
* SortedTable.cs
* TimeSeriesStream.cs

Also add Metadata prop to IDataPacket
2025-11-15 12:38:49 -05:00
John Colvin 8bbe850549 chore: Add 'Rows' prop to BlueTable to avoid accidentally iterating over KeyValuePairs 2025-11-15 12:38:49 -05:00
John Colvin 25860af7bf refactor: Use dependency injection in Spark.Console 2025-11-15 12:38:49 -05:00
John Colvin 68198a058e cleanup: Remove broken settings cmd from Spark.Console 2025-11-15 12:38:49 -05:00