Commit Graph
14501 Commits
Author SHA1 Message Date
John Colvin 94e9c8c3f2 improve: Generate MLP at runtime with list of example files
Adds prompt file list to NaturalLanguageStreamFactoryConfig
2025-11-15 12:46:24 -05:00
John Colvin 0f9e515d01 chore: Add TableDropIndexEvaluator 2025-11-15 12:46:24 -05:00
John Colvin 342ecd9a52 chore: Add SelectorStream/Table 2025-11-15 12:46:24 -05:00
John Colvin d0c316b3fc chore: Add IndexStream/Table and DropIndex ext fn similar to pandas 2025-11-15 12:46:24 -05:00
John Colvin 44db6190a1 fix: Make PythonDataframeStream thread-safe
Input/output happen on different threads
2025-11-15 12:46:24 -05:00
John Colvin 9856e10eb6 fix: Support some mutations with convert_back delegate in SelectorList 2025-11-15 12:46:24 -05:00
John Colvin deda37b057 chore: Add arithmetic operators to PrimitiveVariant 2025-11-15 12:46:24 -05:00
John Colvin ce5095523b feat: Add a bunch of stream/table ext evaluators 2025-11-15 12:46:24 -05:00
John Colvin 7d374346f2 cleanup: Add NotNull attr to "assert_value" nullable ext 2025-11-15 12:46:24 -05:00
John Colvin ad0e9fa3c9 chore: Add BookDelta class 2025-11-15 12:46:24 -05:00
John Colvin befa17d29d chore: Add DoesNotReturnIf(false) attr to unit test Assert condition 2025-11-15 12:46:24 -05:00
John Colvin 3165ba1235 Add some half-baked Brandon Chen streams 2025-11-15 12:46:23 -05:00
John Colvin 53a9e6a62d fix: Support numeric conversions in ArgumentComparer 2025-11-15 12:46:23 -05:00
John Colvin ff816befb0 chore: Add MergeSorted ext fn to merge 2 already sorted enumerables 2025-11-15 12:46:23 -05:00
John Colvin a69c79b6a9 Add TableUpdate 'WithHistory' ext fn
A buffer of the last N updates is included with each update
2025-11-15 12:46:23 -05:00
John Colvin 7219ff5a95 Show IDataStreamFactory operators in DSQA 2025-11-15 12:46:23 -05:00
John Colvin 603f01699c Tweak databento_client.py 2025-11-15 12:46:23 -05:00
John Colvin 0ca318b222 refactor: Launch ThrottleStream via IDataStreamFactory
Not PrimitiveManager
2025-11-15 12:45:08 -05:00
John Colvin 4addb58743 feat: Add BboPriceSummaryStream for Brandon Chen 2025-11-15 12:45:08 -05:00
John Colvin c365330b08 feat: Add GroupByApplyStream 2025-11-15 12:45:08 -05:00
John Colvin 4bb4008ac6 feat: Add SliceTable/Stream 2025-11-15 12:45:07 -05:00
John Colvin 001892ab34 Jupyter/pandas analysis for brandon chen 2025-11-15 12:45:07 -05:00
John Colvin e77ee859a3 fix: Preserve clipboard (copy/paste) when DSQA exits 2025-11-15 12:45:07 -05:00
John Colvin c1a8702a47 ui: Improve dateTime formatting in DSQA 2025-11-15 12:45:07 -05:00
John Colvin 6045c602dc feat: Add pseudo member (extension) function calls to DSL 2025-11-15 12:45:07 -05:00
John Colvin 4a2a814af0 refactor: Move StreamQueryMetadata into StreamFactories namespace 2025-11-15 12:45:07 -05:00
John Colvin 08a655b1d5 Change WellKnownColumn names to match those in mlp
This is easier than changing them everywhere in the master learning prompt.
2025-11-15 12:45:07 -05:00
John Colvin 2830af5644 improve: Add config for natural language streams
They're created by a stream factory now, which has the config handy.
Config includes api key, model name, mlp file, etc.

fix add config for natural language streams
2025-11-15 12:45:07 -05:00
John Colvin 659eb83f4d refactor: Drop null type modifier in TryGetFirst ext
Replace w/ MaybeNullWhen attributes
2025-11-15 12:45:07 -05:00
John Colvin 5d9d364de3 cleanup: Remove unnecessary code 2025-11-15 12:45:07 -05:00
John Colvin 07c1ad3204 qol: Don't connect to sparkd by default in Spark.Console cli 2025-11-15 12:45:07 -05:00
John Colvin e61829f8e0 qol: Change default pubsub url from 'pubsub' to 127.0.0.1
The hostname 'pubsub' works when running pubsub and other services
via docker compose. In practice, it seems we usually just run them on a
single host, without using docker compose.
2025-11-15 12:45:07 -05:00
John Colvin 0cf44fcc40 feat: Add historic market data streams factory
Loads historic md from Databento files in structured paths. It is
designed to work with files downloaded via databento_client.py
2025-11-15 12:45:04 -05:00
John Colvin 2549e34a17 Create LiveBBOStream via BookStreamFactory
Instead of PrimitiveManager
2025-11-15 12:42:40 -05:00
John Colvin 693f2deb67 fix: Don't modify input env in BoundBooleanExpression.Evaluate 2025-11-15 12:42:40 -05:00
John Colvin 63665e9b10 refactor: Pass IMarketDataClient to streams via dependency injection
In general, the streams should not instantiate services, etc. themselves.
2025-11-15 12:42:40 -05:00
John Colvin a79f5ff860 improve: Support all integer types in IBlueRow Id ext fn 2025-11-15 12:42:40 -05:00
John Colvin dae75c507e cleanup: Remove unused code CSharpString.cs 2025-11-15 12:42:40 -05:00
John Colvin 4db9cdd357 feat: Add DiffStream to compute row diffs (like pandas) 2025-11-15 12:42:40 -05:00
John Colvin 18b19ac787 refactor: Rewrite SortStream now that IBlueRowCollection is IList 2025-11-15 12:42:40 -05:00
John Colvin b5ea918378 refactor: Add param for TimeSeriesStream timestamp col name
Default it to 'ts_event' instead of 'timestamp' (this is consistent w/
Databento schema fwiw)
2025-11-15 12:42:40 -05:00
John Colvin 9df4ecf6c8 fix: Trim backticks from nl generated dsl 2025-11-15 12:42:40 -05:00
John Colvin 957f6cb33f feat: Add FeatherReader stream
We don't need this right now, but I already wrote it...it might be
useful later.
2025-11-15 12:42:39 -05:00
John Colvin 03294ffe37 feat: Add function defs to DataStream DSL 2025-11-15 12:42:39 -05:00
John Colvin 0550078de0 chore: Support mutations in AggregateDictionary (optionally) 2025-11-15 12:42:39 -05:00
John Colvin 89129f726e chore: Implement IComparable in BookTuple 2025-11-15 12:42:39 -05:00
John Colvin 97eefb1a1e improve: Serialize to/from parquet in PythonDataframeStream 2025-11-15 12:42:39 -05:00
John Colvin 805693e170 feat: Add ParquetReader/Writer streams 2025-11-15 12:40:48 -05:00
John Colvin 3a7f3ac608 fix: Fire multiple rows at once in replay bbo/trade streams doWork fn
If necessary to keep pace with timestamps in the file.
2025-11-15 12:38:51 -05:00
John Colvin b82cc31d2d refactor: Launch LiveTradeStream via IDataStreamFactory 2025-11-15 12:38:51 -05:00