Commit Graph
14458 Commits
Author SHA1 Message Date
John Colvin 03294ffe37 feat: Add function defs to DataStream DSL 2025-11-15 12:42:39 -05:00
John Colvin 0550078de0 chore: Support mutations in AggregateDictionary (optionally) 2025-11-15 12:42:39 -05:00
John Colvin 89129f726e chore: Implement IComparable in BookTuple 2025-11-15 12:42:39 -05:00
John Colvin 97eefb1a1e improve: Serialize to/from parquet in PythonDataframeStream 2025-11-15 12:42:39 -05:00
John Colvin 805693e170 feat: Add ParquetReader/Writer streams 2025-11-15 12:40:48 -05:00
John Colvin 3a7f3ac608 fix: Fire multiple rows at once in replay bbo/trade streams doWork fn
If necessary to keep pace with timestamps in the file.
2025-11-15 12:38:51 -05:00
John Colvin b82cc31d2d refactor: Launch LiveTradeStream via IDataStreamFactory 2025-11-15 12:38:51 -05:00
John Colvin 34859d7506 chore: Add id field to TradeRecord
Need this for streams
2025-11-15 12:38:51 -05:00
John Colvin 8a9bc1872e fix: Actually use sec filter in MarketDataClient.Trades 2025-11-15 12:38:51 -05:00
John Colvin 6b4ff90a56 style: Rename dispose_thread_safe in PascalCase 2025-11-15 12:38:51 -05:00
John Colvin d2a1b67717 refactor: Make IBlueRowCollection inherit IList<IBlueRow> 2025-11-15 12:38:51 -05:00
John Colvin 144b29c728 improvement: Lookup timestamp col by index rather than name 2025-11-15 12:38:51 -05:00
John Colvin e7625341e3 refactor: Remove hardcoded csv path from ReplayFutureLiveBBOStream
Add constructor overload to accept an input ViewStream
2025-11-15 12:38:51 -05:00
John Colvin fb15b3fb24 improvement: Don't mutate IBlueRowCollection after firing in FilterStream 2025-11-15 12:38:51 -05:00
John Colvin ea0246929a improvement: Make MultiRowRollingMathStream more performant
Avoid creating extra lists and o(n) enumeration
2025-11-15 12:38:50 -05:00
John Colvin 52f35cc421 refactor: Move IBlueTableSchema ext fns into DataStreamCore
The csproj in which IBlueTableSchema is defined.
2025-11-15 12:38:50 -05:00
John Colvin 7e968fec76 tool: Add databento_client.py to fetch (and store) databento data 2025-11-15 12:38:50 -05:00
John Colvin 188acc08ca chore: Copy/paste some log utils from py interop dir to tools/py 2025-11-15 12:38:50 -05:00
John Colvin 29be8c3ff5 refactor: Support dict and list of tuples in py fmt_table 2025-11-15 12:38:50 -05:00
John Colvin eb28e04bf1 refactor: Replace old NLStreamFactory w/ new BlueOrb one
Replace hardcoded file paths with proper config while we're at it
2025-11-15 12:38:50 -05:00
John Colvin 4ef2dbe4d7 refactor: Pass composite stream factory back to recursive factories as service 2025-11-15 12:38:50 -05:00
John Colvin 979051f147 build: Add appsettings.Local.json to DSQA 2025-11-15 12:38:50 -05:00
John Colvin 35a17b0fd0 refactor: Support random access (IList) to data in TableUpdate
We're doing the same thing to IBlueRowCollection. This will allow us to
implement column based operators, like python pandas.
2025-11-15 12:38:50 -05:00
John Colvin d93ce336f6 Add TestCommand back to Spark.Console cli 2025-11-15 12:38:50 -05:00
John Colvin 0d10c2ba5d new: Add GetColumn fn to IBlueRowCollection interface
This allows us to implement python/pandas functionality, much of which is based
on column operations.
2025-11-15 12:38:50 -05:00
John Colvin 2a4d62a1e9 chore: Add various collection classes
These will be useful in stream operators, row indexing, etc.

* AggregateList
* DeltaList
* DictionarySetWrapper
* ImmutableOrderedDictionary
* IMutableListFacade
  - inherits IReadOnlyList<T> and IList<T>
  - throws NotSupportedException in default impls for mutation ops
* IMutableCollectionFacade
  - inherits IReadOnlyCollection<T> and ICollection<T>
  - throws NotSupportedException in default impls for mutation ops
* MergedSortedList
* OrderedDictionary
* ReverseArrayList
* SliceList
* StepList
2025-11-15 12:38:50 -05:00
John Colvin b9e191eeb9 refactor: Drop nullable modifier for TValue in GetValueOrDefault ext fn 2025-11-15 12:38:50 -05:00
John Colvin 75f2678df6 cleanup: Fix build warnings
* BlueOrbNLStream.cs
* ChangeTableNameStream.cs
* FilterStream.cs
* FilterStream.cs
* GroupByOutputTable.cs
* GroupByStream.cs
* JoinForeignTable.cs
* MultiRowRollingMathStream.cs
* MutableAppendStream.cs
* PersistWindowState.cs
* PubsubEchoTestStream.cs
* RowDictionaryTable.cs
* SortedTable.cs
* TimeSeriesStream.cs

Also add Metadata prop to IDataPacket
2025-11-15 12:38:49 -05:00
John Colvin 8bbe850549 chore: Add 'Rows' prop to BlueTable to avoid accidentally iterating over KeyValuePairs 2025-11-15 12:38:49 -05:00
John Colvin 25860af7bf refactor: Use dependency injection in Spark.Console 2025-11-15 12:38:49 -05:00
John Colvin 68198a058e cleanup: Remove broken settings cmd from Spark.Console 2025-11-15 12:38:49 -05:00
John Colvin 7297d667a2 refactor: Use top-level statements for Spark.Console 'main' 2025-11-15 12:38:49 -05:00
John Colvin ace1b5f04d refactor: Consolidate SparkApp and ShellBootstrapper into single class 2025-11-15 12:38:49 -05:00
John Colvin bd252c251c feat: Add stream driven price lines to ladder 2025-11-15 12:38:49 -05:00
John Colvin 57b82cf10d fix: Use IDictionary.GetEnumerator explicitly to iterate keyvals
We really need to replace these old non-generic collections with generic ones.
BlueRow's IDictionaryEnumerator implementation iterates over values when passed
to a foreach loop, not key value pairs.
2025-11-15 12:38:49 -05:00
John Colvin f07d83e7bb refactor: Make CompositeDataStreamFactory class internal 2025-11-15 12:38:49 -05:00
John Colvin 619be37379 feat (ui): Add ConfigServerPicker dialog
So we can connect to different config servers without manually editing spark.conf,
or changing command line args. This will run before the Login dialog.
2025-11-15 12:38:49 -05:00
John Colvin 9348106585 refactor: Move ComboBox styles to ComboBoxes.xaml
I'm working on a ConfigServerPicker dialog, which will run before the main
application starts. This allows the dialog to use the same dropdown styles as
the main app without the overhead of loading generic.xaml.
2025-11-15 12:38:49 -05:00
John Colvin 760cdc0fba cleanup: Fix build warnings, remove unused/redundant code
* Make a few member vars and fns local to the constructor
* Replace a couple non-generic collections with generic ones
* Fix warnings and clean code in:
  - DictionaryExtensions.cs
    * Remove ext fns that are part of dotnet now
  - JoinNameResolver.cs
  - MainForm.cs

cleanup: Remove unused code in ladder

Some props in cs, and StaticResources in xaml were leftover after I combined
the panel price points into a single collection, bound to an ItemsControl.
2025-11-15 12:38:49 -05:00
John Colvin fcd1b0e799 refactor: Use generic collection types in PrimitiveManager 2025-11-15 12:38:49 -05:00
John Colvin eb102a5898 cleanup: Remove unused code in PrimitiveManager.cs 2025-11-15 12:38:48 -05:00
John Colvin fb3262edea cleanup: Use file-scoped namespace in PrimitiveManager.cs
Also sort using statements
2025-11-15 12:38:48 -05:00
John Colvin 6e1e5d4bbf refactor: Add IDataStreamFactory(s) for spatial arb streams 2025-11-15 12:38:48 -05:00
John Colvin 256f7025a2 tool: Run Spark.exe from its own directory
It looks in the working directory for runtime config appsettings.json files.
2025-11-15 12:38:48 -05:00
John Colvin 24892fb1a8 cleanup: Remove unused BooleanExpressionToken Negate 2025-11-15 12:38:48 -05:00
John Colvin e3c2efcfcb doc: Comment why MDConsumer must be kept alive
The language server suggests removing it because it isn't referenced.
However, it contains unmanaged resources that mustn't be prematurely
garbage collected.
2025-11-15 12:38:48 -05:00
John Colvin 29080795f7 refactor: Rename AddRemoteQueryManager -> AddRemoteQueryStreams 2025-11-15 12:38:48 -05:00
John Colvin cd291b7a0d refactor: Hide md client behind interface, create via Dep Injection 2025-11-15 12:38:48 -05:00
John Colvin 8af9d13802 Drop dump-books and dump-securities from spark_api_server cli
These were subcommands were out of place. This exe has one purpose: run
the Spark Api Server.
2025-11-15 12:38:48 -05:00
John Colvin 318d8fb275 cleanup: Remove unused MarketDataClient from OrderSample 2025-11-15 12:38:48 -05:00