Commit Graph
14455 Commits
Author SHA1 Message Date
John Colvin d7300cbddb refactor: Launch LiveTradeStream via IDataStreamFactory 2025-11-19 14:27:48 -05:00
John Colvin c21cd2ba6d chore: Add id field to TradeRecord
Need this for streams
2025-11-19 14:27:48 -05:00
John Colvin 488fbe3c43 fix: Actually use sec filter in MarketDataClient.Trades 2025-11-19 14:27:48 -05:00
John Colvin d25db044af style: Rename dispose_thread_safe in PascalCase 2025-11-19 14:27:48 -05:00
John Colvin d3e31e4e44 refactor: Make IBlueRowCollection inherit IList<IBlueRow> 2025-11-19 14:27:46 -05:00
John Colvin 4fd2073b63 improvement: Lookup timestamp col by index rather than name 2025-11-19 14:19:21 -05:00
John Colvin 0b64bd6544 refactor: Remove hardcoded csv path from ReplayFutureLiveBBOStream
Add constructor overload to accept an input ViewStream
2025-11-19 14:19:19 -05:00
John Colvin d1fc4d4ce9 improvement: Don't mutate IBlueRowCollection after firing in FilterStream 2025-11-19 14:17:05 -05:00
John Colvin 654ef44664 improvement: Make MultiRowRollingMathStream more performant
Avoid creating extra lists and o(n) enumeration
2025-11-19 14:17:05 -05:00
John Colvin 2f83499524 refactor: Move IBlueTableSchema ext fns into DataStreamCore
The csproj in which IBlueTableSchema is defined.
2025-11-19 14:17:03 -05:00
John Colvin cfbd324b6f tool: Add databento_client.py to fetch (and store) databento data 2025-11-19 14:15:01 -05:00
John Colvin ae513ed247 chore: Copy/paste some log utils from py interop dir to tools/py 2025-11-19 14:15:01 -05:00
John Colvin c965f125ed refactor: Support dict and list of tuples in py fmt_table 2025-11-19 14:15:01 -05:00
John Colvin 41aa5ec834 refactor: Replace old NLStreamFactory w/ new BlueOrb one
Replace hardcoded file paths with proper config while we're at it
2025-11-19 14:15:01 -05:00
John Colvin 7651397e22 refactor: Pass composite stream factory back to recursive factories as service 2025-11-19 14:15:01 -05:00
John Colvin 41a800aac6 build: Add appsettings.Local.json to DSQA 2025-11-19 14:15:01 -05:00
John Colvin c549b51d80 refactor: Support random access (IList) to data in TableUpdate
We're doing the same thing to IBlueRowCollection. This will allow us to
implement column based operators, like python pandas.
2025-11-19 14:15:01 -05:00
John Colvin 37c1eda425 Add TestCommand back to Spark.Console cli 2025-11-19 14:15:01 -05:00
John Colvin 52ad459c19 new: Add GetColumn fn to IBlueRowCollection interface
This allows us to implement python/pandas functionality, much of which is based
on column operations.
2025-11-19 14:14:58 -05:00
John Colvin d6158bc9a0 chore: Add various collection classes
These will be useful in stream operators, row indexing, etc.

* AggregateList
* DeltaList
* DictionarySetWrapper
* ImmutableOrderedDictionary
* IMutableListFacade
  - inherits IReadOnlyList<T> and IList<T>
  - throws NotSupportedException in default impls for mutation ops
* IMutableCollectionFacade
  - inherits IReadOnlyCollection<T> and ICollection<T>
  - throws NotSupportedException in default impls for mutation ops
* MergedSortedList
* OrderedDictionary
* ReverseArrayList
* SliceList
* StepList
2025-11-19 14:03:46 -05:00
John Colvin 85babef37b refactor: Drop nullable modifier for TValue in GetValueOrDefault ext fn 2025-11-19 14:03:46 -05:00
John Colvin fb3a6262b3 cleanup: Fix build warnings
* BlueOrbNLStream.cs
* ChangeTableNameStream.cs
* FilterStream.cs
* FilterStream.cs
* GroupByOutputTable.cs
* GroupByStream.cs
* JoinForeignTable.cs
* MultiRowRollingMathStream.cs
* MutableAppendStream.cs
* PersistWindowState.cs
* PubsubEchoTestStream.cs
* RowDictionaryTable.cs
* SortedTable.cs
* TimeSeriesStream.cs

Also add Metadata prop to IDataPacket
2025-11-19 14:03:43 -05:00
John Colvin a3bee2313a chore: Add 'Rows' prop to BlueTable to avoid accidentally iterating over KeyValuePairs 2025-11-19 14:01:02 -05:00
John Colvin d8d9a25f22 refactor: Use dependency injection in Spark.Console 2025-11-19 14:01:02 -05:00
John Colvin 45fcea5670 cleanup: Remove broken settings cmd from Spark.Console 2025-11-19 14:01:01 -05:00
John Colvin 33f42c3097 refactor: Use top-level statements for Spark.Console 'main' 2025-11-19 14:01:01 -05:00
John Colvin 0c5e9c6945 refactor: Consolidate SparkApp and ShellBootstrapper into single class 2025-11-19 14:01:01 -05:00
John Colvin 65a9f31f14 feat: Add stream driven price lines to ladder 2025-11-19 14:01:01 -05:00
John Colvin 789e8debc6 fix: Use IDictionary.GetEnumerator explicitly to iterate keyvals
We really need to replace these old non-generic collections with generic ones.
BlueRow's IDictionaryEnumerator implementation iterates over values when passed
to a foreach loop, not key value pairs.
2025-11-19 14:01:01 -05:00
John Colvin c821a7ed43 refactor: Make CompositeDataStreamFactory class internal 2025-11-19 14:01:01 -05:00
John Colvin 12a3e7a2f4 feat (ui): Add ConfigServerPicker dialog
So we can connect to different config servers without manually editing spark.conf,
or changing command line args. This will run before the Login dialog.
2025-11-19 14:01:01 -05:00
John Colvin 5045b1889d refactor: Move ComboBox styles to ComboBoxes.xaml
I'm working on a ConfigServerPicker dialog, which will run before the main
application starts. This allows the dialog to use the same dropdown styles as
the main app without the overhead of loading generic.xaml.
2025-11-19 14:01:01 -05:00
John Colvin a50a71f058 cleanup: Fix build warnings, remove unused/redundant code
* Make a few member vars and fns local to the constructor
* Replace a couple non-generic collections with generic ones
* Fix warnings and clean code in:
  - DictionaryExtensions.cs
    * Remove ext fns that are part of dotnet now
  - JoinNameResolver.cs
  - MainForm.cs

cleanup: Remove unused code in ladder

Some props in cs, and StaticResources in xaml were leftover after I combined
the panel price points into a single collection, bound to an ItemsControl.
2025-11-19 14:01:01 -05:00
John Colvin f67ab3c458 refactor: Use generic collection types in PrimitiveManager 2025-11-19 14:01:01 -05:00
John Colvin 88233a6315 cleanup: Remove unused code in PrimitiveManager.cs 2025-11-19 14:01:01 -05:00
John Colvin 20db00e832 cleanup: Use file-scoped namespace in PrimitiveManager.cs
Also sort using statements
2025-11-19 14:01:01 -05:00
John Colvin 6d298a0e34 refactor: Add IDataStreamFactory(s) for spatial arb streams 2025-11-19 14:01:01 -05:00
John Colvin 8f76487974 tool: Run Spark.exe from its own directory
It looks in the working directory for runtime config appsettings.json files.
2025-11-19 14:01:01 -05:00
John Colvin 0d60cc75cc cleanup: Remove unused BooleanExpressionToken Negate 2025-11-19 14:01:01 -05:00
John Colvin 62d07ada0f doc: Comment why MDConsumer must be kept alive
The language server suggests removing it because it isn't referenced.
However, it contains unmanaged resources that mustn't be prematurely
garbage collected.
2025-11-19 14:01:00 -05:00
John Colvin bb29c32059 refactor: Rename AddRemoteQueryManager -> AddRemoteQueryStreams 2025-11-19 14:01:00 -05:00
John Colvin a358fe6229 refactor: Hide md client behind interface, create via Dep Injection 2025-11-19 14:00:58 -05:00
John Colvin f7ce346925 Drop dump-books and dump-securities from spark_api_server cli
These were subcommands were out of place. This exe has one purpose: run
the Spark Api Server.
2025-11-19 13:57:48 -05:00
John Colvin e7129b1513 cleanup: Remove unused MarketDataClient from OrderSample 2025-11-19 13:57:48 -05:00
John Colvin 27ee09ade8 build: Launch console for Spark Debug builds, otherwise hide it 2025-11-19 13:57:47 -05:00
John Colvin 63a64af46b cleanup: Remove ancient winforms sdk ui example 2025-11-19 13:57:47 -05:00
John Colvin a4efaa0089 cleanup: Fix a ton of build warnings (cs)
All these warnings have been bothering me for a while. The work isn't done, but
this commit chops them down from ~1000 to ~600.

* AppendCollection.cs
* AppendStreamModified.cs
* BenzingaNewsStream.cs
* BenzingaNewsSymbolStateStream.cs
* BookStream.cs
* CSVSaverStream.cs
* ChartsUtils.cs
* ColumnInfo.cs
* ContractUtils.cs
* ContractUtils.cs
* DiffStream.cs
* FuturesActiveContractStream.cs
* FuzzyJoinOverTimeStream.cs
* HistoricalBarsStream.cs
* HistoricalBarsStream.cs
* InstrumentsStream.cs
* JoinOperandRowIndex.cs
* JoinRow.cs
* JoinStream.cs
* JoinTable.cs
* JoinTableSchema.cs
* LatencyStream.cs
* LeftInternalJoinStream.cs
* LightForeignJoinRow.cs
* LightForeignJoinRowList.cs
* LightLeftForeignJoinRow.cs
* LiteralStream.cs
* NLStream.cs
* NLToDSLStream.cs
* NLToIndexedFewShotPromptStream.cs
* NonIncGroupByStream.cs
* NonIncJoinBookkeeper.cs
* NonIncJoinSpec.cs
* OHLCVStream.cs
* OpenAIPromptStream.cs
* PriceStatsStream.cs
* PriceStatsStream.cs
* ProjectHelper.cs
* ProjectStream.cs
* PubSubServerStateStream.cs
* PubsubHeartbeatStream.cs
* SimpleNode.cs
* SortRow.cs
* SpatialArbStatsStream.cs
* SpatialFuturePairsStream.cs
* SpreadPerformanceEvaluator.cs
* SpreadPerformanceStream.cs
* StreamIOCommon.cs
* SuperJoinStream.cs
* TableMetaDataStream.cs
* TextWriterStream.cs
* TimeAverageStream.cs
2025-11-19 13:57:47 -05:00
John Colvin 7418f5b5a1 style: Normalize indentation in PubsubHeartbeatStream.cs 2025-11-19 13:57:47 -05:00
John Colvin b854390c85 chore: Add GetValue, TryGetValue ext fns to IBlueRow 2025-11-19 13:57:47 -05:00
John Colvin a2f9359077 chore: Add 'Empty' fn for BlueTable/Schema (for convenience) 2025-11-19 13:57:47 -05:00