Commit Graph
303 Commits
Author SHA1 Message Date
John Colvin e7625341e3 refactor: Remove hardcoded csv path from ReplayFutureLiveBBOStream
Add constructor overload to accept an input ViewStream
2025-11-15 12:38:51 -05:00
John Colvin fb15b3fb24 improvement: Don't mutate IBlueRowCollection after firing in FilterStream 2025-11-15 12:38:51 -05:00
John Colvin ea0246929a improvement: Make MultiRowRollingMathStream more performant
Avoid creating extra lists and o(n) enumeration
2025-11-15 12:38:50 -05:00
John Colvin 52f35cc421 refactor: Move IBlueTableSchema ext fns into DataStreamCore
The csproj in which IBlueTableSchema is defined.
2025-11-15 12:38:50 -05:00
John Colvin eb28e04bf1 refactor: Replace old NLStreamFactory w/ new BlueOrb one
Replace hardcoded file paths with proper config while we're at it
2025-11-15 12:38:50 -05:00
John Colvin 4ef2dbe4d7 refactor: Pass composite stream factory back to recursive factories as service 2025-11-15 12:38:50 -05:00
John Colvin 979051f147 build: Add appsettings.Local.json to DSQA 2025-11-15 12:38:50 -05:00
John Colvin 0d10c2ba5d new: Add GetColumn fn to IBlueRowCollection interface
This allows us to implement python/pandas functionality, much of which is based
on column operations.
2025-11-15 12:38:50 -05:00
John Colvin 75f2678df6 cleanup: Fix build warnings
* BlueOrbNLStream.cs
* ChangeTableNameStream.cs
* FilterStream.cs
* FilterStream.cs
* GroupByOutputTable.cs
* GroupByStream.cs
* JoinForeignTable.cs
* MultiRowRollingMathStream.cs
* MutableAppendStream.cs
* PersistWindowState.cs
* PubsubEchoTestStream.cs
* RowDictionaryTable.cs
* SortedTable.cs
* TimeSeriesStream.cs

Also add Metadata prop to IDataPacket
2025-11-15 12:38:49 -05:00
John Colvin 8bbe850549 chore: Add 'Rows' prop to BlueTable to avoid accidentally iterating over KeyValuePairs 2025-11-15 12:38:49 -05:00
John Colvin 25860af7bf refactor: Use dependency injection in Spark.Console 2025-11-15 12:38:49 -05:00
John Colvin 57b82cf10d fix: Use IDictionary.GetEnumerator explicitly to iterate keyvals
We really need to replace these old non-generic collections with generic ones.
BlueRow's IDictionaryEnumerator implementation iterates over values when passed
to a foreach loop, not key value pairs.
2025-11-15 12:38:49 -05:00
John Colvin f07d83e7bb refactor: Make CompositeDataStreamFactory class internal 2025-11-15 12:38:49 -05:00
John Colvin 760cdc0fba cleanup: Fix build warnings, remove unused/redundant code
* Make a few member vars and fns local to the constructor
* Replace a couple non-generic collections with generic ones
* Fix warnings and clean code in:
  - DictionaryExtensions.cs
    * Remove ext fns that are part of dotnet now
  - JoinNameResolver.cs
  - MainForm.cs

cleanup: Remove unused code in ladder

Some props in cs, and StaticResources in xaml were leftover after I combined
the panel price points into a single collection, bound to an ItemsControl.
2025-11-15 12:38:49 -05:00
John Colvin fcd1b0e799 refactor: Use generic collection types in PrimitiveManager 2025-11-15 12:38:49 -05:00
John Colvin eb102a5898 cleanup: Remove unused code in PrimitiveManager.cs 2025-11-15 12:38:48 -05:00
John Colvin fb3262edea cleanup: Use file-scoped namespace in PrimitiveManager.cs
Also sort using statements
2025-11-15 12:38:48 -05:00
John Colvin 6e1e5d4bbf refactor: Add IDataStreamFactory(s) for spatial arb streams 2025-11-15 12:38:48 -05:00
John Colvin 24892fb1a8 cleanup: Remove unused BooleanExpressionToken Negate 2025-11-15 12:38:48 -05:00
John Colvin 29080795f7 refactor: Rename AddRemoteQueryManager -> AddRemoteQueryStreams 2025-11-15 12:38:48 -05:00
John Colvin cd291b7a0d refactor: Hide md client behind interface, create via Dep Injection 2025-11-15 12:38:48 -05:00
John Colvin c4f12be3e4 cleanup: Fix a ton of build warnings (cs)
All these warnings have been bothering me for a while. The work isn't done, but
this commit chops them down from ~1000 to ~600.

* AppendCollection.cs
* AppendStreamModified.cs
* BenzingaNewsStream.cs
* BenzingaNewsSymbolStateStream.cs
* BookStream.cs
* CSVSaverStream.cs
* ChartsUtils.cs
* ColumnInfo.cs
* ContractUtils.cs
* ContractUtils.cs
* DiffStream.cs
* FuturesActiveContractStream.cs
* FuzzyJoinOverTimeStream.cs
* HistoricalBarsStream.cs
* HistoricalBarsStream.cs
* InstrumentsStream.cs
* JoinOperandRowIndex.cs
* JoinRow.cs
* JoinStream.cs
* JoinTable.cs
* JoinTableSchema.cs
* LatencyStream.cs
* LeftInternalJoinStream.cs
* LightForeignJoinRow.cs
* LightForeignJoinRowList.cs
* LightLeftForeignJoinRow.cs
* LiteralStream.cs
* NLStream.cs
* NLToDSLStream.cs
* NLToIndexedFewShotPromptStream.cs
* NonIncGroupByStream.cs
* NonIncJoinBookkeeper.cs
* NonIncJoinSpec.cs
* OHLCVStream.cs
* OpenAIPromptStream.cs
* PriceStatsStream.cs
* PriceStatsStream.cs
* ProjectHelper.cs
* ProjectStream.cs
* PubSubServerStateStream.cs
* PubsubHeartbeatStream.cs
* SimpleNode.cs
* SortRow.cs
* SpatialArbStatsStream.cs
* SpatialFuturePairsStream.cs
* SpreadPerformanceEvaluator.cs
* SpreadPerformanceStream.cs
* StreamIOCommon.cs
* SuperJoinStream.cs
* TableMetaDataStream.cs
* TextWriterStream.cs
* TimeAverageStream.cs
2025-11-15 12:16:33 -05:00
John Colvin 54d7279832 style: Normalize indentation in PubsubHeartbeatStream.cs 2025-11-15 12:16:33 -05:00
John Colvin 03df578a81 chore: Add GetValue, TryGetValue ext fns to IBlueRow 2025-11-15 12:16:32 -05:00
John Colvin 7a50ad2cc5 chore: Add 'Empty' fn for BlueTable/Schema (for convenience) 2025-11-15 12:16:32 -05:00
John Colvin 0cc4b83417 cleanup: Remove redundant code block in HistoricalBarsStream.cs 2025-11-15 12:16:32 -05:00
John Colvin 4f1efea066 cleanup: Fix indentation and build warnings in TestStream.cs 2025-11-15 12:16:32 -05:00
John Colvin e2816aaa33 refactor: Decouple ViewStreamDatabase from TableStream, ISubscriberCore 2025-11-15 12:16:32 -05:00
John Colvin e2a101eb20 feat: Add SpreadBook stream 2025-11-15 12:16:32 -05:00
John Colvin 45fcb2e20b refactor: Use HostBuilder to launch spark w/ streams 2025-11-15 12:16:32 -05:00
John Colvin 15883e42b6 fix (dsqa): Resolve name conflicts with new BlueOrb streams 2025-11-14 15:02:16 -05:00
John Colvin 34caf54015 fix: Fix typo in "DateTimeExpansionStream" type name
Previously, it was "DateTimeExpanstion" (erroneous "t")
2025-11-14 15:02:16 -05:00
John Colvin 05644ed6cf feat: Add DataStream WebApp
Included in DataStreamSuite.sln, but not DataStreamQueryAnalyzer.sln
2025-11-14 15:00:45 -05:00
John Colvin d6a88a7ac9 cleanup: Remove unused code 2025-11-14 14:59:33 -05:00
John Colvin f7e620db3e Reduce log noise in AsyncTCPConnection_AsyncWrapper 2025-11-14 14:59:33 -05:00
John Colvin c745407ae7 TEMP changes to appsettings.json (at least i should review them before making permanent) 2025-11-14 14:59:33 -05:00
John Colvin 1bb7d6eca2 Add ColumnIndex class to resolve column names to indices
Improves performance compared to accessing columns by name
2025-11-14 14:59:33 -05:00
John Colvin 519ef8b0af Increase SQLiteEnumerator page size to 65536
This performs better, and doesn't seem to be too large in practice
2025-11-14 14:59:33 -05:00
John Colvin 6b93c07271 Serialize TimeSpan w/ microsecond precision in DataPacket json 2025-11-14 14:59:33 -05:00
John Colvin c3288c721d Format DateTimes w/ microsecond precision in DataPacketJsonConverter 2025-11-14 14:59:33 -05:00
John Colvin 2622626eed Add BlueColumnTable for pandas integration
This will make it easier/more natural to support pandas column expressions
2025-11-14 14:59:33 -05:00
John Colvin f119f30b62 fix load IDataStreamFactory(s) with depdendency injection 2025-11-14 14:57:43 -05:00
John Colvin 4877355fee refactor: Use dependency injection in DSQA 2025-11-14 14:52:07 -05:00
John Colvin ada7d4bed1 refactor: Load IDataStreamFactory(s) with dependency injection
So we can create ViewStreams that depend on other services, config, etc.
Previously, the PrimitiveManager relied purely on static construction of
types through reflection.
2025-11-14 14:52:07 -05:00
John Colvin f65b6b82f2 refactor: Replace DataPacket w/ IDataPacket in stream handlers
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-03 16:21:32 -05:00
John Colvin 1155fd6997 refactor: Change namespace to Bts.DataStreamCore for a few types
Since they're in the DataStreamCore csproj
2025-11-03 16:19:50 -05:00
John Colvin 28545b5099 fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2025-11-03 16:19:48 -05:00
John Colvin 04c1ad7be8 Assert that local ip address is successfully obtained 2025-11-03 16:19:48 -05:00
John Colvin 2441a5267d improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2025-11-03 16:19:48 -05:00
John Colvin 12f44f63aa Assert TcpConnection endpoint matches expected 2025-11-03 16:19:48 -05:00