John Colvin
0d10c2ba5d
new: Add GetColumn fn to IBlueRowCollection interface
...
This allows us to implement python/pandas functionality, much of which is based
on column operations.
2025-11-15 12:38:50 -05:00
John Colvin
75f2678df6
cleanup: Fix build warnings
...
* BlueOrbNLStream.cs
* ChangeTableNameStream.cs
* FilterStream.cs
* FilterStream.cs
* GroupByOutputTable.cs
* GroupByStream.cs
* JoinForeignTable.cs
* MultiRowRollingMathStream.cs
* MutableAppendStream.cs
* PersistWindowState.cs
* PubsubEchoTestStream.cs
* RowDictionaryTable.cs
* SortedTable.cs
* TimeSeriesStream.cs
Also add Metadata prop to IDataPacket
2025-11-15 12:38:49 -05:00
John Colvin
8bbe850549
chore: Add 'Rows' prop to BlueTable to avoid accidentally iterating over KeyValuePairs
2025-11-15 12:38:49 -05:00
John Colvin
25860af7bf
refactor: Use dependency injection in Spark.Console
2025-11-15 12:38:49 -05:00
John Colvin
57b82cf10d
fix: Use IDictionary.GetEnumerator explicitly to iterate keyvals
...
We really need to replace these old non-generic collections with generic ones.
BlueRow's IDictionaryEnumerator implementation iterates over values when passed
to a foreach loop, not key value pairs.
2025-11-15 12:38:49 -05:00
John Colvin
f07d83e7bb
refactor: Make CompositeDataStreamFactory class internal
2025-11-15 12:38:49 -05:00
John Colvin
760cdc0fba
cleanup: Fix build warnings, remove unused/redundant code
...
* Make a few member vars and fns local to the constructor
* Replace a couple non-generic collections with generic ones
* Fix warnings and clean code in:
- DictionaryExtensions.cs
* Remove ext fns that are part of dotnet now
- JoinNameResolver.cs
- MainForm.cs
cleanup: Remove unused code in ladder
Some props in cs, and StaticResources in xaml were leftover after I combined
the panel price points into a single collection, bound to an ItemsControl.
2025-11-15 12:38:49 -05:00
John Colvin
fcd1b0e799
refactor: Use generic collection types in PrimitiveManager
2025-11-15 12:38:49 -05:00
John Colvin
eb102a5898
cleanup: Remove unused code in PrimitiveManager.cs
2025-11-15 12:38:48 -05:00
John Colvin
fb3262edea
cleanup: Use file-scoped namespace in PrimitiveManager.cs
...
Also sort using statements
2025-11-15 12:38:48 -05:00
John Colvin
6e1e5d4bbf
refactor: Add IDataStreamFactory(s) for spatial arb streams
2025-11-15 12:38:48 -05:00
John Colvin
24892fb1a8
cleanup: Remove unused BooleanExpressionToken Negate
2025-11-15 12:38:48 -05:00
John Colvin
29080795f7
refactor: Rename AddRemoteQueryManager -> AddRemoteQueryStreams
2025-11-15 12:38:48 -05:00
John Colvin
cd291b7a0d
refactor: Hide md client behind interface, create via Dep Injection
2025-11-15 12:38:48 -05:00
John Colvin
c4f12be3e4
cleanup: Fix a ton of build warnings (cs)
...
All these warnings have been bothering me for a while. The work isn't done, but
this commit chops them down from ~1000 to ~600.
* AppendCollection.cs
* AppendStreamModified.cs
* BenzingaNewsStream.cs
* BenzingaNewsSymbolStateStream.cs
* BookStream.cs
* CSVSaverStream.cs
* ChartsUtils.cs
* ColumnInfo.cs
* ContractUtils.cs
* ContractUtils.cs
* DiffStream.cs
* FuturesActiveContractStream.cs
* FuzzyJoinOverTimeStream.cs
* HistoricalBarsStream.cs
* HistoricalBarsStream.cs
* InstrumentsStream.cs
* JoinOperandRowIndex.cs
* JoinRow.cs
* JoinStream.cs
* JoinTable.cs
* JoinTableSchema.cs
* LatencyStream.cs
* LeftInternalJoinStream.cs
* LightForeignJoinRow.cs
* LightForeignJoinRowList.cs
* LightLeftForeignJoinRow.cs
* LiteralStream.cs
* NLStream.cs
* NLToDSLStream.cs
* NLToIndexedFewShotPromptStream.cs
* NonIncGroupByStream.cs
* NonIncJoinBookkeeper.cs
* NonIncJoinSpec.cs
* OHLCVStream.cs
* OpenAIPromptStream.cs
* PriceStatsStream.cs
* PriceStatsStream.cs
* ProjectHelper.cs
* ProjectStream.cs
* PubSubServerStateStream.cs
* PubsubHeartbeatStream.cs
* SimpleNode.cs
* SortRow.cs
* SpatialArbStatsStream.cs
* SpatialFuturePairsStream.cs
* SpreadPerformanceEvaluator.cs
* SpreadPerformanceStream.cs
* StreamIOCommon.cs
* SuperJoinStream.cs
* TableMetaDataStream.cs
* TextWriterStream.cs
* TimeAverageStream.cs
2025-11-15 12:16:33 -05:00
John Colvin
54d7279832
style: Normalize indentation in PubsubHeartbeatStream.cs
2025-11-15 12:16:33 -05:00
John Colvin
03df578a81
chore: Add GetValue, TryGetValue ext fns to IBlueRow
2025-11-15 12:16:32 -05:00
John Colvin
7a50ad2cc5
chore: Add 'Empty' fn for BlueTable/Schema (for convenience)
2025-11-15 12:16:32 -05:00
John Colvin
0cc4b83417
cleanup: Remove redundant code block in HistoricalBarsStream.cs
2025-11-15 12:16:32 -05:00
John Colvin
4f1efea066
cleanup: Fix indentation and build warnings in TestStream.cs
2025-11-15 12:16:32 -05:00
John Colvin
e2816aaa33
refactor: Decouple ViewStreamDatabase from TableStream, ISubscriberCore
2025-11-15 12:16:32 -05:00
John Colvin
e2a101eb20
feat: Add SpreadBook stream
2025-11-15 12:16:32 -05:00
John Colvin
45fcb2e20b
refactor: Use HostBuilder to launch spark w/ streams
2025-11-15 12:16:32 -05:00
John Colvin
15883e42b6
fix (dsqa): Resolve name conflicts with new BlueOrb streams
2025-11-14 15:02:16 -05:00
John Colvin
34caf54015
fix: Fix typo in "DateTimeExpansionStream" type name
...
Previously, it was "DateTimeExpanstion" (erroneous "t")
2025-11-14 15:02:16 -05:00
John Colvin
05644ed6cf
feat: Add DataStream WebApp
...
Included in DataStreamSuite.sln, but not DataStreamQueryAnalyzer.sln
2025-11-14 15:00:45 -05:00
John Colvin
d6a88a7ac9
cleanup: Remove unused code
2025-11-14 14:59:33 -05:00
John Colvin
f7e620db3e
Reduce log noise in AsyncTCPConnection_AsyncWrapper
2025-11-14 14:59:33 -05:00
John Colvin
c745407ae7
TEMP changes to appsettings.json (at least i should review them before making permanent)
2025-11-14 14:59:33 -05:00
John Colvin
1bb7d6eca2
Add ColumnIndex class to resolve column names to indices
...
Improves performance compared to accessing columns by name
2025-11-14 14:59:33 -05:00
John Colvin
519ef8b0af
Increase SQLiteEnumerator page size to 65536
...
This performs better, and doesn't seem to be too large in practice
2025-11-14 14:59:33 -05:00
John Colvin
6b93c07271
Serialize TimeSpan w/ microsecond precision in DataPacket json
2025-11-14 14:59:33 -05:00
John Colvin
c3288c721d
Format DateTimes w/ microsecond precision in DataPacketJsonConverter
2025-11-14 14:59:33 -05:00
John Colvin
2622626eed
Add BlueColumnTable for pandas integration
...
This will make it easier/more natural to support pandas column expressions
2025-11-14 14:59:33 -05:00
John Colvin
f119f30b62
fix load IDataStreamFactory(s) with depdendency injection
2025-11-14 14:57:43 -05:00
John Colvin
4877355fee
refactor: Use dependency injection in DSQA
2025-11-14 14:52:07 -05:00
John Colvin
ada7d4bed1
refactor: Load IDataStreamFactory(s) with dependency injection
...
So we can create ViewStreams that depend on other services, config, etc.
Previously, the PrimitiveManager relied purely on static construction of
types through reflection.
2025-11-14 14:52:07 -05:00
John Colvin
f65b6b82f2
refactor: Replace DataPacket w/ IDataPacket in stream handlers
...
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-03 16:21:32 -05:00
John Colvin
1155fd6997
refactor: Change namespace to Bts.DataStreamCore for a few types
...
Since they're in the DataStreamCore csproj
2025-11-03 16:19:50 -05:00
John Colvin
28545b5099
fix: Don't assert on exception in BooleanExpression.evaluate
...
Bugs (fixed):
* DataStream WebApp Server sometimes crashes when the user enters a
bad query.
2025-11-03 16:19:48 -05:00
John Colvin
04c1ad7be8
Assert that local ip address is successfully obtained
2025-11-03 16:19:48 -05:00
John Colvin
2441a5267d
improvement: Resolve hostnames when comparing ip addrs
...
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2025-11-03 16:19:48 -05:00
John Colvin
12f44f63aa
Assert TcpConnection endpoint matches expected
2025-11-03 16:19:48 -05:00
John Colvin
c2fe578ccb
Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
...
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2025-11-03 16:19:48 -05:00
John Colvin
a01fc6cc15
fix: Format PubSubRoute as json in log msgs
...
It was previously using the default ToString, which simply returns the type name
2025-11-03 16:19:48 -05:00
John Colvin
d80cd9c0fd
build: Upgrade packages (deps) in DataStreamTests
2025-11-03 16:19:48 -05:00
John Colvin
d707847079
cleanup: Remove unused ViewStreamDatabase config
2025-11-03 16:19:48 -05:00
John Colvin
118e2a9f2e
Remove reference to spark protobuf Config in ViewStream
...
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-03 16:19:48 -05:00
John Colvin
1481a3fd07
refactor: Run RemoteQueryManager as an IHostedService
...
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-11-03 16:19:45 -05:00
John Colvin
8637bb9254
Assert ViewStreamDatabase.instance is only created once
...
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-03 16:13:44 -05:00