Commit Graph
282 Commits
Author SHA1 Message Date
John Colvin c4f12be3e4 cleanup: Fix a ton of build warnings (cs)
All these warnings have been bothering me for a while. The work isn't done, but
this commit chops them down from ~1000 to ~600.

* AppendCollection.cs
* AppendStreamModified.cs
* BenzingaNewsStream.cs
* BenzingaNewsSymbolStateStream.cs
* BookStream.cs
* CSVSaverStream.cs
* ChartsUtils.cs
* ColumnInfo.cs
* ContractUtils.cs
* ContractUtils.cs
* DiffStream.cs
* FuturesActiveContractStream.cs
* FuzzyJoinOverTimeStream.cs
* HistoricalBarsStream.cs
* HistoricalBarsStream.cs
* InstrumentsStream.cs
* JoinOperandRowIndex.cs
* JoinRow.cs
* JoinStream.cs
* JoinTable.cs
* JoinTableSchema.cs
* LatencyStream.cs
* LeftInternalJoinStream.cs
* LightForeignJoinRow.cs
* LightForeignJoinRowList.cs
* LightLeftForeignJoinRow.cs
* LiteralStream.cs
* NLStream.cs
* NLToDSLStream.cs
* NLToIndexedFewShotPromptStream.cs
* NonIncGroupByStream.cs
* NonIncJoinBookkeeper.cs
* NonIncJoinSpec.cs
* OHLCVStream.cs
* OpenAIPromptStream.cs
* PriceStatsStream.cs
* PriceStatsStream.cs
* ProjectHelper.cs
* ProjectStream.cs
* PubSubServerStateStream.cs
* PubsubHeartbeatStream.cs
* SimpleNode.cs
* SortRow.cs
* SpatialArbStatsStream.cs
* SpatialFuturePairsStream.cs
* SpreadPerformanceEvaluator.cs
* SpreadPerformanceStream.cs
* StreamIOCommon.cs
* SuperJoinStream.cs
* TableMetaDataStream.cs
* TextWriterStream.cs
* TimeAverageStream.cs
2025-11-15 12:16:33 -05:00
John Colvin 54d7279832 style: Normalize indentation in PubsubHeartbeatStream.cs 2025-11-15 12:16:33 -05:00
John Colvin 03df578a81 chore: Add GetValue, TryGetValue ext fns to IBlueRow 2025-11-15 12:16:32 -05:00
John Colvin 7a50ad2cc5 chore: Add 'Empty' fn for BlueTable/Schema (for convenience) 2025-11-15 12:16:32 -05:00
John Colvin 0cc4b83417 cleanup: Remove redundant code block in HistoricalBarsStream.cs 2025-11-15 12:16:32 -05:00
John Colvin 4f1efea066 cleanup: Fix indentation and build warnings in TestStream.cs 2025-11-15 12:16:32 -05:00
John Colvin e2816aaa33 refactor: Decouple ViewStreamDatabase from TableStream, ISubscriberCore 2025-11-15 12:16:32 -05:00
John Colvin e2a101eb20 feat: Add SpreadBook stream 2025-11-15 12:16:32 -05:00
John Colvin 45fcb2e20b refactor: Use HostBuilder to launch spark w/ streams 2025-11-15 12:16:32 -05:00
John Colvin 15883e42b6 fix (dsqa): Resolve name conflicts with new BlueOrb streams 2025-11-14 15:02:16 -05:00
John Colvin 34caf54015 fix: Fix typo in "DateTimeExpansionStream" type name
Previously, it was "DateTimeExpanstion" (erroneous "t")
2025-11-14 15:02:16 -05:00
John Colvin 05644ed6cf feat: Add DataStream WebApp
Included in DataStreamSuite.sln, but not DataStreamQueryAnalyzer.sln
2025-11-14 15:00:45 -05:00
John Colvin d6a88a7ac9 cleanup: Remove unused code 2025-11-14 14:59:33 -05:00
John Colvin f7e620db3e Reduce log noise in AsyncTCPConnection_AsyncWrapper 2025-11-14 14:59:33 -05:00
John Colvin c745407ae7 TEMP changes to appsettings.json (at least i should review them before making permanent) 2025-11-14 14:59:33 -05:00
John Colvin 1bb7d6eca2 Add ColumnIndex class to resolve column names to indices
Improves performance compared to accessing columns by name
2025-11-14 14:59:33 -05:00
John Colvin 519ef8b0af Increase SQLiteEnumerator page size to 65536
This performs better, and doesn't seem to be too large in practice
2025-11-14 14:59:33 -05:00
John Colvin 6b93c07271 Serialize TimeSpan w/ microsecond precision in DataPacket json 2025-11-14 14:59:33 -05:00
John Colvin c3288c721d Format DateTimes w/ microsecond precision in DataPacketJsonConverter 2025-11-14 14:59:33 -05:00
John Colvin 2622626eed Add BlueColumnTable for pandas integration
This will make it easier/more natural to support pandas column expressions
2025-11-14 14:59:33 -05:00
John Colvin f119f30b62 fix load IDataStreamFactory(s) with depdendency injection 2025-11-14 14:57:43 -05:00
John Colvin 4877355fee refactor: Use dependency injection in DSQA 2025-11-14 14:52:07 -05:00
John Colvin ada7d4bed1 refactor: Load IDataStreamFactory(s) with dependency injection
So we can create ViewStreams that depend on other services, config, etc.
Previously, the PrimitiveManager relied purely on static construction of
types through reflection.
2025-11-14 14:52:07 -05:00
John Colvin f65b6b82f2 refactor: Replace DataPacket w/ IDataPacket in stream handlers
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-03 16:21:32 -05:00
John Colvin 1155fd6997 refactor: Change namespace to Bts.DataStreamCore for a few types
Since they're in the DataStreamCore csproj
2025-11-03 16:19:50 -05:00
John Colvin 28545b5099 fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2025-11-03 16:19:48 -05:00
John Colvin 04c1ad7be8 Assert that local ip address is successfully obtained 2025-11-03 16:19:48 -05:00
John Colvin 2441a5267d improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2025-11-03 16:19:48 -05:00
John Colvin 12f44f63aa Assert TcpConnection endpoint matches expected 2025-11-03 16:19:48 -05:00
John Colvin c2fe578ccb Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2025-11-03 16:19:48 -05:00
John Colvin a01fc6cc15 fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2025-11-03 16:19:48 -05:00
John Colvin d80cd9c0fd build: Upgrade packages (deps) in DataStreamTests 2025-11-03 16:19:48 -05:00
John Colvin d707847079 cleanup: Remove unused ViewStreamDatabase config 2025-11-03 16:19:48 -05:00
John Colvin 118e2a9f2e Remove reference to spark protobuf Config in ViewStream
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-03 16:19:48 -05:00
John Colvin 1481a3fd07 refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-11-03 16:19:45 -05:00
John Colvin 8637bb9254 Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-03 16:13:44 -05:00
John Colvin 7c61c2ef6d cleanup: Remove unused code from ViewStreamDatabase 2025-11-03 16:13:44 -05:00
John Colvin 0668e0cd8c fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2025-11-03 16:13:44 -05:00
John Colvin 3ae88248cc cleanup: Remove unnecessary deps from csprojs 2025-11-03 16:13:43 -05:00
John Colvin 1367cea9ca fix: Catch exception when creating TcpClient 2025-11-03 16:13:43 -05:00
John Colvin b384d89820 fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-11-03 16:13:43 -05:00
John Colvin d669d24aea fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-11-03 16:13:43 -05:00
John Colvin e5bc6667e7 Reduce task wait/timeout code in AsyncTCPConnector 2025-11-03 16:13:43 -05:00
John Colvin 41512cb811 fix build warnings: Make props nullable strings in SharedStreamConfig 2025-11-03 16:13:43 -05:00
John Colvin 01d600b821 Assert that System publishers 'doWork' is always called on same thread 2025-11-03 16:13:43 -05:00
John Colvin 9e1cb14523 refactor: Model config profiles after dotnet appsettings
Namely, in DataStreamPublisher and PubSubServer. This is part of a
broader paradigm shift towards dotnet factory pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Rename Configurations/<name>.json -> appsettings.<name>.json
* Move reusable startup code into new Bts.Extensions.Hosting proj
2025-11-03 16:13:41 -05:00
John Colvin cd7c04dce5 Add ErrorChanged event to propagate DataStream exceptions 2025-11-03 16:13:19 -05:00
John Colvin fc79c451b2 refactor: Change Configuration.initialize to be async
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-11-03 16:13:19 -05:00
John Colvin 826e03af94 cleanup: Remove unused PubSubConfig props 2025-11-03 16:13:19 -05:00
John Colvin deae8c7fb0 Fix dsqa config json 2025-11-03 16:13:18 -05:00