John Colvin
c4f12be3e4
cleanup: Fix a ton of build warnings (cs)
...
All these warnings have been bothering me for a while. The work isn't done, but
this commit chops them down from ~1000 to ~600.
* AppendCollection.cs
* AppendStreamModified.cs
* BenzingaNewsStream.cs
* BenzingaNewsSymbolStateStream.cs
* BookStream.cs
* CSVSaverStream.cs
* ChartsUtils.cs
* ColumnInfo.cs
* ContractUtils.cs
* ContractUtils.cs
* DiffStream.cs
* FuturesActiveContractStream.cs
* FuzzyJoinOverTimeStream.cs
* HistoricalBarsStream.cs
* HistoricalBarsStream.cs
* InstrumentsStream.cs
* JoinOperandRowIndex.cs
* JoinRow.cs
* JoinStream.cs
* JoinTable.cs
* JoinTableSchema.cs
* LatencyStream.cs
* LeftInternalJoinStream.cs
* LightForeignJoinRow.cs
* LightForeignJoinRowList.cs
* LightLeftForeignJoinRow.cs
* LiteralStream.cs
* NLStream.cs
* NLToDSLStream.cs
* NLToIndexedFewShotPromptStream.cs
* NonIncGroupByStream.cs
* NonIncJoinBookkeeper.cs
* NonIncJoinSpec.cs
* OHLCVStream.cs
* OpenAIPromptStream.cs
* PriceStatsStream.cs
* PriceStatsStream.cs
* ProjectHelper.cs
* ProjectStream.cs
* PubSubServerStateStream.cs
* PubsubHeartbeatStream.cs
* SimpleNode.cs
* SortRow.cs
* SpatialArbStatsStream.cs
* SpatialFuturePairsStream.cs
* SpreadPerformanceEvaluator.cs
* SpreadPerformanceStream.cs
* StreamIOCommon.cs
* SuperJoinStream.cs
* TableMetaDataStream.cs
* TextWriterStream.cs
* TimeAverageStream.cs
2025-11-15 12:16:33 -05:00
John Colvin
54d7279832
style: Normalize indentation in PubsubHeartbeatStream.cs
2025-11-15 12:16:33 -05:00
John Colvin
03df578a81
chore: Add GetValue, TryGetValue ext fns to IBlueRow
2025-11-15 12:16:32 -05:00
John Colvin
7a50ad2cc5
chore: Add 'Empty' fn for BlueTable/Schema (for convenience)
2025-11-15 12:16:32 -05:00
John Colvin
0cc4b83417
cleanup: Remove redundant code block in HistoricalBarsStream.cs
2025-11-15 12:16:32 -05:00
John Colvin
4f1efea066
cleanup: Fix indentation and build warnings in TestStream.cs
2025-11-15 12:16:32 -05:00
John Colvin
e2816aaa33
refactor: Decouple ViewStreamDatabase from TableStream, ISubscriberCore
2025-11-15 12:16:32 -05:00
John Colvin
e2a101eb20
feat: Add SpreadBook stream
2025-11-15 12:16:32 -05:00
John Colvin
45fcb2e20b
refactor: Use HostBuilder to launch spark w/ streams
2025-11-15 12:16:32 -05:00
John Colvin
15883e42b6
fix (dsqa): Resolve name conflicts with new BlueOrb streams
2025-11-14 15:02:16 -05:00
John Colvin
34caf54015
fix: Fix typo in "DateTimeExpansionStream" type name
...
Previously, it was "DateTimeExpanstion" (erroneous "t")
2025-11-14 15:02:16 -05:00
John Colvin
05644ed6cf
feat: Add DataStream WebApp
...
Included in DataStreamSuite.sln, but not DataStreamQueryAnalyzer.sln
2025-11-14 15:00:45 -05:00
John Colvin
d6a88a7ac9
cleanup: Remove unused code
2025-11-14 14:59:33 -05:00
John Colvin
f7e620db3e
Reduce log noise in AsyncTCPConnection_AsyncWrapper
2025-11-14 14:59:33 -05:00
John Colvin
c745407ae7
TEMP changes to appsettings.json (at least i should review them before making permanent)
2025-11-14 14:59:33 -05:00
John Colvin
1bb7d6eca2
Add ColumnIndex class to resolve column names to indices
...
Improves performance compared to accessing columns by name
2025-11-14 14:59:33 -05:00
John Colvin
519ef8b0af
Increase SQLiteEnumerator page size to 65536
...
This performs better, and doesn't seem to be too large in practice
2025-11-14 14:59:33 -05:00
John Colvin
6b93c07271
Serialize TimeSpan w/ microsecond precision in DataPacket json
2025-11-14 14:59:33 -05:00
John Colvin
c3288c721d
Format DateTimes w/ microsecond precision in DataPacketJsonConverter
2025-11-14 14:59:33 -05:00
John Colvin
2622626eed
Add BlueColumnTable for pandas integration
...
This will make it easier/more natural to support pandas column expressions
2025-11-14 14:59:33 -05:00
John Colvin
f119f30b62
fix load IDataStreamFactory(s) with depdendency injection
2025-11-14 14:57:43 -05:00
John Colvin
4877355fee
refactor: Use dependency injection in DSQA
2025-11-14 14:52:07 -05:00
John Colvin
ada7d4bed1
refactor: Load IDataStreamFactory(s) with dependency injection
...
So we can create ViewStreams that depend on other services, config, etc.
Previously, the PrimitiveManager relied purely on static construction of
types through reflection.
2025-11-14 14:52:07 -05:00
John Colvin
f65b6b82f2
refactor: Replace DataPacket w/ IDataPacket in stream handlers
...
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-03 16:21:32 -05:00
John Colvin
1155fd6997
refactor: Change namespace to Bts.DataStreamCore for a few types
...
Since they're in the DataStreamCore csproj
2025-11-03 16:19:50 -05:00
John Colvin
28545b5099
fix: Don't assert on exception in BooleanExpression.evaluate
...
Bugs (fixed):
* DataStream WebApp Server sometimes crashes when the user enters a
bad query.
2025-11-03 16:19:48 -05:00
John Colvin
04c1ad7be8
Assert that local ip address is successfully obtained
2025-11-03 16:19:48 -05:00
John Colvin
2441a5267d
improvement: Resolve hostnames when comparing ip addrs
...
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2025-11-03 16:19:48 -05:00
John Colvin
12f44f63aa
Assert TcpConnection endpoint matches expected
2025-11-03 16:19:48 -05:00
John Colvin
c2fe578ccb
Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
...
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2025-11-03 16:19:48 -05:00
John Colvin
a01fc6cc15
fix: Format PubSubRoute as json in log msgs
...
It was previously using the default ToString, which simply returns the type name
2025-11-03 16:19:48 -05:00
John Colvin
d80cd9c0fd
build: Upgrade packages (deps) in DataStreamTests
2025-11-03 16:19:48 -05:00
John Colvin
d707847079
cleanup: Remove unused ViewStreamDatabase config
2025-11-03 16:19:48 -05:00
John Colvin
118e2a9f2e
Remove reference to spark protobuf Config in ViewStream
...
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2025-11-03 16:19:48 -05:00
John Colvin
1481a3fd07
refactor: Run RemoteQueryManager as an IHostedService
...
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-11-03 16:19:45 -05:00
John Colvin
8637bb9254
Assert ViewStreamDatabase.instance is only created once
...
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-11-03 16:13:44 -05:00
John Colvin
7c61c2ef6d
cleanup: Remove unused code from ViewStreamDatabase
2025-11-03 16:13:44 -05:00
John Colvin
0668e0cd8c
fix: Define BoooleanConstant before StringConstant
...
Otherwise the former masks the latter and prevents it from matching
2025-11-03 16:13:44 -05:00
John Colvin
3ae88248cc
cleanup: Remove unnecessary deps from csprojs
2025-11-03 16:13:43 -05:00
John Colvin
1367cea9ca
fix: Catch exception when creating TcpClient
2025-11-03 16:13:43 -05:00
John Colvin
b384d89820
fix: Make SystemSubscriptionMapPublisher threadsafe
...
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-11-03 16:13:43 -05:00
John Colvin
d669d24aea
fix: Add a few null checks
...
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-11-03 16:13:43 -05:00
John Colvin
e5bc6667e7
Reduce task wait/timeout code in AsyncTCPConnector
2025-11-03 16:13:43 -05:00
John Colvin
41512cb811
fix build warnings: Make props nullable strings in SharedStreamConfig
2025-11-03 16:13:43 -05:00
John Colvin
01d600b821
Assert that System publishers 'doWork' is always called on same thread
2025-11-03 16:13:43 -05:00
John Colvin
9e1cb14523
refactor: Model config profiles after dotnet appsettings
...
Namely, in DataStreamPublisher and PubSubServer. This is part of a
broader paradigm shift towards dotnet factory pattern.
https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines
* Rename Configurations/<name>.json -> appsettings.<name>.json
* Move reusable startup code into new Bts.Extensions.Hosting proj
2025-11-03 16:13:41 -05:00
John Colvin
cd7c04dce5
Add ErrorChanged event to propagate DataStream exceptions
2025-11-03 16:13:19 -05:00
John Colvin
fc79c451b2
refactor: Change Configuration.initialize to be async
...
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-11-03 16:13:19 -05:00
John Colvin
826e03af94
cleanup: Remove unused PubSubConfig props
2025-11-03 16:13:19 -05:00
John Colvin
deae8c7fb0
Fix dsqa config json
2025-11-03 16:13:18 -05:00