John Colvin
bb1e7c30c8
build: Add msbuild flag to reference projs or local nugets
...
So we can use project refs for dev builds, and package refs for releases.
2026-01-04 21:19:00 +00:00
John Colvin
6568c29ad4
cleanup: Remove unnecessary ProjectReference
2026-01-04 21:19:00 +00:00
John Colvin
59b1aead62
refactor: Use HostBuilder to launch spark w/ streams
2026-01-04 21:19:00 +00:00
John Colvin
a515468e37
refactor: Rename TableUpdate props to PascalCase
...
Makes the sdk look more polished, professional
2026-01-04 21:19:00 +00:00
John Colvin
c50a006d7d
deploy: Update dotnet7 -> dotnet8 in docker images
2026-01-04 21:19:00 +00:00
John Colvin
69e4ed84c1
build: Generate version info via git cli in build scripts
...
Instead of GitVersion.MsBuild nuget
2026-01-04 21:18:58 +00:00
John Colvin
c4a6b4591d
cleanup: Remove last vestiges of spark proto config from DataStreamCore
2026-01-04 21:08:13 +00:00
John Colvin
8f3fa18bca
refactor: Rename spark_api csproj Bts.Spark.Api
...
This follows the conventional naming practices, where the project shares its
name with the namespace. Doing this in anticipation of the csharp sdk we are
about to release (for Ophir).
2026-01-04 21:08:13 +00:00
John Colvin
f60782915c
refactor: Rename spark_core csproj Bts.Spark.Core
...
This follows the conventional naming practices, where the project shares its
name with the namespace. Doing this in anticipation of the csharp sdk we are
about to release (for Ophir).
2026-01-04 21:08:13 +00:00
John Colvin
3565f78be8
cleanup (build): Remove redundant csproj refs
...
For example, projects that reference spark_api don't need an explicit reference
to spark_core, since spark_api already references it. The same goes for projects
that reference DataStream and DataStreamCore.
2026-01-04 21:08:13 +00:00
John Colvin
67f7124ed5
build: Remove DSQA from DataStreamSuite.sln
...
Because it doesn't build on linux
2026-01-04 21:08:12 +00:00
John Colvin
1c579ee063
refactor: Change namespace to Bts.DataStreamCore for a few types
...
Since they're in the DataStreamCore csproj
2026-01-04 21:08:12 +00:00
John Colvin
856cd552eb
refactor: Add Api Server service modules for Dependency Injection
...
* Part of a broader paradigm shift towards Dependency Injection and
Factory Pattern.
* Adds "api-server" service to docker compose
* Replaces MarketDataClient w/ IMarketDataClient in a few streams
* Replaces ViewStreamDatabase w/ IWorkScheduler in a few streams
2026-01-04 21:08:12 +00:00
John Colvin
687a9741db
fix: Don't assert on exception in BooleanExpression.evaluate
...
Bugs (fixed):
* DataStream WebApp Server sometimes crashes when the user enters a
bad query.
2026-01-04 21:08:12 +00:00
John Colvin
7faa0ade24
diag: Assert that local ip address is successfully obtained
2026-01-04 21:08:12 +00:00
John Colvin
4d6d68aff6
improve: Resolve hostnames when comparing ip addrs
...
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2026-01-04 21:08:12 +00:00
John Colvin
dcf291d6b2
diag: Assert TcpConnection endpoint matches expected
2026-01-04 21:08:12 +00:00
John Colvin
a6b83988e7
diag: Catch exceptions on AsyncTCPConnector_AsyncWrapper bg thread
...
Since it spawns a new thread on each attempt, uncaught exceptions were
previously undetected, and caused silent failures down the pipe. This
should make it easier to track such bugs down.
2026-01-04 21:08:12 +00:00
John Colvin
2c097500f4
fix: Format PubSubRoute as json in log msgs
...
It was previously using the default ToString, which simply returns the type name
2026-01-04 21:08:12 +00:00
John Colvin
81a212e7a3
build: Upgrade packages (deps) in DataStreamTests
2026-01-04 21:08:12 +00:00
John Colvin
5058187d5b
cleanup: Remove reference to spark protobuf Config in ViewStream
...
In general, we should keep BlueOrb dependencies on spark/sparkd to a minimum.
The dotnet dependency injection architecture provides a robust config framework
we can use instead, based on json files, environment variables, and command-line
arguments.
2026-01-04 21:08:12 +00:00
John Colvin
0253a0f60f
refactor: Run RemoteQueryManager as an IHostedService
...
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2026-01-04 21:08:12 +00:00
John Colvin
970555d2a1
diag: Assert ViewStreamDatabase.instance is only created once
...
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2026-01-04 21:08:12 +00:00
John Colvin
1d46ea061a
cleanup: Remove unused code from ViewStreamDatabase
2026-01-04 21:08:12 +00:00
John Colvin
4bd12dc465
fix: Define BoooleanConstant before StringConstant
...
Otherwise the former masks the latter and prevents it from matching
2026-01-04 21:08:12 +00:00
John Colvin
ad1d404314
cleanup: Remove unnecessary deps from csprojs
2026-01-04 21:08:12 +00:00
John Colvin
a7b6da27ae
fix: Catch exception when creating TcpClient
2026-01-04 21:08:12 +00:00
John Colvin
de4a55b34c
fix: Make SystemSubscriptionMapPublisher threadsafe
...
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2026-01-04 21:08:12 +00:00
John Colvin
2e14598667
fix: Add a few null checks
...
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2026-01-04 21:08:12 +00:00
John Colvin
3038d9154f
improve: Reduce task wait/timeout code in AsyncTCPConnector
2026-01-04 21:08:12 +00:00
John Colvin
5805758187
diag: Assert System publishers 'doWork' is always called on same thread
2026-01-04 21:08:12 +00:00
John Colvin
2c1b7b95a4
improve: Propagate DataStream exceptions via ErrorChanged event
2026-01-04 21:08:12 +00:00
John Colvin
60af47c4eb
cleanup: Remove unused PubSubConfig props
2026-01-04 21:08:12 +00:00
John Colvin
b0d0594e3d
fix: Always call 'handle_internal_pubs' in PubSubMessageReactor main loop
2026-01-04 21:08:12 +00:00
John Colvin
b29b2a0adb
fix: Fix a few column types in in PubSubServerTagStatsStream
...
* total_bytes (int -> long)
* last_update_time (DateTime -> long)
2026-01-04 21:08:11 +00:00
John Colvin
76fa35382e
refactor: Add DataStream service modules for Dependency Injection
...
Namely, in DataStreamPublisher and PubSubServer. This is part of a broader
paradigm shift towards Dependency Injection and Factory Pattern.
https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines
* Use interfaces to decouple services:
- Decouple ManagedSubscriber from ISubscriberCore impl
- Decouple PubTableBroker from IPublisherCore impl
- Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
* Model config profiles after dotnet appsettings
- Rename Configurations/<name>.json -> appsettings.<name>.json
- Make json fields match corresponding csharp props verbatim
* The Dependency Injection framework doesn't support JsonPropertyName
attributes. Config class props must match json fields verbatim.
* Move reusable startup code into new Bts.Extensions.Hosting proj
* Add pubsub and md-publisher services to compose.yaml
2026-01-04 21:08:11 +00:00
John Colvin
716bc92394
refactor: Only support 1 tag type in IPubSubServerTagHandler
2025-12-30 11:42:23 -05:00
John Colvin
3a6141f3ae
refactor: Change Configuration.initialize to be async
...
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-12-30 11:42:20 -05:00
John Colvin
7e32ac9e06
cleanup: Remove unused pubsub tag handler code
...
* Removed 'IPubSubServerTagHandler.register'
- All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
- This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
- No reason these interfaces have to be tied together
2025-12-30 11:38:21 -05:00
John Colvin
8fc4df1427
refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
...
This matches the file name
2025-12-30 11:38:21 -05:00
John Colvin
2020ebed2d
cleanup: Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
...
In config json files (not actual code change)
2025-12-30 11:38:20 -05:00
John Colvin
cc483ca4b5
diag: Fix typos in log msgs
...
* Bad string interopolation syntax
2025-12-30 11:38:20 -05:00
John Colvin
7d6e4efa4f
cleanup: Fix numerous build warnings in DataStreamSuite
...
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.
* Compression related files
- BZip2InputStream (disable nullability warnings)
- BZip2OutputStream (disable nullability warnings)
- DeflaterHuffman (disable nullability warnings)
- Inflater (disable nullability warnings)
- InflaterDynHeader (disable nullability warnings)
- ParallelJoinTest (disable nullability warnings)
- TarHeader
- TarInputStream (disable nullability warnings)
- ZipEntry
- ZipFile
- ZipInputStream
* AsyncTCPConnection_APM
* AsyncTCPConnection_APM
* InflaterHuffmanTree
* ManagedSubscriber
* ManagedTCPMessageHub
* MapStream
* PubSubClientMessage
- Normalize indentation/whitespace
- Fix some typos
- Address nullability issues to fix build warnings
- Modernize code
* PubSubMessageReactor
* PubSubServerManagedTCPConnection
* PubSubSingleTagStateEvent
* PubSubState
* PublisherCore
* PubsubDeserializationController
* RemoteMessage
- make 'message' nullable to resolve warnings elsewhere (in calling code)
* SharedStreamConfig
* SingleTagData
* SubscriberCore
* SuperMapStream
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SystemSubscriptionMapPublisher
* ThreadSafeQueue
* TradeBarsStream
Only use Registry on windows.
2025-12-30 11:38:20 -05:00
John Colvin
02f566a970
cleanup: Modernize PubSubRoute code, json parsing
2025-12-30 11:38:20 -05:00
John Colvin
d0d05a5c18
cleanup: Remove unused DateTime grammar from exp parser
2025-12-30 11:38:20 -05:00
John Colvin
3fafeca9ad
cleanup: Remove empty file (clearly added by accident)
2025-12-30 11:38:20 -05:00
John Colvin
f5a4e7c410
fix: Remove query from literal error stream
...
The parser can't handle colons in the expression (it throws/crashes)
2025-12-30 11:38:17 -05:00
John Colvin
a0b2a8b186
chore: Add async SQLiteEnumerator to read table as IBlueRowCollection
2025-12-30 11:36:35 -05:00
arham
e69012a5ad
fix: sanitized dsl in websocket
2025-12-03 13:35:08 +00:00
arham
4237d58f76
Add: metadata to dsl callback
2025-12-03 13:34:37 +00:00