John Colvin
d5911a7657
build: Clean up data_files cmake proj (support --target clean)
2025-12-05 15:53:32 +00:00
John Colvin
96e7c5102c
Add api server to docker compose
2025-12-05 15:53:32 +00:00
John Colvin
ea21c3e7ff
build: Resolve app data dir and shared mem prefix at runtime
...
We can't define them at compile time anymore since we build multiple exes that
depend on libspark in the same dir (e.g., api-server, pubsub, sparkd, etc.),
and each one needs its own shared memory and app data filespaces.
For shared memory:
* Prefix all shm files with the proc name and an environment variable,
BTS_SHARED_MEMORY_PREFIX (if defined). The env var is only necessary
when 2 exe's with the same name run simultaneously.
For app data:
* On linux, the default is:
/home/<user_name>/.local/share/bts/<proc_name>
* On windows, the default is:
/User/<user_name>/AppData/Roaming/bts/<proc_name>
* This path can be overridden in the environment variable:
<proc_name>__APPDATA
- For example, SPARKD__APPDATA, PUBSUB__APPDATA, etc.
2025-12-05 15:53:32 +00:00
John Colvin
26853d597a
Don't log "initializing spark config" if it has already been initialized
2025-12-05 15:53:32 +00:00
John Colvin
87d26c076b
refactor: Use dependency injection in spark api server
2025-12-05 15:53:32 +00:00
John Colvin
f98208a8a6
refactor: Add handy WebsocketExtensions
...
* IsSocketStateOpen
* IsClosedProperly
2025-12-05 15:53:32 +00:00
John Colvin
afd107402a
fix: Don't assert on exception in BooleanExpression.evaluate
...
Bugs (fixed):
* DataStream WebApp Server sometimes crashes when the user enters a
bad query.
2025-12-05 15:53:32 +00:00
John Colvin
d628f38bb1
diag: Assert that local ip address is successfully obtained
2025-12-05 15:53:32 +00:00
John Colvin
4ade38ef9a
improvement: Resolve hostnames when comparing ip addrs
...
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't
been tested yet...so we should assume it doesn't work).
2025-12-05 15:53:32 +00:00
John Colvin
0f9ea5139c
diag: Assert TcpConnection endpoint matches expected
2025-12-05 15:53:32 +00:00
John Colvin
789f74e3f6
diag: Catch exceptions on AsyncTCPConnector_AsyncWrapper bg thread
...
Since it spawns a new thread on each attempt, uncaught exceptions were
previously undetected, and caused silent failures down the pipe. This
should make it easier to track such bugs down.
2025-12-05 15:53:32 +00:00
John Colvin
75d95080b7
fix: Format PubSubRoute as json in log msgs
...
It was previously using the default ToString, which simply returns the type name
2025-12-05 15:53:32 +00:00
John Colvin
9287a57be3
build: Upgrade packages (deps) in DataStreamTests
2025-12-05 15:53:32 +00:00
John Colvin
cb20975417
cleanup: Remove unused ViewStreamDatabase config
2025-12-05 15:53:32 +00:00
John Colvin
25b13d9dff
cleanup: Remove reference to spark protobuf Config in ViewStream
...
In general, we should keep BlueOrb dependencies on spark/sparkd to a minimum.
The dotnet dependency injection architecture provides a robust config framework
we can use instead, based on json files, environment variables, and command-line
arguments.
2025-12-05 15:53:32 +00:00
John Colvin
d7b0296fbe
refactor: Run RemoteQueryManager as an IHostedService
...
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2025-12-05 15:53:32 +00:00
John Colvin
69b1ac243e
fix: Ensure spark config init and login are only performed once
...
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.
It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2025-12-05 15:53:31 +00:00
John Colvin
f3f2e5834c
diag: Assert ViewStreamDatabase.instance is only created once
...
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2025-12-05 15:53:31 +00:00
John Colvin
d93946ce61
cleanup: Remove unused code from ViewStreamDatabase
2025-12-05 15:53:31 +00:00
John Colvin
99e50f539f
fix: Define BoooleanConstant before StringConstant
...
Otherwise the former masks the latter and prevents it from matching
2025-12-05 15:53:31 +00:00
John Colvin
504d87ae14
cleanup: Remove unnecessary deps from csprojs
2025-12-05 15:53:31 +00:00
John Colvin
a8fc2a6179
fix: Catch exception when creating TcpClient
2025-12-05 15:53:31 +00:00
John Colvin
061f323026
fix: Make SystemSubscriptionMapPublisher threadsafe
...
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2025-12-05 15:53:31 +00:00
John Colvin
60488a4770
fix: Add a few null checks
...
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2025-12-05 15:53:31 +00:00
John Colvin
5f517e4a7c
improvement: Reduce task wait/timeout code in AsyncTCPConnector
2025-12-05 15:53:31 +00:00
John Colvin
a04fb84656
diag: Assert System publishers 'doWork' is always called on same thread
2025-12-05 15:53:31 +00:00
John Colvin
8bd4a5131d
deploy: Add pubsub and md-publisher services to compose.yaml
2025-12-05 15:53:31 +00:00
John Colvin
95c4b6b036
refactor: Model config profiles after dotnet appsettings
...
Namely, in DataStreamPublisher and PubSubServer. This is part of a
broader paradigm shift towards dotnet factory pattern.
https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines
* Rename Configurations/<name>.json -> appsettings.<name>.json
* Move reusable startup code into new Bts.Extensions.Hosting proj
2025-12-05 15:53:31 +00:00
John Colvin
d2300fbbb9
improvement: Propagate DataStream exceptions via ErrorChanged event
2025-11-30 16:44:31 +00:00
John Colvin
12c21f12fe
refactor: Change Configuration.initialize to be async
...
Tweak SparkCoreServices to synchronize Configuration.init globally
2025-11-30 16:44:31 +00:00
John Colvin
5061a1bbd7
cleanup: Remove unused PubSubConfig props
2025-11-30 16:44:31 +00:00
John Colvin
f6aec08919
fix: Always call 'handle_internal_pubs' in PubSubMessageReactor main loop
2025-11-30 16:44:31 +00:00
John Colvin
652f0e0043
fix: Fix a few column types in in PubSubServerTagStatsStream
...
* total_bytes (int -> long)
* last_update_time (DateTime -> long)
2025-11-30 16:44:31 +00:00
John Colvin
6889a971ff
refactor: Add DataStreamPublisher service modules for Dependency Injection
...
* Change DataStreamPublisher json fields to match cs class property names verbatim
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
2025-11-30 16:44:31 +00:00
John Colvin
c5f4d146f3
refactor: Add PubSub service modules for Dependency Injection
...
* Get rid of JsonPropertyName attributes in PubSubConfig
- The Dependency Injection framework (based around dotnet ApplicationBuilder)
does not support this attribute, and maps property names verbatim to json fields.
* Change PubSubServer json fields to match cs class property names verbatim
2025-11-30 16:44:31 +00:00
John Colvin
414ced01ee
refactor: Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl
2025-11-30 16:44:20 +00:00
John Colvin
1fb8cf6220
refactor: Decouple ManagedSubscriber from ISubscriberCore impl
2025-11-30 16:44:20 +00:00
John Colvin
79b17579c3
refactor: Only support 1 tag type in IPubSubServerTagHandler
2025-11-30 16:44:20 +00:00
John Colvin
8fe24fd022
refactor: Decouple PubTableBroker from IPublisherCore impl
...
Reference interface rather than the PublisherCore impl.
2025-11-30 16:44:20 +00:00
John Colvin
f0b89ffa23
refactor: Add Spark service modules for Dependency Injection
2025-11-30 16:44:18 +00:00
John Colvin
632f2c588f
fix: Dispose market data sub in various md streams
2025-11-21 14:55:01 -05:00
John Colvin
bbf1e87766
cleanup: Remove unused pubsub tag handler code
...
* Removed 'IPubSubServerTagHandler.register'
- All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
- This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
- No reason these interfaces have to be tied together
2025-11-21 14:55:01 -05:00
John Colvin
137be2b861
refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
...
This matches the file name
2025-11-21 14:55:01 -05:00
John Colvin
ab6db8f2da
cleanup: Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
...
In config json files (not actual code change)
2025-11-21 14:55:01 -05:00
John Colvin
366769a778
chore: Fix typos in log msgs
...
* Bad string interopolation syntax
2025-11-21 14:55:01 -05:00
John Colvin
a76cbafdbd
cleanup: Fix numerous build warnings in DataStreamSuite
...
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.
* Compression related files
- BZip2InputStream (disable nullability warnings)
- BZip2OutputStream (disable nullability warnings)
- DeflaterHuffman (disable nullability warnings)
- Inflater (disable nullability warnings)
- InflaterDynHeader (disable nullability warnings)
- ParallelJoinTest (disable nullability warnings)
- TarHeader
- TarInputStream (disable nullability warnings)
- ZipEntry
- ZipFile
- ZipInputStream
* AsyncTCPConnection_APM
* AsyncTCPConnection_APM
* InflaterHuffmanTree
* ManagedSubscriber
* ManagedTCPMessageHub
* MapStream
* PubSubClientMessage
- Normalize indentation/whitespace
- Fix some typos
- Address nullability issues to fix build warnings
- Modernize code
* PubSubMessageReactor
* PubSubServerManagedTCPConnection
* PubSubSingleTagStateEvent
* PubSubState
* PublisherCore
* PubsubDeserializationController
* RemoteMessage
- make 'message' nullable to resolve warnings elsewhere (in calling code)
* SharedStreamConfig
* SingleTagData
* SubscriberCore
* SuperMapStream
* SystemHeartbeatPublisher
* SystemStatsPublisher
* ThreadSafeQueue
* TradeBarsStream
fix build (warn): Only use Registry on windows
2025-11-21 14:54:58 -05:00
John Colvin
a442e27d3d
cleanup: Modernize PubSubRoute code, json parsing
2025-11-21 14:20:49 -05:00
John Colvin
c3da5a5d0c
cleanup: Remove unused DateTime grammar from exp parser
2025-11-21 14:20:48 -05:00
John Colvin
13fc7e0455
cleanup: Remove empty file (clearly added by accident)
2025-11-21 14:20:48 -05:00
John Colvin
0a8f002a6e
fix: Remove query from literal error stream
...
The parser can't handle colons in the expression (it throws/crashes)
2025-11-21 14:09:28 -05:00