Commit Graph
80 Commits
Author SHA1 Message Date
John Colvin bb1e7c30c8 build: Add msbuild flag to reference projs or local nugets
So we can use project refs for dev builds, and package refs for releases.
2026-01-04 21:19:00 +00:00
John Colvin a515468e37 refactor: Rename TableUpdate props to PascalCase
Makes the sdk look more polished, professional
2026-01-04 21:19:00 +00:00
John Colvin c4a6b4591d cleanup: Remove last vestiges of spark proto config from DataStreamCore 2026-01-04 21:08:13 +00:00
John Colvin 3565f78be8 cleanup (build): Remove redundant csproj refs
For example, projects that reference spark_api don't need an explicit reference
to spark_core, since spark_api already references it. The same goes for projects
that reference DataStream and DataStreamCore.
2026-01-04 21:08:13 +00:00
John Colvin 1c579ee063 refactor: Change namespace to Bts.DataStreamCore for a few types
Since they're in the DataStreamCore csproj
2026-01-04 21:08:12 +00:00
John Colvin 856cd552eb refactor: Add Api Server service modules for Dependency Injection
* Part of a broader paradigm shift towards Dependency Injection and
  Factory Pattern.
* Adds "api-server" service to docker compose
* Replaces MarketDataClient w/ IMarketDataClient in a few streams
* Replaces ViewStreamDatabase w/ IWorkScheduler in a few streams
2026-01-04 21:08:12 +00:00
John Colvin 5058187d5b cleanup: Remove reference to spark protobuf Config in ViewStream
In general, we should keep BlueOrb dependencies on spark/sparkd to a minimum.
The dotnet dependency injection architecture provides a robust config framework
we can use instead, based on json files, environment variables, and command-line
arguments.
2026-01-04 21:08:12 +00:00
John Colvin 970555d2a1 diag: Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2026-01-04 21:08:12 +00:00
John Colvin 1d46ea061a cleanup: Remove unused code from ViewStreamDatabase 2026-01-04 21:08:12 +00:00
John Colvin 2e14598667 fix: Add a few null checks
* TimeBasedFilter
* TableAccumulator2
* NonIncGroupByStream
2026-01-04 21:08:12 +00:00
John Colvin 2c1b7b95a4 improve: Propagate DataStream exceptions via ErrorChanged event 2026-01-04 21:08:12 +00:00
John Colvin 76fa35382e refactor: Add DataStream service modules for Dependency Injection
Namely, in DataStreamPublisher and PubSubServer. This is part of a broader
paradigm shift towards Dependency Injection and Factory Pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Use interfaces to decouple services:
  - Decouple ManagedSubscriber from ISubscriberCore impl
  - Decouple PubTableBroker from IPublisherCore impl
  - Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl

* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
* Model config profiles after dotnet appsettings
  - Rename Configurations/<name>.json -> appsettings.<name>.json
  - Make json fields match corresponding csharp props verbatim
    * The Dependency Injection framework doesn't support JsonPropertyName
      attributes. Config class props must match json fields verbatim.

* Move reusable startup code into new Bts.Extensions.Hosting proj
* Add pubsub and md-publisher services to compose.yaml
2026-01-04 21:08:11 +00:00
John Colvin 716bc92394 refactor: Only support 1 tag type in IPubSubServerTagHandler 2025-12-30 11:42:23 -05:00
John Colvin 7e32ac9e06 cleanup: Remove unused pubsub tag handler code
* Removed 'IPubSubServerTagHandler.register'
  - All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
  - This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
  - No reason these interfaces have to be tied together
2025-12-30 11:38:21 -05:00
arham 4237d58f76 Add: metadata to dsl callback 2025-12-03 13:34:37 +00:00
John Colvin 10bb90c731 fix: Use IDictionary.GetEnumerator explicitly to iterate keyvals
We really need to replace these old non-generic collections with generic ones.
BlueRow's IDictionaryEnumerator implementation iterates over values when passed
to a foreach loop, not key value pairs.
2025-11-19 23:55:35 -05:00
John Colvin 2014931653 feat: Add live bbo stream 2025-11-19 10:33:31 -05:00
John Colvin aa740e786c refactor: Automatically add id column in BlueTableSchemaBuilder 2025-11-19 00:36:56 -05:00
John Colvin 5080960b97 refactor: Replace DataPacket w/ IDataPacket in stream handlers
This commit also renames BlueTableSchemaAdaptor to BlueTableSchemaAdapter,
and makes it implement IBlueTableSchema.
2025-11-19 00:36:56 -05:00
John Colvin 770e14752a chore: Add json facilities for DataStream websocket api 2025-11-19 00:36:56 -05:00
John Colvin e24b877e58 Add gen purpose blue row/table utils for data streams
* Add BlueRowReflection<T> so ViewStream implementations can use
  strong row types without needing to implement IBlueRow
* Add default implementations for IDictionary members in IBlueRow to make
  implementing IBlueRow less tedious
2025-11-19 00:31:53 -05:00
Taha Afzal 09abe4a432 Added RowBasedAppendStream 2025-11-18 03:10:30 +00:00
Taha Afzal 66c0baa150 Adding meta data to DataPacket, NLStream adds nl -> dsl as a name value pair in the meta data. Also removing windows ref from DataStreamStudio.sln 2025-09-24 13:16:28 +00:00
John Colvin 7e79a21cd4 Add some nifty DataPacket/TableUpdate IObservable utils 2025-07-15 10:13:53 -04:00
Taha 0858eaac23 Polishing BlueOrb streams 2025-07-13 21:21:10 -04:00
John Colvin 5163ca4cae refactor: Organize libspark proj w/ stuttering include
This new structure provides several benefits:

* Allows us to install multiple CMake projects in the same destination
  dir (without include path conflicts).
  - This makes shipping the trading algo sdk a trivial excercise. Simply
    copy the lib and include dirs from the install dest. That's it!
* Puts us in a position to break up the monolithic libspark into smaller
  modules, which are easier to manage.

Windows specific notes:

* Get rid of unnecessary script vs_filters.cmake
* Define WebsocketConnectionGateway directly in cs (instead of proto)
  - The proto would no longer be in the include path passed to protoc.
    Moving it to c# because there wasn't a compelling reason for it to
    be a proto in the first place.
* Set spark output prefix to "lib" on windows
  - So we get "libspark.dll" instead of "spark.dll", which conflicts
    with the csharp lib, "Spark.dll".
* Copy libspark.dll to net8.0 & net8.0-windows in a single cmake build
  target

fix build: Copy libspark to both dirs net8.0, net8.0-windows
2025-06-16 11:08:53 -04:00
Taha 70fae06a30 BlueOrb streams and MLP 2025-06-08 21:52:38 -04:00
John Colvin 043efab5a2 build: Upgrade c# to net8.0 2025-04-28 17:12:58 +00:00
Taha aab8e83543 Some rewrite of streams that fetch bars, and making the other business streams thin wrapper over stream primitives 2024-11-26 22:33:26 -05:00
Taha 6163887cd3 Bunch of CME queries are now working 2024-10-30 13:36:23 -04:00
Taha f9aa9dd982 Merge branch 'development' of bitbucket.org:bluetradesys/spark into development 2024-10-15 23:48:34 -04:00
Taha 22f1ab44f9 Basic set of streams that can talks ot the LLM and get DSL back from few-shot prompt 2024-10-15 23:38:49 -04:00
Taha 288b7712db Added and modified more streams to address teh CME NL queries 2024-10-09 13:59:01 -04:00
Taha 7e219aec7c Merging 2024-07-17 02:06:17 -04:00
Taha f06beff22f Added new benzinga streams 2024-07-17 02:02:01 -04:00
Taha Afzal 233e2576af Merging 2024-07-05 21:21:27 +00:00
Taha 050e21e22b Adding TableMetaDataStream 2024-07-04 23:39:04 -04:00
Taha Afzal 672ad6d237 Cleaning Config 1.0 (not completely done) 2024-06-30 23:17:20 -04:00
Taha 393f304be4 some buf gixes + cleanup 2024-06-26 00:23:19 -04:00
Taha 87b6a2044c PubSubServer's linux issues fixed 2024-06-23 21:13:37 -04:00
John Colvin cfc60766d8 Change line endings to crlf in a few cs files 2024-06-06 09:21:00 -05:00
John Colvin b03fec7f79 Change line endings to crlf in a few cs files 2024-06-06 09:20:59 -05:00
John Colvin e44601d1b1 fix merge: Bring back few fns that SchemaGuesserStream needs
* BlueDataUtility.get_typed_schema
* BlueDataUtility.get_typed_table
2024-06-06 09:20:05 -05:00
Taha fac0a9c25b RemoteQueryStream 2024-06-06 09:20:05 -05:00
John Colvin 65f412ec9f fix: Fix myLoadAllTypes impl
* Fix lowercase typo in hardcoded assemblies (case matters now that we
  run on Linux)
* Fix resolution to full path (previous impl used deprecated functions,
  and incorrectly appended relative paths)
2024-06-06 09:15:15 -05:00
John Colvin 5e7e7e8ef4 test: Add FilterStreamTests 2024-06-05 15:44:35 +00:00
John Colvin 015d846941 feat: Add Pandas(Script)Stream 2024-06-05 15:44:35 +00:00
John Colvin 1824d5f9a4 feat: Add [incremental] MongoCollectionStream 2024-06-03 13:46:51 +00:00
John Colvin d3eb29087b refactor: Impl IDictionary<int, IBlueRow> in BlueTable 2024-06-03 13:46:51 +00:00
John Colvin 5741ec9ab2 cleanup: Remove unnecessary usings (namespaces) 2024-06-03 13:46:51 +00:00