Commit Graph
484 Commits
Author SHA1 Message Date
John Colvin b061509126 build: Compile spark-windows with clang
Made a couple minor changes to cpp to address new werrors
2026-03-20 17:02:44 +00:00
John Colvin 74475cb68f deploy: Resolve profile dirs with regex patterns 2026-03-20 17:02:44 +00:00
John Colvin f181d34cbe build: Check if venv already active before activating 2026-03-20 17:02:44 +00:00
John Colvin 7de98a313f refactor: Break up Bts.Spark.Core into a few diff projs
This is in anticipation of decoupling BlueOrb from Spark.

* Bts.Collections
* Bts.Core
* Bts.Disposables
* Bts.Numerics
2026-03-08 16:00:49 +00:00
John Colvin 21052090a2 fix (pre-emptive): Format dotnet flag opts correctly
In pack_spark_sdk.py, the dotnet pack cmd is constructed with a dict of
opts. If the value part is None, the opt should be formatted as a flag,
"--key" instead of "--key None".
2026-03-07 13:32:03 -05:00
John Colvin 9bd32b8225 fix build (pack cs sdk): Fix --(no-)cs toggle behavior
* Always run "dotnet pack", regardless of csharp toggle opt
* Run "dotnet publish" if csharp toggled on
2026-03-07 13:20:56 -05:00
John Colvin 5bc2e29096 fix build: Only check "dsqa" flag when building on windows
The --dsqa opt isn't defined when running on linux.
2026-02-21 17:55:12 +00:00
John Colvin a605e57bee build: Add dsqa toggle to build_data_stream.py
Windows only
2026-02-20 02:33:38 -05:00
John Colvin 57b59215e6 fix build: Don't echo parenthesis in .cmd if block
It's not allowed (because windows).
2026-02-18 02:54:25 -05:00
John Colvin 3842efcef1 build: Define build py requirements in separate files
No need to install backend build deps when building the front-end, and
vice versa. The build scripts (build_spark.cmd, build_sparkd.sh, etc.)
automatically install only the necessary requirements now.

Note: We'll keep the monolith requirements.txt file around (at least for
now), as it includes more than just build deps (e.g., numpy, pandas, and
jupyter for data analysis; pybit for bybit plugins, etc.).
2026-02-03 16:57:32 +00:00
John Colvin 635107eb68 cleanup: Remove unused py code 2026-02-03 05:20:29 +00:00
John Colvin 01c61bb16d fix build: Publish spark_api_server csproj instead of the sln
There's a version conflict with System.Data.Common when publishing the sln.
2026-01-22 05:50:49 +00:00
John Colvin 2eb80082d8 refactor: Move Bts.Extensions.Hosting, Bts.Net to cs dir
They aren't exclusively used by DataStream csprojs
2026-01-21 14:20:39 -05:00
John Colvin d55ef854ba fix build: Don't actually copy files when building with --dry-run 2026-01-21 14:03:01 -05:00
John Colvin b1417ece5d fix build: Fix cmake windows preset in publish_api_server.py
Changed from 'spark' -> 'spark-windows'
2026-01-21 14:03:01 -05:00
John Colvin f6ae219a72 build: Add a few toggle opts to publish py scripts
--(no-)cpp           Enable/disable cpp build
--(no-)cs            Enable/disable csharp build
--(no-)csproj-refs   Depend on csproj refs, or locally built nugets
2026-01-21 14:03:01 -05:00
arham f19d2d695b Fix: Updated path so that profiles work 2026-01-16 06:55:09 +00:00
John Colvin f88915a940 fix build: Add --spark-version opt in build_data_stream.py 2026-01-07 02:34:56 -05:00
John Colvin 549529aa60 tool: Add script to add BTS github nuget source
Where we host the BTS sdk nugets
2026-01-04 21:19:00 +00:00
John Colvin 8c458d48bb feat: Add server side csharp sdk
For Ophir
2026-01-04 21:19:00 +00:00
John Colvin bb1e7c30c8 build: Add msbuild flag to reference projs or local nugets
So we can use project refs for dev builds, and package refs for releases.
2026-01-04 21:19:00 +00:00
John Colvin 1f68f47b42 build: Propagate py exit codes back to the shell
* publish_spark.cmd
2026-01-04 21:19:00 +00:00
John Colvin 4bc1a4c833 build (qol): Add opts to only build cpp or cs
Usually, we're only working in one language or the other, and don't need
to rebuild both.
2026-01-04 21:19:00 +00:00
John Colvin 69e4ed84c1 build: Generate version info via git cli in build scripts
Instead of GitVersion.MsBuild nuget
2026-01-04 21:18:58 +00:00
John Colvin 4d5d0f8573 build: Consolidate cmake presets 2026-01-04 21:08:13 +00:00
John Colvin 76fa35382e refactor: Add DataStream service modules for Dependency Injection
Namely, in DataStreamPublisher and PubSubServer. This is part of a broader
paradigm shift towards Dependency Injection and Factory Pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Use interfaces to decouple services:
  - Decouple ManagedSubscriber from ISubscriberCore impl
  - Decouple PubTableBroker from IPublisherCore impl
  - Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl

* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
* Model config profiles after dotnet appsettings
  - Rename Configurations/<name>.json -> appsettings.<name>.json
  - Make json fields match corresponding csharp props verbatim
    * The Dependency Injection framework doesn't support JsonPropertyName
      attributes. Config class props must match json fields verbatim.

* Move reusable startup code into new Bts.Extensions.Hosting proj
* Add pubsub and md-publisher services to compose.yaml
2026-01-04 21:08:11 +00:00
John Colvin 8d6e94384c cleanup: Fix benign typos in build scripts
Change BUILD_SPARK_PY var name to BUILD_PY (these scripts build other
projects than 'spark').
2025-12-30 11:38:20 -05:00
John Colvin 0b3fa813f1 tool: Add script to dump structured logs to console 2025-07-17 16:25:23 +00:00
John Colvin 83f1b5b567 dev qol: In envsource.fish, split PATH vars into list
Assume they are semi-colon delimited.
2025-06-24 10:48:42 -04:00
John Colvin e6466f8309 chore: Add CANCEL_ORDER op to binance.env 2025-06-16 11:08:56 -04:00
John Colvin 7dea9b3792 fix: Don't raise ex if var isn't defined in req path 2025-06-16 11:08:56 -04:00
John Colvin 4681f22e71 deploy: Move configurations to new stutter dir "app_sparkd" 2025-06-16 11:08:54 -04:00
John Colvin 5163ca4cae refactor: Organize libspark proj w/ stuttering include
This new structure provides several benefits:

* Allows us to install multiple CMake projects in the same destination
  dir (without include path conflicts).
  - This makes shipping the trading algo sdk a trivial excercise. Simply
    copy the lib and include dirs from the install dest. That's it!
* Puts us in a position to break up the monolithic libspark into smaller
  modules, which are easier to manage.

Windows specific notes:

* Get rid of unnecessary script vs_filters.cmake
* Define WebsocketConnectionGateway directly in cs (instead of proto)
  - The proto would no longer be in the include path passed to protoc.
    Moving it to c# because there wasn't a compelling reason for it to
    be a proto in the first place.
* Set spark output prefix to "lib" on windows
  - So we get "libspark.dll" instead of "spark.dll", which conflicts
    with the csharp lib, "Spark.dll".
* Copy libspark.dll to net8.0 & net8.0-windows in a single cmake build
  target

fix build: Copy libspark to both dirs net8.0, net8.0-windows
2025-06-16 11:08:53 -04:00
John Colvin 5ad7755b28 tool: Make open orders symbol param optional in binance.env 2025-05-20 20:08:34 +00:00
John Colvin df4ca124c3 cleanup: Remove redundant script get_binance_md.sh
I should have removed this when I added binance_rest_client.sh
2025-05-20 20:08:34 +00:00
John Colvin 2c8b9ee288 tool: Add envsource.fish to load .env files in fish 2025-05-20 20:08:34 +00:00
John Colvin f02f28e347 tool: Add a few vars to binance.env 2025-05-20 20:08:34 +00:00
John Colvin c8a89c1e8d cleanup: Rename get_binance_md.py -> binance_client.py
This is a more appropriate name (it's not just md anymore).
2025-05-20 20:08:34 +00:00
John Colvin 7ae9e07a30 tool: Use binance.env file in python client
Like the bash impl (binance_rest_client.sh), it gets urls, params
(templates), and api credentials from env vars.
2025-05-20 20:08:34 +00:00
John Colvin af9eb670c1 build: Update build scripts to target dotnet 8 2025-04-30 12:06:28 -04:00
John Colvin 61f04c4605 fix: Implement state machine for binance cxl/replace 2025-04-28 17:12:59 +00:00
John Colvin 043efab5a2 build: Upgrade c# to net8.0 2025-04-28 17:12:58 +00:00
John Colvin 54cb5a2ab2 tool: Add script to configure aws sso 2025-04-28 17:12:58 +00:00
John Colvin 10436f0f3b tool: Rename binance_orders_client.sh -> binance_rest_client.sh
It supports md and order/trade requests
2025-04-28 17:12:58 +00:00
John Colvin eeffb735ae deploy (fix): Override existing values when loading env files 2025-04-16 13:48:24 +00:00
John Colvin b602997f3a deploy: Build sparkd if any service starts with "sparkd"
Previously, we only built sparkd for the "sparkd" service (i.e., exact match)
2025-04-16 13:48:24 +00:00
arham 14e3eee0bc Add: config changes to make server controller work and set up hk zen server 2025-04-09 09:42:47 -04:00
John Colvin 716c4dd674 tool: Add script to dump sparkd logs to sqlite db 2025-03-19 14:41:23 +00:00
John Colvin 10b2d38de8 tool: Add cmd to tail sparkd log file using docker compose 2025-03-17 00:51:54 +00:00
John Colvin 3f58c2ce84 Add script to install aws cli
The aws cli can be used to configure credentials in the shell in order
to retrieve secrets (e.g., production api keys for alphaworx, etc.).
2025-03-10 05:24:49 +00:00