Commit Graph
95 Commits
Author SHA1 Message Date
arham 94bfdacf8b add: trade latency logs 2026-07-02 16:21:08 +00:00
John Colvin c0448c7c8d deploy: Resolve profile dirs with regex patterns 2026-03-26 16:20:41 -04:00
John Colvin 21052090a2 fix (pre-emptive): Format dotnet flag opts correctly
In pack_spark_sdk.py, the dotnet pack cmd is constructed with a dict of
opts. If the value part is None, the opt should be formatted as a flag,
"--key" instead of "--key None".
2026-03-07 13:32:03 -05:00
John Colvin 9bd32b8225 fix build (pack cs sdk): Fix --(no-)cs toggle behavior
* Always run "dotnet pack", regardless of csharp toggle opt
* Run "dotnet publish" if csharp toggled on
2026-03-07 13:20:56 -05:00
John Colvin 5bc2e29096 fix build: Only check "dsqa" flag when building on windows
The --dsqa opt isn't defined when running on linux.
2026-02-21 17:55:12 +00:00
John Colvin a605e57bee build: Add dsqa toggle to build_data_stream.py
Windows only
2026-02-20 02:33:38 -05:00
John Colvin 3842efcef1 build: Define build py requirements in separate files
No need to install backend build deps when building the front-end, and
vice versa. The build scripts (build_spark.cmd, build_sparkd.sh, etc.)
automatically install only the necessary requirements now.

Note: We'll keep the monolith requirements.txt file around (at least for
now), as it includes more than just build deps (e.g., numpy, pandas, and
jupyter for data analysis; pybit for bybit plugins, etc.).
2026-02-03 16:57:32 +00:00
John Colvin 635107eb68 cleanup: Remove unused py code 2026-02-03 05:20:29 +00:00
John Colvin 01c61bb16d fix build: Publish spark_api_server csproj instead of the sln
There's a version conflict with System.Data.Common when publishing the sln.
2026-01-22 05:50:49 +00:00
John Colvin 2eb80082d8 refactor: Move Bts.Extensions.Hosting, Bts.Net to cs dir
They aren't exclusively used by DataStream csprojs
2026-01-21 14:20:39 -05:00
John Colvin d55ef854ba fix build: Don't actually copy files when building with --dry-run 2026-01-21 14:03:01 -05:00
John Colvin b1417ece5d fix build: Fix cmake windows preset in publish_api_server.py
Changed from 'spark' -> 'spark-windows'
2026-01-21 14:03:01 -05:00
John Colvin f6ae219a72 build: Add a few toggle opts to publish py scripts
--(no-)cpp           Enable/disable cpp build
--(no-)cs            Enable/disable csharp build
--(no-)csproj-refs   Depend on csproj refs, or locally built nugets
2026-01-21 14:03:01 -05:00
John Colvin f88915a940 fix build: Add --spark-version opt in build_data_stream.py 2026-01-07 02:34:56 -05:00
John Colvin 8c458d48bb feat: Add server side csharp sdk
For Ophir
2026-01-04 21:19:00 +00:00
John Colvin bb1e7c30c8 build: Add msbuild flag to reference projs or local nugets
So we can use project refs for dev builds, and package refs for releases.
2026-01-04 21:19:00 +00:00
John Colvin 4bc1a4c833 build (qol): Add opts to only build cpp or cs
Usually, we're only working in one language or the other, and don't need
to rebuild both.
2026-01-04 21:19:00 +00:00
John Colvin 69e4ed84c1 build: Generate version info via git cli in build scripts
Instead of GitVersion.MsBuild nuget
2026-01-04 21:18:58 +00:00
John Colvin 4d5d0f8573 build: Consolidate cmake presets 2026-01-04 21:08:13 +00:00
John Colvin 76fa35382e refactor: Add DataStream service modules for Dependency Injection
Namely, in DataStreamPublisher and PubSubServer. This is part of a broader
paradigm shift towards Dependency Injection and Factory Pattern.

https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines

* Use interfaces to decouple services:
  - Decouple ManagedSubscriber from ISubscriberCore impl
  - Decouple PubTableBroker from IPublisherCore impl
  - Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl

* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
* Model config profiles after dotnet appsettings
  - Rename Configurations/<name>.json -> appsettings.<name>.json
  - Make json fields match corresponding csharp props verbatim
    * The Dependency Injection framework doesn't support JsonPropertyName
      attributes. Config class props must match json fields verbatim.

* Move reusable startup code into new Bts.Extensions.Hosting proj
* Add pubsub and md-publisher services to compose.yaml
2026-01-04 21:08:11 +00:00
John Colvin 4681f22e71 deploy: Move configurations to new stutter dir "app_sparkd" 2025-06-16 11:08:54 -04:00
John Colvin af9eb670c1 build: Update build scripts to target dotnet 8 2025-04-30 12:06:28 -04:00
John Colvin 61f04c4605 fix: Implement state machine for binance cxl/replace 2025-04-28 17:12:59 +00:00
John Colvin eeffb735ae deploy (fix): Override existing values when loading env files 2025-04-16 13:48:24 +00:00
John Colvin b602997f3a deploy: Build sparkd if any service starts with "sparkd"
Previously, we only built sparkd for the "sparkd" service (i.e., exact match)
2025-04-16 13:48:24 +00:00
arham 14e3eee0bc Add: config changes to make server controller work and set up hk zen server 2025-04-09 09:42:47 -04:00
John Colvin 716c4dd674 tool: Add script to dump sparkd logs to sqlite db 2025-03-19 14:41:23 +00:00
John Colvin b555850015 build: Package & deploy sparkd with docker compose 2025-03-10 03:05:59 +00:00
John Colvin c8fc649b6b fix build: Pass config arg to cmake --build
For Ninja Multi-Config builds (i.e., api server and data streams)
2025-01-28 19:59:09 +00:00
John Colvin b1f4654b3c Fix benign typo in publish_spark.py 2025-01-21 17:28:50 -05:00
John Colvin 9d350334c0 fix build (installer): Fix install path typo in publish_spark.py 2025-01-21 17:28:05 -05:00
John Colvin 86eba20d61 build: Make compiler/build type case insensitive 2025-01-05 06:25:11 +00:00
John Colvin 22ae51fb99 build: Clean up build py scripts 2025-01-03 16:59:07 +00:00
John Colvin 838dbcd886 build: In py build scripts, exit with code 1 on failure
Previously, they would exit with 0 no matter what happened.
2025-01-03 16:59:07 +00:00
John Colvin 4d41704dd2 build: Configure admin webapp build entirely in CMakeLists
Get rid of build_admin_web_app.py
2025-01-03 16:59:07 +00:00
John Colvin 8cde8ee6e1 build: Add script to publish the api server
This will build the app as a self-contained executable here:
build/Release/bin/net7.0/linux-x64/publish
2024-11-28 16:35:21 +00:00
John Colvin 7b5a6031af build: Use consistent naming conventions in CMake presets
We'll use similar naming conventions for our (vscode) tasks
2024-06-06 09:15:17 -05:00
John Colvin 54e96a06cb build: Build DSQA on Windows in build_data_stream.py
It was only building DataStreamSuite.sln before, which is crossplatform
now so it no longer includes DSQA.
2024-06-06 09:15:16 -05:00
John Colvin 6049826e90 build: Make build_data_stream.py cross platform
Previously it only worked on Windows
2024-06-06 09:15:16 -05:00
John Colvin f1744450ca build: Add script to build api server 2024-06-06 09:15:15 -05:00
John Colvin b1ab3a66d2 build: Fix old build_sparkd.py script
Things have become a lot simpler since it was originally written.
2024-06-05 15:44:36 +00:00
John Colvin 2d8047ba80 build: Tidy up build/publish_spark scripts
In preparation for the first deployment of the "development" branch.
2024-04-02 01:47:47 -05:00
John Colvin fd9eaa65ed build: Add script to build DataStreamSuite and cpp deps 2023-09-30 13:54:58 -04:00
John Colvin cc2959249e build: Add lexer/parser.cs files to src ctrl
Move all lexer/parser files into a single dir, DataStreamGrammar, for
organizational purposes and to avoid dependency conflicts when
generating code. Note that we'll only need to regen cs if we modify the
parser/lexer files, which seems unlikely.
2023-09-03 13:53:07 -04:00
John Colvin e1b6971667 build: Greatly simplify build_spark.py
Make it a thin wrapper around dotnet/cmake clis
2023-08-23 00:46:21 -04:00
John Colvin cf34c13f03 build: Replace build_api_server.py w/ cmake/dotnet clis
Add instructions to build the Api Server in README
2023-08-22 14:38:39 -04:00
John Colvin 7545d3fd5c Merge remote-tracking branch 'origin/spark_crypto.prod' into development 2023-08-04 13:40:04 -04:00
John Colvin cde0f44b7d build: Default to Release cfg for 'publish' target 2023-07-18 22:25:06 -04:00
John Colvin 5230d6bab5 build: Clean up build_spark.py 2023-06-23 15:33:40 +00:00
John Colvin e3cd910acd build: Use CMakePresets in build_spark.py 2023-06-21 11:07:45 -04:00