John Colvin
c0448c7c8d
deploy: Resolve profile dirs with regex patterns
2026-03-26 16:20:41 -04:00
John Colvin
21052090a2
fix (pre-emptive): Format dotnet flag opts correctly
...
In pack_spark_sdk.py, the dotnet pack cmd is constructed with a dict of
opts. If the value part is None, the opt should be formatted as a flag,
"--key" instead of "--key None".
2026-03-07 13:32:03 -05:00
John Colvin
9bd32b8225
fix build (pack cs sdk): Fix --(no-)cs toggle behavior
...
* Always run "dotnet pack", regardless of csharp toggle opt
* Run "dotnet publish" if csharp toggled on
2026-03-07 13:20:56 -05:00
John Colvin
5bc2e29096
fix build: Only check "dsqa" flag when building on windows
...
The --dsqa opt isn't defined when running on linux.
2026-02-21 17:55:12 +00:00
John Colvin
a605e57bee
build: Add dsqa toggle to build_data_stream.py
...
Windows only
2026-02-20 02:33:38 -05:00
John Colvin
3842efcef1
build: Define build py requirements in separate files
...
No need to install backend build deps when building the front-end, and
vice versa. The build scripts (build_spark.cmd, build_sparkd.sh, etc.)
automatically install only the necessary requirements now.
Note: We'll keep the monolith requirements.txt file around (at least for
now), as it includes more than just build deps (e.g., numpy, pandas, and
jupyter for data analysis; pybit for bybit plugins, etc.).
2026-02-03 16:57:32 +00:00
John Colvin
635107eb68
cleanup: Remove unused py code
2026-02-03 05:20:29 +00:00
John Colvin
01c61bb16d
fix build: Publish spark_api_server csproj instead of the sln
...
There's a version conflict with System.Data.Common when publishing the sln.
2026-01-22 05:50:49 +00:00
John Colvin
2eb80082d8
refactor: Move Bts.Extensions.Hosting, Bts.Net to cs dir
...
They aren't exclusively used by DataStream csprojs
2026-01-21 14:20:39 -05:00
John Colvin
d55ef854ba
fix build: Don't actually copy files when building with --dry-run
2026-01-21 14:03:01 -05:00
John Colvin
b1417ece5d
fix build: Fix cmake windows preset in publish_api_server.py
...
Changed from 'spark' -> 'spark-windows'
2026-01-21 14:03:01 -05:00
John Colvin
f6ae219a72
build: Add a few toggle opts to publish py scripts
...
--(no-)cpp Enable/disable cpp build
--(no-)cs Enable/disable csharp build
--(no-)csproj-refs Depend on csproj refs, or locally built nugets
2026-01-21 14:03:01 -05:00
John Colvin
f88915a940
fix build: Add --spark-version opt in build_data_stream.py
2026-01-07 02:34:56 -05:00
John Colvin
8c458d48bb
feat: Add server side csharp sdk
...
For Ophir
2026-01-04 21:19:00 +00:00
John Colvin
bb1e7c30c8
build: Add msbuild flag to reference projs or local nugets
...
So we can use project refs for dev builds, and package refs for releases.
2026-01-04 21:19:00 +00:00
John Colvin
4bc1a4c833
build (qol): Add opts to only build cpp or cs
...
Usually, we're only working in one language or the other, and don't need
to rebuild both.
2026-01-04 21:19:00 +00:00
John Colvin
69e4ed84c1
build: Generate version info via git cli in build scripts
...
Instead of GitVersion.MsBuild nuget
2026-01-04 21:18:58 +00:00
John Colvin
4d5d0f8573
build: Consolidate cmake presets
2026-01-04 21:08:13 +00:00
John Colvin
76fa35382e
refactor: Add DataStream service modules for Dependency Injection
...
Namely, in DataStreamPublisher and PubSubServer. This is part of a broader
paradigm shift towards Dependency Injection and Factory Pattern.
https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection-guidelines
* Use interfaces to decouple services:
- Decouple ManagedSubscriber from ISubscriberCore impl
- Decouple PubTableBroker from IPublisherCore impl
- Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
* Model config profiles after dotnet appsettings
- Rename Configurations/<name>.json -> appsettings.<name>.json
- Make json fields match corresponding csharp props verbatim
* The Dependency Injection framework doesn't support JsonPropertyName
attributes. Config class props must match json fields verbatim.
* Move reusable startup code into new Bts.Extensions.Hosting proj
* Add pubsub and md-publisher services to compose.yaml
2026-01-04 21:08:11 +00:00
John Colvin
4681f22e71
deploy: Move configurations to new stutter dir "app_sparkd"
2025-06-16 11:08:54 -04:00
John Colvin
af9eb670c1
build: Update build scripts to target dotnet 8
2025-04-30 12:06:28 -04:00
John Colvin
61f04c4605
fix: Implement state machine for binance cxl/replace
2025-04-28 17:12:59 +00:00
John Colvin
eeffb735ae
deploy (fix): Override existing values when loading env files
2025-04-16 13:48:24 +00:00
John Colvin
b602997f3a
deploy: Build sparkd if any service starts with "sparkd"
...
Previously, we only built sparkd for the "sparkd" service (i.e., exact match)
2025-04-16 13:48:24 +00:00
arham
14e3eee0bc
Add: config changes to make server controller work and set up hk zen server
2025-04-09 09:42:47 -04:00
John Colvin
716c4dd674
tool: Add script to dump sparkd logs to sqlite db
2025-03-19 14:41:23 +00:00
John Colvin
b555850015
build: Package & deploy sparkd with docker compose
2025-03-10 03:05:59 +00:00
John Colvin
c8fc649b6b
fix build: Pass config arg to cmake --build
...
For Ninja Multi-Config builds (i.e., api server and data streams)
2025-01-28 19:59:09 +00:00
John Colvin
b1f4654b3c
Fix benign typo in publish_spark.py
2025-01-21 17:28:50 -05:00
John Colvin
9d350334c0
fix build (installer): Fix install path typo in publish_spark.py
2025-01-21 17:28:05 -05:00
John Colvin
86eba20d61
build: Make compiler/build type case insensitive
2025-01-05 06:25:11 +00:00
John Colvin
22ae51fb99
build: Clean up build py scripts
2025-01-03 16:59:07 +00:00
John Colvin
838dbcd886
build: In py build scripts, exit with code 1 on failure
...
Previously, they would exit with 0 no matter what happened.
2025-01-03 16:59:07 +00:00
John Colvin
4d41704dd2
build: Configure admin webapp build entirely in CMakeLists
...
Get rid of build_admin_web_app.py
2025-01-03 16:59:07 +00:00
John Colvin
8cde8ee6e1
build: Add script to publish the api server
...
This will build the app as a self-contained executable here:
build/Release/bin/net7.0/linux-x64/publish
2024-11-28 16:35:21 +00:00
John Colvin
7b5a6031af
build: Use consistent naming conventions in CMake presets
...
We'll use similar naming conventions for our (vscode) tasks
2024-06-06 09:15:17 -05:00
John Colvin
54e96a06cb
build: Build DSQA on Windows in build_data_stream.py
...
It was only building DataStreamSuite.sln before, which is crossplatform
now so it no longer includes DSQA.
2024-06-06 09:15:16 -05:00
John Colvin
6049826e90
build: Make build_data_stream.py cross platform
...
Previously it only worked on Windows
2024-06-06 09:15:16 -05:00
John Colvin
f1744450ca
build: Add script to build api server
2024-06-06 09:15:15 -05:00
John Colvin
b1ab3a66d2
build: Fix old build_sparkd.py script
...
Things have become a lot simpler since it was originally written.
2024-06-05 15:44:36 +00:00
John Colvin
2d8047ba80
build: Tidy up build/publish_spark scripts
...
In preparation for the first deployment of the "development" branch.
2024-04-02 01:47:47 -05:00
John Colvin
fd9eaa65ed
build: Add script to build DataStreamSuite and cpp deps
2023-09-30 13:54:58 -04:00
John Colvin
cc2959249e
build: Add lexer/parser.cs files to src ctrl
...
Move all lexer/parser files into a single dir, DataStreamGrammar, for
organizational purposes and to avoid dependency conflicts when
generating code. Note that we'll only need to regen cs if we modify the
parser/lexer files, which seems unlikely.
2023-09-03 13:53:07 -04:00
John Colvin
e1b6971667
build: Greatly simplify build_spark.py
...
Make it a thin wrapper around dotnet/cmake clis
2023-08-23 00:46:21 -04:00
John Colvin
cf34c13f03
build: Replace build_api_server.py w/ cmake/dotnet clis
...
Add instructions to build the Api Server in README
2023-08-22 14:38:39 -04:00
John Colvin
7545d3fd5c
Merge remote-tracking branch 'origin/spark_crypto.prod' into development
2023-08-04 13:40:04 -04:00
John Colvin
cde0f44b7d
build: Default to Release cfg for 'publish' target
2023-07-18 22:25:06 -04:00
John Colvin
5230d6bab5
build: Clean up build_spark.py
2023-06-23 15:33:40 +00:00
John Colvin
e3cd910acd
build: Use CMakePresets in build_spark.py
2023-06-21 11:07:45 -04:00
John Colvin
cb47177b8e
build: Remove unused 'link_build_dir' arg
2023-06-21 11:07:45 -04:00