Commit Graph
431 Commits
Author SHA1 Message Date
John Colvin fd6926eee8 Incorporate env files into docker compose
In order to support local (ephemeral) config. For example, to run sparkd
with replay md to troubleshoot an issue.
2024-11-26 18:53:32 +00:00
John Colvin dd10feb286 build: Add 'compose-file' arg to deploy script 2024-11-26 18:53:32 +00:00
John Colvin 5739f948b8 build: Change publish_spark.py default verbosity to 2
This equates to STATUS and normal in cmake and dotnet, respectively
2024-11-26 18:53:32 +00:00
John Colvin 4a193bd0e2 Add scripts to deploy apps via docker compose 2024-11-26 18:53:32 +00:00
John Colvin 75907e2bd5 Clean up build scripts 2024-11-26 18:53:32 +00:00
John Colvin e402b8ad83 build: Configure admin webapp build entirely in CMakeLists
Get rid of build_admin_web_app.py
2024-11-25 20:58:11 +00:00
John Colvin b523ab22f0 refactor: Move domain types into new lib_domain
Also rename files in snake_case (standard naming convention for protos)
https://protobuf.dev/programming-guides/style/
2024-11-25 20:58:11 +00:00
John Colvin 38f6c71551 refactor: Organize libspark proj w/ stuttering include
This new structure provides several benefits:

* Allows us to install multiple CMake projects in the same destination
  dir (without include path conflicts).
  - This makes shipping the trading algo sdk a trivial excercise. Simply
    copy the lib and include dirs from the install dest. That's it!
* Puts us in a position to break up the monolithic libspark into smaller
  modules, which are easier to manage.
2024-11-25 16:49:10 +00:00
John Colvin 0248b73bde Add api server to docker compose 2024-11-25 15:26:59 +00:00
John Colvin ea0f1153e4 Configure datastream deployment with docker-compose
To deploy to aws, connect to our aws vpn then run:
tools/deploy-docker-compose.sh

The deploy script does the following:
* builds sparkd
* builds datastream apps (pubsub, md-publisher, and webapp)
* uploads the build to aws
* bounces all services (sparkd, pubsub, md-publisher, and webapp)

Note: the name 'md-publisher' refers to DataStreamPublisher w/ MarketData
configuration
2024-11-25 15:26:58 +00:00
John Colvin 97229a8f1a build: Change publish_data_stream.py to publish the sln
We were publishing individual projects as a workaround for the
dependency issue (bug) that caused some projects to incorrectly target
net8.0. The fact that publishing individual projects "just worked" was
probably dumb luck. Now that the underlying issue has been addressed, we
can resume publishing the sln as a whole.
2024-11-25 15:26:58 +00:00
John Colvin c7205fcdd0 fix: Add self-signed certificate to Dockerfile for ds webapp 2024-11-25 15:26:58 +00:00
John Colvin 2939cb601f Define webapp specific env vars in generic datastream app deploy script
For now, this makes things easier, as I don't need to write a special script just for the webapp
2024-11-25 15:26:58 +00:00
John Colvin 2454f89ec4 Add config profile and Dockerfile args to deploy script 2024-11-25 15:26:58 +00:00
John Colvin df13b0c76f Copy DataStream WebApp 'dist' to 'wwwroot' in deploy script 2024-11-25 15:26:58 +00:00
John Colvin 4deeacda4a Add scripts to deploy data stream apps 2024-11-25 15:26:58 +00:00
John Colvin 50596abce3 Publish Bts.DataStreamWebApp.Server along with PubSub & DataStream 2024-11-25 15:26:58 +00:00
John Colvin d64ddb3477 build: Add scripts to publish data stream apps 2024-11-25 15:26:57 +00:00
John Colvin 61affcf749 Fix build scripts so they propagate the exit code back to the shell
I noticed this was broken because launch.json configurations would still start the debugger
even when the preLaunchTask (build) failed!
2024-11-25 15:26:57 +00:00
John Colvin 008e84dcf0 build: Use separate cmd to build DataStreamSuite/QueryAnalyzer.sln
The DataStreamSuite.sln builds the new webapp now, which will
(eventually) replace DSQA. We'll keep it around for now, but with a
clean separation from the main build.
2024-11-25 15:26:57 +00:00
John Colvin 9cc0108a01 feat: Add DataStream WebApp
Included in DataStreamSuite.sln, but not DataStreamQueryAnalyzer.sln
2024-11-25 15:26:57 +00:00
John Colvin 60c784f371 cleanup: Fix benign typos in build scripts
Change BUILD_SPARK_PY var name to BUILD_PY (because these scripts to actually build 'spark')
2024-11-25 15:26:57 +00:00
John Colvin b69b7b3350 vscode: Use '\' dir separator in show_last_run.cmd
This script only runs on Windows
2024-06-08 15:08:25 -05:00
John Colvin f93d28ee76 vscode: Add task to view logs for last DSQA run 2024-06-07 11:34:36 -05:00
John Colvin 7b5a6031af build: Use consistent naming conventions in CMake presets
We'll use similar naming conventions for our (vscode) tasks
2024-06-06 09:15:17 -05:00
John Colvin 54e96a06cb build: Build DSQA on Windows in build_data_stream.py
It was only building DataStreamSuite.sln before, which is crossplatform
now so it no longer includes DSQA.
2024-06-06 09:15:16 -05:00
John Colvin 6049826e90 build: Make build_data_stream.py cross platform
Previously it only worked on Windows
2024-06-06 09:15:16 -05:00
John Colvin f1744450ca build: Add script to build api server 2024-06-06 09:15:15 -05:00
John Colvin 3a7827a75b build: Automatically load vs dev prompt for win builds 2024-06-06 09:15:15 -05:00
John Colvin b1ab3a66d2 build: Fix old build_sparkd.py script
Things have become a lot simpler since it was originally written.
2024-06-05 15:44:36 +00:00
John Colvin ceca96e10a Add build_data_stream.sh script for Linux 2024-06-05 15:44:35 +00:00
John Colvin df1b45ca3e cleanup: Remove leftover legacy files 2024-05-25 15:37:33 +00:00
John Colvin 2d8047ba80 build: Tidy up build/publish_spark scripts
In preparation for the first deployment of the "development" branch.
2024-04-02 01:47:47 -05:00
John Colvin a091a95f50 build: Add .cmd wrapper script for build_data_stream.py 2023-10-02 19:38:38 -04:00
John Colvin fd9eaa65ed build: Add script to build DataStreamSuite and cpp deps 2023-09-30 13:54:58 -04:00
John Colvin cc2959249e build: Add lexer/parser.cs files to src ctrl
Move all lexer/parser files into a single dir, DataStreamGrammar, for
organizational purposes and to avoid dependency conflicts when
generating code. Note that we'll only need to regen cs if we modify the
parser/lexer files, which seems unlikely.
2023-09-03 13:53:07 -04:00
John Colvin e1b6971667 build: Greatly simplify build_spark.py
Make it a thin wrapper around dotnet/cmake clis
2023-08-23 00:46:21 -04:00
John Colvin cf34c13f03 build: Replace build_api_server.py w/ cmake/dotnet clis
Add instructions to build the Api Server in README
2023-08-22 14:38:39 -04:00
John Colvin 7545d3fd5c Merge remote-tracking branch 'origin/spark_crypto.prod' into development 2023-08-04 13:40:04 -04:00
John Colvin 2ed23159ca style: Clean up jenkins build script
* Get rid of double backslashes in quoted paths to improve readability
* Call 'activate.bat' instead of 'activate' (official python docs say to
  use the former)
* Add docs
2023-07-26 15:26:30 -04:00
John Colvin faa8438adf build: Make jenkins build script independent of working dir
This alleviates an unneccessary burden on callers (previously, it had to
be run from the repo root dir).
2023-07-26 11:59:03 -04:00
John Colvin 9e4354fb55 fix buid: Don't goto ERROR just bc venv already exists
This seems like a typo
2023-07-18 23:40:18 -04:00
John Colvin e6bb4c73a8 fix build (jenkins): Upgrade pip via 'python -m pip'
For some reason 'pip install --upgrade pip' works on my laptop, but fails on
the Jenkins box.
2023-07-18 22:53:32 -04:00
John Colvin e43a2a4818 build: Add build script for new Jenkins (on aws) 2023-07-18 22:25:56 -04:00
John Colvin cde0f44b7d build: Default to Release cfg for 'publish' target 2023-07-18 22:25:06 -04:00
John Colvin 5230d6bab5 build: Clean up build_spark.py 2023-06-23 15:33:40 +00:00
John Colvin e3cd910acd build: Use CMakePresets in build_spark.py 2023-06-21 11:07:45 -04:00
John Colvin cb47177b8e build: Remove unused 'link_build_dir' arg 2023-06-21 11:07:45 -04:00
John Colvin af0b7d9146 fix build: Set cmake policy CMP0076 before target_sources
* Removes warning when configuring build on Windows
* Moves 'vs_filters.cmake' to 'cmake' dir (w/ other cmake scripts)
2023-06-21 11:07:45 -04:00
John Colvin e1234fe583 fix: Fix typo in log helper script 'show_last_run.sh'
So it actually applies the 'num_lines' override arg
2023-06-07 13:16:21 -04:00