Commit Graph
423 Commits
Author SHA1 Message Date
John Colvin 99da1d3dc3 Add api server to docker compose 2024-10-21 13:29:59 +00:00
John Colvin 146a52f0ab Configure datastream deployment with docker-compose
To deploy to aws, connect to our aws vpn then run:
tools/deploy-docker-compose.sh

The deploy script does the following:
* builds sparkd
* builds datastream apps (pubsub, md-publisher, and webapp)
* uploads the build to aws
* bounces all services (sparkd, pubsub, md-publisher, and webapp)

Note: the name 'md-publisher' refers to DataStreamPublisher w/ MarketData
configuration
2024-10-21 13:29:59 +00:00
John Colvin 6c23240d9c build: Change publish_data_stream.py to publish the sln
We were publishing individual projects as a workaround for the
dependency issue (bug) that caused some projects to incorrectly target
net8.0. The fact that publishing individual projects "just worked" was
probably dumb luck. Now that the underlying issue has been addressed, we
can resume publishing the sln as a whole.
2024-10-21 13:29:58 +00:00
John Colvin e76c4d3aaf fix: Add self-signed certificate to Dockerfile for ds webapp 2024-10-21 13:29:58 +00:00
John Colvin aa7dda3dca Define webapp specific env vars in generic datastream app deploy script
For now, this makes things easier, as I don't need to write a special script just for the webapp
2024-10-21 13:29:58 +00:00
John Colvin af2536bbb9 Add config profile and Dockerfile args to deploy script 2024-10-21 13:29:58 +00:00
John Colvin b6cb7fcd1a Copy DataStream WebApp 'dist' to 'wwwroot' in deploy script 2024-10-21 13:29:58 +00:00
John Colvin 894add7df6 Add scripts to deploy data stream apps 2024-10-21 13:29:58 +00:00
John Colvin d78079f442 Publish Bts.DataStreamWebApp.Server along with PubSub & DataStream 2024-10-21 13:29:58 +00:00
John Colvin d688965160 build: Add scripts to publish data stream apps 2024-10-21 13:29:57 +00:00
John Colvin bdd5beb783 Fix build scripts so they propagate the exit code back to the shell
I noticed this was broken because launch.json configurations would still start the debugger
even when the preLaunchTask (build) failed!
2024-10-21 13:29:57 +00:00
John Colvin a0029ed4c6 build: Use separate cmd to build DataStreamSuite/QueryAnalyzer.sln
The DataStreamSuite.sln builds the new webapp now, which will
(eventually) replace DSQA. We'll keep it around for now, but with a
clean separation from the main build.
2024-10-21 13:29:57 +00:00
John Colvin add3aa8bd3 feat: Add DataStream WebApp
Included in DataStreamSuite.sln, but not DataStreamQueryAnalyzer.sln
2024-10-21 13:29:57 +00:00
John Colvin 14c6d5f687 cleanup: Fix benign typos in build scripts
Change BUILD_SPARK_PY var name to BUILD_PY (because these scripts to actually build 'spark')
2024-10-21 13:29:57 +00:00
John Colvin b69b7b3350 vscode: Use '\' dir separator in show_last_run.cmd
This script only runs on Windows
2024-06-08 15:08:25 -05:00
John Colvin f93d28ee76 vscode: Add task to view logs for last DSQA run 2024-06-07 11:34:36 -05:00
John Colvin 7b5a6031af build: Use consistent naming conventions in CMake presets
We'll use similar naming conventions for our (vscode) tasks
2024-06-06 09:15:17 -05:00
John Colvin 54e96a06cb build: Build DSQA on Windows in build_data_stream.py
It was only building DataStreamSuite.sln before, which is crossplatform
now so it no longer includes DSQA.
2024-06-06 09:15:16 -05:00
John Colvin 6049826e90 build: Make build_data_stream.py cross platform
Previously it only worked on Windows
2024-06-06 09:15:16 -05:00
John Colvin f1744450ca build: Add script to build api server 2024-06-06 09:15:15 -05:00
John Colvin 3a7827a75b build: Automatically load vs dev prompt for win builds 2024-06-06 09:15:15 -05:00
John Colvin b1ab3a66d2 build: Fix old build_sparkd.py script
Things have become a lot simpler since it was originally written.
2024-06-05 15:44:36 +00:00
John Colvin ceca96e10a Add build_data_stream.sh script for Linux 2024-06-05 15:44:35 +00:00
John Colvin df1b45ca3e cleanup: Remove leftover legacy files 2024-05-25 15:37:33 +00:00
John Colvin 2d8047ba80 build: Tidy up build/publish_spark scripts
In preparation for the first deployment of the "development" branch.
2024-04-02 01:47:47 -05:00
John Colvin a091a95f50 build: Add .cmd wrapper script for build_data_stream.py 2023-10-02 19:38:38 -04:00
John Colvin fd9eaa65ed build: Add script to build DataStreamSuite and cpp deps 2023-09-30 13:54:58 -04:00
John Colvin cc2959249e build: Add lexer/parser.cs files to src ctrl
Move all lexer/parser files into a single dir, DataStreamGrammar, for
organizational purposes and to avoid dependency conflicts when
generating code. Note that we'll only need to regen cs if we modify the
parser/lexer files, which seems unlikely.
2023-09-03 13:53:07 -04:00
John Colvin e1b6971667 build: Greatly simplify build_spark.py
Make it a thin wrapper around dotnet/cmake clis
2023-08-23 00:46:21 -04:00
John Colvin cf34c13f03 build: Replace build_api_server.py w/ cmake/dotnet clis
Add instructions to build the Api Server in README
2023-08-22 14:38:39 -04:00
John Colvin 7545d3fd5c Merge remote-tracking branch 'origin/spark_crypto.prod' into development 2023-08-04 13:40:04 -04:00
John Colvin 2ed23159ca style: Clean up jenkins build script
* Get rid of double backslashes in quoted paths to improve readability
* Call 'activate.bat' instead of 'activate' (official python docs say to
  use the former)
* Add docs
2023-07-26 15:26:30 -04:00
John Colvin faa8438adf build: Make jenkins build script independent of working dir
This alleviates an unneccessary burden on callers (previously, it had to
be run from the repo root dir).
2023-07-26 11:59:03 -04:00
John Colvin 9e4354fb55 fix buid: Don't goto ERROR just bc venv already exists
This seems like a typo
2023-07-18 23:40:18 -04:00
John Colvin e6bb4c73a8 fix build (jenkins): Upgrade pip via 'python -m pip'
For some reason 'pip install --upgrade pip' works on my laptop, but fails on
the Jenkins box.
2023-07-18 22:53:32 -04:00
John Colvin e43a2a4818 build: Add build script for new Jenkins (on aws) 2023-07-18 22:25:56 -04:00
John Colvin cde0f44b7d build: Default to Release cfg for 'publish' target 2023-07-18 22:25:06 -04:00
John Colvin 5230d6bab5 build: Clean up build_spark.py 2023-06-23 15:33:40 +00:00
John Colvin e3cd910acd build: Use CMakePresets in build_spark.py 2023-06-21 11:07:45 -04:00
John Colvin cb47177b8e build: Remove unused 'link_build_dir' arg 2023-06-21 11:07:45 -04:00
John Colvin af0b7d9146 fix build: Set cmake policy CMP0076 before target_sources
* Removes warning when configuring build on Windows
* Moves 'vs_filters.cmake' to 'cmake' dir (w/ other cmake scripts)
2023-06-21 11:07:45 -04:00
John Colvin e1234fe583 fix: Fix typo in log helper script 'show_last_run.sh'
So it actually applies the 'num_lines' override arg
2023-06-07 13:16:21 -04:00
John Colvin e64dc49a64 fix build: Respect cfg path order in gen'd defaults
The path order represents the value-inheritance hierarchy. This way,
values from files earlier in the path take precedence over those from
later files.
2023-06-07 13:16:21 -04:00
John Colvin 82c126362d fix build: Fix a few path errors in publish target 2023-05-31 22:13:52 -04:00
arhamGIT 3dd7d0999e merge 2023-04-26 12:33:32 -04:00
John Colvin c85e221e9f build: Automatically run build_spark.py in venv 2023-04-20 00:46:23 -04:00
John Colvin 947c071cee build: Remove unused cmake def 2023-04-20 00:46:22 -04:00
John Colvin d8611536ee fix build script: Fix venv path 2023-04-14 15:16:32 -04:00
John Colvin c94ffd1a56 cli: Add spark.console.cmd (alias for Spark.Console.exe) 2023-04-14 15:16:32 -04:00
John Colvin cddf081e19 fix (cli): Read %~dp0 before SHIFT
I just discovered that, when SHIFT is run (the first time), the value of
%~dp0 changes from the script's parent dir to the current working dir
2023-04-14 13:18:28 -04:00