John Colvin
99da1d3dc3
Add api server to docker compose
2024-10-21 13:29:59 +00:00
John Colvin
146a52f0ab
Configure datastream deployment with docker-compose
...
To deploy to aws, connect to our aws vpn then run:
tools/deploy-docker-compose.sh
The deploy script does the following:
* builds sparkd
* builds datastream apps (pubsub, md-publisher, and webapp)
* uploads the build to aws
* bounces all services (sparkd, pubsub, md-publisher, and webapp)
Note: the name 'md-publisher' refers to DataStreamPublisher w/ MarketData
configuration
2024-10-21 13:29:59 +00:00
John Colvin
6c23240d9c
build: Change publish_data_stream.py to publish the sln
...
We were publishing individual projects as a workaround for the
dependency issue (bug) that caused some projects to incorrectly target
net8.0. The fact that publishing individual projects "just worked" was
probably dumb luck. Now that the underlying issue has been addressed, we
can resume publishing the sln as a whole.
2024-10-21 13:29:58 +00:00
John Colvin
e76c4d3aaf
fix: Add self-signed certificate to Dockerfile for ds webapp
2024-10-21 13:29:58 +00:00
John Colvin
aa7dda3dca
Define webapp specific env vars in generic datastream app deploy script
...
For now, this makes things easier, as I don't need to write a special script just for the webapp
2024-10-21 13:29:58 +00:00
John Colvin
af2536bbb9
Add config profile and Dockerfile args to deploy script
2024-10-21 13:29:58 +00:00
John Colvin
b6cb7fcd1a
Copy DataStream WebApp 'dist' to 'wwwroot' in deploy script
2024-10-21 13:29:58 +00:00
John Colvin
894add7df6
Add scripts to deploy data stream apps
2024-10-21 13:29:58 +00:00
John Colvin
d78079f442
Publish Bts.DataStreamWebApp.Server along with PubSub & DataStream
2024-10-21 13:29:58 +00:00
John Colvin
d688965160
build: Add scripts to publish data stream apps
2024-10-21 13:29:57 +00:00
John Colvin
bdd5beb783
Fix build scripts so they propagate the exit code back to the shell
...
I noticed this was broken because launch.json configurations would still start the debugger
even when the preLaunchTask (build) failed!
2024-10-21 13:29:57 +00:00
John Colvin
a0029ed4c6
build: Use separate cmd to build DataStreamSuite/QueryAnalyzer.sln
...
The DataStreamSuite.sln builds the new webapp now, which will
(eventually) replace DSQA. We'll keep it around for now, but with a
clean separation from the main build.
2024-10-21 13:29:57 +00:00
John Colvin
add3aa8bd3
feat: Add DataStream WebApp
...
Included in DataStreamSuite.sln, but not DataStreamQueryAnalyzer.sln
2024-10-21 13:29:57 +00:00
John Colvin
14c6d5f687
cleanup: Fix benign typos in build scripts
...
Change BUILD_SPARK_PY var name to BUILD_PY (because these scripts to actually build 'spark')
2024-10-21 13:29:57 +00:00
John Colvin
b69b7b3350
vscode: Use '\' dir separator in show_last_run.cmd
...
This script only runs on Windows
2024-06-08 15:08:25 -05:00
John Colvin
f93d28ee76
vscode: Add task to view logs for last DSQA run
2024-06-07 11:34:36 -05:00
John Colvin
7b5a6031af
build: Use consistent naming conventions in CMake presets
...
We'll use similar naming conventions for our (vscode) tasks
2024-06-06 09:15:17 -05:00
John Colvin
54e96a06cb
build: Build DSQA on Windows in build_data_stream.py
...
It was only building DataStreamSuite.sln before, which is crossplatform
now so it no longer includes DSQA.
2024-06-06 09:15:16 -05:00
John Colvin
6049826e90
build: Make build_data_stream.py cross platform
...
Previously it only worked on Windows
2024-06-06 09:15:16 -05:00
John Colvin
f1744450ca
build: Add script to build api server
2024-06-06 09:15:15 -05:00
John Colvin
3a7827a75b
build: Automatically load vs dev prompt for win builds
2024-06-06 09:15:15 -05:00
John Colvin
b1ab3a66d2
build: Fix old build_sparkd.py script
...
Things have become a lot simpler since it was originally written.
2024-06-05 15:44:36 +00:00
John Colvin
ceca96e10a
Add build_data_stream.sh script for Linux
2024-06-05 15:44:35 +00:00
John Colvin
df1b45ca3e
cleanup: Remove leftover legacy files
2024-05-25 15:37:33 +00:00
John Colvin
2d8047ba80
build: Tidy up build/publish_spark scripts
...
In preparation for the first deployment of the "development" branch.
2024-04-02 01:47:47 -05:00
John Colvin
a091a95f50
build: Add .cmd wrapper script for build_data_stream.py
2023-10-02 19:38:38 -04:00
John Colvin
fd9eaa65ed
build: Add script to build DataStreamSuite and cpp deps
2023-09-30 13:54:58 -04:00
John Colvin
cc2959249e
build: Add lexer/parser.cs files to src ctrl
...
Move all lexer/parser files into a single dir, DataStreamGrammar, for
organizational purposes and to avoid dependency conflicts when
generating code. Note that we'll only need to regen cs if we modify the
parser/lexer files, which seems unlikely.
2023-09-03 13:53:07 -04:00
John Colvin
e1b6971667
build: Greatly simplify build_spark.py
...
Make it a thin wrapper around dotnet/cmake clis
2023-08-23 00:46:21 -04:00
John Colvin
cf34c13f03
build: Replace build_api_server.py w/ cmake/dotnet clis
...
Add instructions to build the Api Server in README
2023-08-22 14:38:39 -04:00
John Colvin
7545d3fd5c
Merge remote-tracking branch 'origin/spark_crypto.prod' into development
2023-08-04 13:40:04 -04:00
John Colvin
2ed23159ca
style: Clean up jenkins build script
...
* Get rid of double backslashes in quoted paths to improve readability
* Call 'activate.bat' instead of 'activate' (official python docs say to
use the former)
* Add docs
2023-07-26 15:26:30 -04:00
John Colvin
faa8438adf
build: Make jenkins build script independent of working dir
...
This alleviates an unneccessary burden on callers (previously, it had to
be run from the repo root dir).
2023-07-26 11:59:03 -04:00
John Colvin
9e4354fb55
fix buid: Don't goto ERROR just bc venv already exists
...
This seems like a typo
2023-07-18 23:40:18 -04:00
John Colvin
e6bb4c73a8
fix build (jenkins): Upgrade pip via 'python -m pip'
...
For some reason 'pip install --upgrade pip' works on my laptop, but fails on
the Jenkins box.
2023-07-18 22:53:32 -04:00
John Colvin
e43a2a4818
build: Add build script for new Jenkins (on aws)
2023-07-18 22:25:56 -04:00
John Colvin
cde0f44b7d
build: Default to Release cfg for 'publish' target
2023-07-18 22:25:06 -04:00
John Colvin
5230d6bab5
build: Clean up build_spark.py
2023-06-23 15:33:40 +00:00
John Colvin
e3cd910acd
build: Use CMakePresets in build_spark.py
2023-06-21 11:07:45 -04:00
John Colvin
cb47177b8e
build: Remove unused 'link_build_dir' arg
2023-06-21 11:07:45 -04:00
John Colvin
af0b7d9146
fix build: Set cmake policy CMP0076 before target_sources
...
* Removes warning when configuring build on Windows
* Moves 'vs_filters.cmake' to 'cmake' dir (w/ other cmake scripts)
2023-06-21 11:07:45 -04:00
John Colvin
e1234fe583
fix: Fix typo in log helper script 'show_last_run.sh'
...
So it actually applies the 'num_lines' override arg
2023-06-07 13:16:21 -04:00
John Colvin
e64dc49a64
fix build: Respect cfg path order in gen'd defaults
...
The path order represents the value-inheritance hierarchy. This way,
values from files earlier in the path take precedence over those from
later files.
2023-06-07 13:16:21 -04:00
John Colvin
82c126362d
fix build: Fix a few path errors in publish target
2023-05-31 22:13:52 -04:00
arhamGIT
3dd7d0999e
merge
2023-04-26 12:33:32 -04:00
John Colvin
c85e221e9f
build: Automatically run build_spark.py in venv
2023-04-20 00:46:23 -04:00
John Colvin
947c071cee
build: Remove unused cmake def
2023-04-20 00:46:22 -04:00
John Colvin
d8611536ee
fix build script: Fix venv path
2023-04-14 15:16:32 -04:00
John Colvin
c94ffd1a56
cli: Add spark.console.cmd (alias for Spark.Console.exe)
2023-04-14 15:16:32 -04:00
John Colvin
cddf081e19
fix (cli): Read %~dp0 before SHIFT
...
I just discovered that, when SHIFT is run (the first time), the value of
%~dp0 changes from the script's parent dir to the current working dir
2023-04-14 13:18:28 -04:00