Compare commits

..
Author SHA1 Message Date
John Colvin 8e8d46354e Put cme config url in GlbxMarketDataConfig proto
Instead of using magic strings
2024-10-14 15:51:45 +00:00
John Colvin 10a68fba30 refactor: Change raw IDGenerator ptr to shared_ptr 2024-10-14 15:51:45 +00:00
John Colvin 01d677d8b0 refactor: Return row id in IStore::retrieve_one
This is a prerequisite to synchronizing security ids stored via IDGenerator
(the "new" way to store ids, added when we implemented Deribit), and
SecurityDB (the original way from spark days). The problem was that these
2 implementations were using separate databases. We need to unify the
interfaces between these storage services to avoid breaking existing code.
2024-10-14 15:51:45 +00:00
John Colvin 474d8747dc Add database dir and file fns to SparkDataDirectory 2024-10-14 15:51:45 +00:00
John Colvin dab8e939d5 Interpret db strng param as name or path in SqliteStore ctr 2024-10-14 15:51:45 +00:00
John Colvin 7fd501369e refactor: Normalize sec key indexing for old md sources
Now both old and new (domain) md sources index securities by
the same string format.
2024-10-14 15:51:45 +00:00
John Colvin d9838fd394 Add ostream operator for Venue 2024-10-14 15:51:45 +00:00
John Colvin 531498394b Add ostream operator for SecurityType 2024-10-14 15:51:44 +00:00
John Colvin 9d08ddfa53 refactor: Normalize sec key indexing for new (domain) md sources
This is part of a larger effort to unify the way we index securities in order
to generate unique ids. Older md sources (e.g., MDP3Listener) use
SecurityIDRecorder to generate ids, which uses a different database and
different key-string formatting than the IDGenerator used by the newer md
sources (e.g., GeneralMarketDataListener and various ccapi-based sources).
2024-10-14 15:51:44 +00:00
John Colvin 98f85b90ee refactor: Replace ConfigDownloader w/ lib_http fns
Essentially all it did was fetch config files via http GET. I ported some http
fns over from the cme-websocket-connector repo, which uses curlpp instead of
curl, and also supports async requests (coroutines) via the 'concurrencpp'

https://github.com/David-Haim/concurrencpp
2024-10-14 15:51:44 +00:00
John Colvin 752e2bac82 cleanup: Get rid of another Security type aliases 2024-10-14 15:51:44 +00:00
John Colvin eb12f4e582 cleanup: Remove unused code in Security.h/cpp 2024-10-14 15:51:44 +00:00
John Colvin 9ede4bb6eb Fix typo in 'Quantity' FMT_IMPL
I didn't realize the "value()" function rounded out the decimal,
so I replaced it with a call to "full_value()".
2024-10-14 15:51:44 +00:00
John Colvin 2c56ab2578 diag: Log when sqlite db is created 2024-10-14 15:51:44 +00:00
John Colvin 517d155c4a diag: Log when an MDProducer is created 2024-10-14 15:51:44 +00:00
John Colvin bf6fdcada7 diag: Add logs around synthetic security discovery in MDConsumer 2024-10-14 15:51:44 +00:00
John Colvin c8abb64538 refactor: Break shared mem md code up into smaller files
For easier reading/navigation
2024-10-14 15:51:44 +00:00
John Colvin 345a281a47 refactor: Break md relay/query server code up into smaller files
To make code navigation easier. This was a difficulty which I recently
confronted, while investigating a bug that prevented synthetics from
showing up the ui.
2024-10-14 15:51:44 +00:00
John Colvin de98e99a96 refactor: Get rid of 'TimeStamp' typedef in MDStructure.h
Before this commit, we had 4 different TimeStamp types/identifiers, which is
confusing enough as it is:

* bts::core::TimeStamp
  - in system/time_stamp.h
* bts::utils::TimeStamp
  - in spark/utils/TimeStamp.h
* bts::TimeStamp
  - in system/bts_chrono.h
  - alias for std::chrono::system_clock::time_point
* bts::md::TimeStamp
  - in spark/md/Shm/MDStructure.h
  - alias for bts::utils::TimeStamp
  - removed in this commit

Additional aliases for these TimeStamps only adds to the confusion (esp
when the alias comes from a nested include).
2024-10-14 15:51:44 +00:00
John Colvin dc0fd3d137 Make security 'formattable'
So it can be used as a param in LOG statements
2024-10-14 15:51:44 +00:00
John Colvin 01e73f93b3 refactor: Change refs/raw ptrs to smart ptrs in md gateway 2024-10-14 15:51:44 +00:00
John Colvin 22a225820d Fix typo in market_data.conf 2024-10-14 15:51:44 +00:00
John Colvin 2ca8c72eb8 Fix typo in namespace, co2 -> md::MDConsumerChannel
Not sure how this happened...
2024-10-14 15:51:44 +00:00
John Colvin 04b70f36c4 Add BTC and ETH to VENUE_SIM in symbols.conf 2024-10-14 15:51:44 +00:00
John Colvin a378fdb862 fix: Resolve main logging dir in new SparkDataDirectory class 2024-10-14 15:51:44 +00:00
John Colvin 25396d70ff refactor: Replace if/else with switch cases 2024-10-14 15:51:44 +00:00
John Colvin c3f88bd505 fix: Resolve onixs log & data dirs relative to app data root
When our logging code moved to lib_system, it lost access to the config
system. As a result, the "log_path" function, which was used by ICEMain
and MDP3Main, broke. I added a new class, SparkDataDirectory, to be the
central/common authority on paths to local storage.
2024-10-14 15:51:44 +00:00
John Colvin c7f3819b3c cleanup: Remove over-engineered subcommands
We don't need explicit subcommands to 'run onixs' md (just use config).
2024-10-14 15:51:44 +00:00
John Colvin 88de4ab76a webapp check pt? 2024-10-14 15:51:44 +00:00
John Colvin 7697f311f0 fix: Ensure IDGenerator instance is created before deref 2024-10-14 15:51:44 +00:00
John Colvin 8cd2bb3d49 fix: Expand undefined vars to empty string
Bugs (fixed):
  * Md shared memory file names have unexpanded variable: '${BTS_SHARED_MEMORY_PREFIX}'
2024-10-14 15:51:44 +00:00
John Colvin 5e62b42404 fix: Emplace MDConsumerHandle instead of copy assign 2024-10-14 15:51:44 +00:00
John Colvin 8665d2137f Generate c++ protos w/ relative paths
We shouldn't be generating protos defined outside of the current CMakeLists.txt dir.
2024-10-14 15:51:44 +00:00
John Colvin 811dee2d04 fix build: Make venue conversion constexpr 2024-10-14 15:51:44 +00:00
John Colvin 373b377da4 build: Use new/robust init_library fn in lib_domain 2024-10-14 15:51:44 +00:00
John Colvin 11afb0ff40 refactor: Move config services to lib_system
The proto files are still in lib_spark, w/ the exception of
system_config.proto, since that is a dep of ConfigurationState.
2024-10-14 15:51:44 +00:00
John Colvin 01b0ebe01d refactor: Move logging into lib_system
This is a fundamental part of our application suite, so system seems
like a sensible home.

Also moved a few logging deps (to lib_system):
* bts_chrono.h
* bts_no_copy.h
* defer.h
* dispatcher.h/cpp
* file_utils.h/cpp
* type_name.h
2024-10-14 15:20:47 +00:00
John Colvin 8bc692605f build: Move http code into its own lib 2024-10-14 15:19:20 +00:00
John Colvin 80da994a14 refactor: Move thread_manager to lib_system 2024-10-13 20:42:19 +00:00
John Colvin e5ba13eded build (linux): Migrate catch2 dep to vcpkg (from externals) 2024-10-13 20:42:19 +00:00
John Colvin bc8ca9afd8 build (linux): Migrate fmt dep to vcpkg (from externals)
Now both linux and windows use vcpkg's fmt
2024-10-13 20:41:37 +00:00
John Colvin 17d6a64df8 cleanup: Remove unnecessary includes 2024-10-13 19:38:42 +00:00
John Colvin ca4e796560 refactor: Move threads.proto to lib_system 2024-10-13 19:38:38 +00:00
John Colvin 32691e8c9c refactor: Move warn_ctrl.h into lib_system
This is another logging dep
2024-10-12 19:54:51 +00:00
John Colvin 0be0501716 build: Link boost_filesystem in lib_system 2024-10-12 19:54:51 +00:00
John Colvin 1e2c361231 refactor: Move os.h/cpp and deps to lib_system
This is a prerequisite for the next objective: move logging into its own lib.
As logging is a core dependency for so much of our system, this will enable
us to move more complex services (like md providers) into smaller libs.
2024-10-12 19:54:51 +00:00
John Colvin f4b7f4fd88 cleanup: Remove unused code 2024-10-12 19:54:51 +00:00
John Colvin 05af82fefc cleanup: Remove more unnecessary includes 2024-10-12 19:54:51 +00:00
John Colvin 7de36a877f refactor: Move SecurityMarketDataStatistics to lib_domain 2024-10-12 19:54:51 +00:00
John Colvin 732ed02dea cleanup: Remove unnecessary code 2024-10-12 19:54:51 +00:00
John Colvin bdf4ea595d cleanup: Remove empty header FTXSecurity.h 2024-10-12 19:54:51 +00:00
John Colvin dc90f2b79a cleanup: Remove temporary redirect headers
These were used to make code migration smoother (to the new lib_system and lib_domain)

* spark/algorithm/TryCatch.h -> system/algorithm/try_catch.h
* spark/core/BoolName.h -> system/bool_name.h
* spark/core/EnumMacros.h -> system/enum_macros.h
* spark/core/FMT.h -> system/FMT.h
* spark/core/FMT_IMPL.h -> system/FMT_IMPL.h
* spark/core/TimeSpan.h -> system/time_span.h
* spark/core/TimeStamp.h -> system/time_stamp.h
* spark/domain/Price.h -> domain/price.h
* spark/io/JsonReader.h -> system/json/json_reader.h
* spark/io/JsonDumper.h -> system/json/json_dumper.h
2024-10-12 19:54:49 +00:00
John Colvin 07b3947738 refactor: Move RFQ to lib_domain 2024-10-12 19:53:27 +00:00
John Colvin cc8e0bd6a9 refactor: Move AggressorSide proto to lib_domain
Get rid of non-proto AggressorSide enum while we're at it (consolidate to 1 type)
2024-10-12 19:53:27 +00:00
John Colvin 5959538aca cleanup: Remove more unused code 2024-10-12 19:53:27 +00:00
John Colvin 65f435598c refactor: Move a few Order types to lib_domain 2024-10-12 19:53:27 +00:00
John Colvin 70d89e5bc1 refactor: Move TimeInForce to lib_domain 2024-10-12 19:53:27 +00:00
John Colvin 87722765db cleanup: Remove unused code 2024-10-12 19:53:27 +00:00
John Colvin b6961f558a refactor: Move TradeRecord into lib_domain 2024-10-12 19:53:27 +00:00
John Colvin 351fca18e1 refactor: Move TradingStatus enum to lib_domain 2024-10-12 19:53:27 +00:00
John Colvin 13a5f78ab1 build: Update webpack-cli in admin webapp 2024-10-12 19:53:27 +00:00
John Colvin 8735187963 build: Configure admin webapp build entirely in CMakeLists
Get rid of build_admin_web_app.py
2024-10-12 19:53:27 +00:00
John Colvin edda8ae20c refactor: Move domain types into new lib_domain
Also rename files in snake_case (standard naming convention for protos)
https://protobuf.dev/programming-guides/style/
2024-10-12 19:53:26 +00:00
John Colvin b48bfbae79 refactor: Move collections into lib_system 2024-10-12 19:30:05 +00:00
John Colvin 465b293e54 refactor: Move a handful of types/utils to lib_system 2024-10-12 19:30:04 +00:00
John Colvin 7b37c67568 build: Add preset to build with 6 jobs (-j 6)
Otherwise the devcontainer may hog too much cpu and cause the container to freeze
2024-10-12 19:30:04 +00:00
John Colvin a57e1d5cf3 cleanup: Remove unused includes
Mostly collection macro headers
2024-10-12 19:30:04 +00:00
John Colvin 97b99f87d1 cleanup (build): Move protobuf service generation cmake into lib_spark
Because lib_spark owns all the services
2024-10-12 19:30:04 +00:00
John Colvin d071021305 refactor: Rename protobuf out dir "include" -> "protobufs"
This is a more accurate and precise name. The dir contains c++ code generated
from .proto files - both sources (.cc) and headers (.pb.h).
2024-10-12 19:30:04 +00:00
John Colvin f2553695ea refactor: Move ValueWrapper into lib_system
So other projects can use it
2024-10-12 19:30:04 +00:00
John Colvin a2fab4afaf Include bts_assert.h in bts_fatal.h
We shouldn't have to include bts_assert.h when we're already including bts_fatal.h
2024-10-12 19:30:04 +00:00
John Colvin 32a8b82cd4 refactor: Split MarketDataGateway up into providers/sources
MarketDataGateway still exists, but is greatly simplified. It creates
several `IMarketDataProvider`s, each of which generates a list of
`IMarketDataSource`s. The market data sources provide market data to an
`IMarketDataListener`.

* EmumbaMarketDataProvider
* CCApiMarketDataProvider
* PyMarketDataProvider
* OnixsMarketDataProvider
* ReplayMarketDataProvider

As a side effect, individual py market data sources are now explicitly
limited to a single venue (it probably should have been this way all
along, since the shared memory writers do not expect updates to involve
multiple venues).
2024-10-12 19:30:03 +00:00
John Colvin eea830e9f8 cleanup: Remove superfluous code 2024-10-12 18:38:37 +00:00
John Colvin c5ca6ad712 improvement: Set SHARED_MEMORY_PREFIX to an environment variable
So we can effectively define it at runtime. We don't want to have to
produce different builds just to change the shared mem prefix.
2024-10-12 18:38:37 +00:00
John Colvin fee7b7bc47 fix: Always prepend proc name to shared mem prefix
Our shared memory model isn't really designed to be shared across processes.
Maybe some day it will.
2024-10-12 18:38:37 +00:00
John Colvin c06c768f76 cleanup: Remove redundant macro def 2024-10-12 18:38:37 +00:00
John Colvin 71710ae88c Skip md recording thread loop if no recorders configured to run 2024-10-12 18:38:37 +00:00
John Colvin 4671c12b6f Handle signal interrupts explicitly in sparkd
Otherwise the python interpretter, which is instantiated when we load
python md plugins, prevents the application from exiting.
2024-10-12 18:38:37 +00:00
John Colvin a5489e8fc7 feat: Add support for python md plugins 2024-10-12 18:38:36 +00:00
John Colvin 84432dd67a fix build: Ensure py scripts use unix line endings
I'm not sure why, but my sparkd build (linux) suddenly began failing to
generate cpp headers for protos. The error message indicated it was due
to an unexpected carriage return (\r) in plugin.py:

/usr/bin/env: ‘python\r’: No such file or directory
2024-10-12 18:37:16 +00:00
John Colvin b58be00de0 refactor: Add lib_system proj for common utils
Migrates the following:
* Delegate.h
* bts_array_size.h
* string_utils.h

Note: File names in the new lib are snake_case (this seems to be the
standard naming convention for c++).
2024-10-12 18:37:16 +00:00
John Colvin 427974f9cb cleanup: Trivial whitespace cleanup in sparkd CMakeLists 2024-10-12 18:37:16 +00:00
John Colvin 6ac36da920 build: Add cmake fns to symlink and install local libs
The code needed to do this (inline) is ugly and verbose
2024-10-12 18:37:16 +00:00
John Colvin fe0d1d1af1 cleanup: Use conventional UPPER_CASE var names in sparkd CMakeLists 2024-10-12 18:37:16 +00:00
John Colvin 98ecc530a0 cleanup: Trivial whitespace cleanup in Protobuf.cmake 2024-10-12 18:37:16 +00:00
John Colvin 1b71c62b0e build: Remove explicit targets from sparkd build preset
So it builds the default targets (all). They were only listed explicitly
in the first place so we could build specific targets in vs code using
the dropdown
2024-10-12 18:37:16 +00:00
John Colvin 5753e22968 build: Clean up data_files cmake proj (support --target clean) 2024-10-12 18:37:16 +00:00
John Colvin 5356b0694b refactor: Organize libspark proj w/ stuttering include
This new structure provides several benefits:

* Allows us to install multiple CMake projects in the same destination
  dir (without include path conflicts).
  - This makes shipping the trading algo sdk a trivial excercise. Simply
    copy the lib and include dirs from the install dest. That's it!
* Puts us in a position to break up the monolithic libspark into smaller
  modules, which are easier to manage.
2024-10-12 18:37:13 +00:00
John Colvin 8e59e43c39 vscode: Add debug cfg to skip sparkd build 2024-09-25 06:11:16 +00:00
John Colvin e3e6a6e794 fix build: Add ShmSegmentName.cpp to windows client sources 2024-09-25 06:11:16 +00:00
John Colvin 3eecb0e32b Make a few timestamp fns const
They should have been const in the first place
2024-09-25 06:11:16 +00:00
John Colvin b03790f9da Add config profiles to libspark
This is part of a larger effort to unify the libspark build, and eliminate
compile time differences between the builds used by our various services
2024-09-25 06:11:13 +00:00
John Colvin 2bde71f09b Add api server to docker compose 2024-09-22 18:47:05 +00:00
John Colvin af9c28a512 Rename pingtest "sparkd-ping-test", make it depend on sparkd
Add it to a "ping-test" profile, as we may add similar tests for other services
2024-09-22 18:47:05 +00:00
John Colvin d6c80ca972 cleanup: Remove unnecessary env vars from webapp docker compose 2024-09-22 18:47:05 +00:00
John Colvin bdc439df62 Change api server ports so they don't conflict w/ ds webapp 2024-09-22 18:47:05 +00:00
John Colvin c99345fee0 vscode: Fix sparkd debug config 2024-09-22 18:47:05 +00:00
John Colvin b8b2318f2e build: Resolve app data dir and shared mem prefix at runtime
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
2024-09-22 18:47:04 +00:00
John Colvin 1f85b124fb Don't log "initializing spark config" if it has already been initialized 2024-09-22 18:46:10 +00:00
John Colvin 2b12479c48 vscode: Add debug cfg for spark api server 2024-09-22 18:46:10 +00:00
John Colvin d9b2db778f Use dependency injection in spark api server 2024-09-22 18:46:10 +00:00
John Colvin 2acd911e39 Add SparkLoggerAdapter for Msft ILogger 2024-09-22 18:46:10 +00:00
John Colvin 898e5b5822 Display exception stack trace w/ 1 call per line in ds webapp 2024-09-22 18:46:10 +00:00
John Colvin a3eb3ea310 Remove unused ErrorView from ds webapp 2024-09-22 18:46:10 +00:00
John Colvin 8d8da22003 Propagate ViewStream errors to webapp frontend 2024-09-22 18:46:10 +00:00
John Colvin 30636606a0 Add ErrorChanged event to propagate DataStream exceptions 2024-09-22 18:46:10 +00:00
John Colvin a1caaf9c07 Set columns to empty array rather than null in ds webapp
If no data has been received. Since the 'columns' property was marked
'required', this eliminates a warning that was being logged to console.
2024-09-22 18:46:10 +00:00
John Colvin 9bef5fd30e Improve alignment/spacing in ds webapp 2024-09-22 18:46:10 +00:00
John Colvin 45aa7c1d8f Configure datastream deployment with docker-compose
To deploy to aws, connect to our aws vpn then run:
tools/deploy-docker-compose.sh

The deploy script does the following:
* builds sparkd
* builds datastream apps (pubsub, md-publisher, and webapp)
* uploads the build to aws
* bounces all services (sparkd, pubsub, md-publisher, and webapp)

Note: the name 'md-publisher' refers to DataStreamPublisher w/ MarketData
configuration
2024-09-22 18:46:10 +00:00
John Colvin ae8fa71532 Remove "BTS Cloud Streams" title from webapp
Space is precious when viewing on phone (tbh, it didn't look that good
anyways).
2024-09-22 18:46:10 +00:00
John Colvin ac34840858 fix: Ensure md-publisher waits for pubsub and sparkd
Bugs (fixed):
  * md-publisher silently fails if it tries to connect before those services are "ready".

The Docker compose "depends_on" property ensures sparkd and pubsub
containers are up, but doesn't prevent startup race bugs (e.g., if
md-publisher and pubsub start around the same time, and md-publisher
tries to connect before pubsub has started accepting connections). To
address these race conditions, we use a nifty tool called "dockerize"

References:
  * https://github.com/jwilder/dockerize
2024-09-22 18:46:10 +00:00
John Colvin 568c59fdff Replace ErrorView component w/ robust QueryStatusView 2024-09-22 18:46:10 +00:00
John Colvin 41ebf9df20 Fix typos in query button theme 2024-09-22 18:46:10 +00:00
John Colvin 132b7c4e28 fix: Don't assert on exception in BooleanExpression.evaluate
Bugs (fixed):
  * DataStream WebApp Server sometimes crashes when the user enters a
    bad query.
2024-09-22 18:46:10 +00:00
John Colvin 21a815c78a Change VITE_DATASTREAM_WEBSOCKET_URL to localhost 2024-09-22 18:46:10 +00:00
John Colvin 96199192f6 Assert that local ip address is successfully obtained 2024-09-22 18:46:10 +00:00
John Colvin 5913992348 improvement: Resolve hostnames when comparing ip addrs
So we can to configure pubsub routes by hostname or ipv4 (ipv6 hasn't been
tested yet...so we should assume it doesn't work).
2024-09-22 18:46:10 +00:00
John Colvin 07c09b748b Assert TcpConnection endpoint matches expected 2024-09-22 18:46:10 +00:00
John Colvin e71f36d012 Fix typo in log msg (was missing interpolation '$') 2024-09-22 18:46:10 +00:00
John Colvin c320bd23d6 Catch exceptions on AsyncTCPConnector_AsyncWrapper background thread
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
2024-09-22 18:46:10 +00:00
John Colvin b9ba9b0430 fix: Format PubSubRoute as json in log msgs
It was previously using the default ToString, which simply returns the type name
2024-09-22 18:46:10 +00:00
John Colvin 4eb80820bf build: Upgrade packages (deps) in DataStreamTests 2024-09-22 18:46:10 +00:00
John Colvin f58b40f6cc cleanup: Remove unused ViewStreamDatabase config 2024-09-22 18:46:10 +00:00
John Colvin f941c100b7 vscode: Add PubSubServer debug cfg to skip build 2024-09-22 18:46:10 +00:00
John Colvin 314244c693 Configure apps as a distributed system via Docker compose 2024-09-22 18:46:08 +00:00
John Colvin 6798efd9f5 Remove reference to spark protobuf Config in ViewStream
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
2024-09-22 18:36:36 +00:00
John Colvin d6b6ab08ac refactor: Run RemoteQueryManager as an IHostedService
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2024-09-22 18:36:36 +00:00
John Colvin 13bb4d41a8 Remove inappropriate "Unable to dispose subscription" log msg 2024-09-22 18:36:36 +00:00
John Colvin 2cf72277b3 fix: Ensure spark config init and login are only performed once
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.

It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
2024-09-22 18:36:36 +00:00
John Colvin d81782a1b8 Move misplaced log abt initializing query text to appropriate place 2024-09-22 18:36:36 +00:00
John Colvin 64f5ed4bd6 Assert ViewStreamDatabase.instance is only created once
The instance must be initialized via ViewStreamDatabase.SetGlobalInstance
2024-09-22 18:36:36 +00:00
John Colvin 080b8177b5 cleanup: Remove unused code from ViewStreamDatabase 2024-09-22 18:36:36 +00:00
John Colvin 9ec0a5b8e5 Include subscription id in ds websocket updates 2024-09-22 18:36:36 +00:00
John Colvin aaa9995ad0 fix: Define BoooleanConstant before StringConstant
Otherwise the former masks the latter and prevents it from matching
2024-09-22 18:36:36 +00:00
John Colvin e763d79f26 Log event data in websocket open, close, error callbacks 2024-09-22 18:36:36 +00:00
John Colvin d20580cebb refactor: Bring back Material DataGrid
Fix the column resize bugs that made it previously unusable
2024-09-22 18:36:36 +00:00
John Colvin 727e6a94e6 refactor: Improve react webapp state management
* Add queryTextResetTime as part of the CannedQuery state
  (the QueryControl compares the canned query click time with the time of the
  last user input, and applies the text from the most recent change)

* Replace 'isRunning' (bool) with a 'queryState' (string) to reflect the subscription status
  value received from the server. Less indirection this way.

* Replace inappropriate useMemo call w/ useEffect
2024-09-22 18:36:36 +00:00
John Colvin d73a0de07d cleanup: Get rid of magic const (subscriptionId) in webapp App.jsx 2024-09-22 18:36:36 +00:00
John Colvin f5f9692c33 build: Copy configuration profiles to publish dir 2024-09-22 18:36:36 +00:00
John Colvin 4583a09b70 cleanup: Remove unnecessary deps from csprojs 2024-09-22 18:36:36 +00:00
John Colvin 0236bb9e58 build: Change publish_data_stream.py to publish the sln
We were publishing individual projects as a workaround for the
dependency issue (bug) that caused some projects to incorrectly target
net8.0. The fact that publishing individual projects "just worked" was
probably dumb luck. Now that the underlying issue has been addressed, we
can resume publishing the sln as a whole.
2024-09-22 18:36:36 +00:00
John Colvin b29f32e75d dev: Install docker in devcontainer
So we can start various services like sparkd, pubsub, ds publishers, ds
webapp, etc. from inside the devcontainer.
2024-09-22 18:36:36 +00:00
John Colvin 9d1c00f4d6 dev: Remove dotnet8 from devcontainer
We'll upgrade to dotnet8 eventually, but now isn't the time (too much
more important work to be done).
2024-09-22 18:36:36 +00:00
John Colvin 361da57419 dev: Run devcontainer on host network
So we don't need to worry about publishing the right ports (and don't
need to rebuild the container everytime the list of ports changes).
2024-09-22 18:36:36 +00:00
John Colvin ce8ae14a54 fix: Downgrade Microsoft.Extensions.Hosting to 7.0.1
A handful of csprojs were referencing version 8.0.0. Since that version
targets net8.0, I have no idea why it didn't cause the build to fail.
Instead, this framework mismatch led to a mysterious runtime crash with
the following error, despite the dll in question being present in the
build dir:

Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'System.Diagnostics.DiagnosticSource, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
The system cannot find the file specified.

It is worth noting that this crash would only happen for exes published
as part of the DataStreamSuite.sln with the "--self-contained" flag. For
example, the Bts.DataStreamSuite.Server exe would always fail if
published via:

dotnet publish --configuration Release --sc --framework net7.0 --runtime linux-x64 <DataStreamSuite.sln>
2024-09-22 18:36:36 +00:00
John Colvin 0e7b7c5599 Add HttpsRedirection middleware first in the webapp pipeline
Middleware handlers are invoked in the order in which they are added. Any middleware
that is added before HttpsRedirection may can prevent the request from being redirected.
As of now, only development-specific middleware is added ahead of HttpsRedircetion.
2024-09-22 18:36:36 +00:00
John Colvin 2fecb73e88 build: Copy configuration profiles to build dir
So they are readily available to publish/deploy scripts.
2024-09-22 18:36:36 +00:00
John Colvin b45d93342e Fix benign typo in Scanner/Parser msbuild targets
The output path referenced a non-existant msbuild metadata prop "RelativeOutputDir"
(the actual prop is called "RelativeDir"). Also, the OutputFile path is relative to the
project dir (the 'MSBuildProjectDir' prop), not the file being included.
2024-09-22 18:36:36 +00:00
John Colvin 048834c2c4 fix: Catch exception when creating TcpClient 2024-09-22 18:36:36 +00:00
John Colvin ba6938b09a fix: Catch WebSocketException in DataStreamSubscription 2024-09-22 18:36:36 +00:00
John Colvin 52eaa5622b fix: Add null check to NonIncGroupByStream 2024-09-22 18:36:36 +00:00
John Colvin 012ddeff7e fix: Add null check to TableAccumulator2 2024-09-22 18:36:36 +00:00
John Colvin 727235950f fix: Add self-signed certificate to Dockerfile for ds webapp 2024-09-22 18:36:36 +00:00
John Colvin 8b779597db fix: Make SystemSubscriptionMapPublisher threadsafe
The PubSubState.onTagMapChanged event may not be invoked on
the same thread as doWork.
2024-09-22 18:36:36 +00:00
John Colvin 5eecd2ffbc fix: Add null check in TimeBasedFilter.cs 2024-09-22 18:36:36 +00:00
John Colvin b12c28a074 Reduce task wait/timeout code in AsyncTCPConnector 2024-09-22 18:36:36 +00:00
John Colvin 02d19c522d Add a few canned queries to AwsProd.json for ds webapp 2024-09-22 18:36:36 +00:00
John Colvin a00e849192 fix: Copy canned queries to clipboard on click
Instead of trying to update the text directly, which causes the front end
to overwrite the user's changes everytime the data updates from then on.
2024-09-22 18:36:36 +00:00
John Colvin d0601ad32d fix: Convert invalid doubles to nan in ds webapp serializer 2024-09-22 18:36:36 +00:00
John Colvin 149ab49212 Increase datagrid maxheight in ds webapp to 1000px 2024-09-22 18:36:36 +00:00
John Colvin 23c9f0a9de Improve error display in ds webapp 2024-09-22 18:36:36 +00:00
John Colvin 22b04c8ec8 fix: Dispose existing sub on subscribe req in ds webapp
Even if new sub has parse error. This fixes a bug where the previous
subscription would continue firing and updating the frontend, when it
should just display the error info to the user and cease updates.
2024-09-22 18:36:36 +00:00
John Colvin fdba4601ab fix: Get rid of "UseSpa" calls in webapp Production environment
This was leftover from a misguided attempt to fix a bug that turned out
to be due to an omitted environment var:
ASPNETCORE_HOSTINGSTARTUPASSEMBLIES=Microsoft.AspNetCore.SpaProxy
2024-09-22 18:36:35 +00:00
John Colvin dd2dffd337 Define webapp specific env vars in generic datastream app deploy script
For now, this makes things easier, as I don't need to write a special script just for the webapp
2024-09-22 18:36:35 +00:00
John Colvin 2c58415a4f Add config profile and Dockerfile args to deploy script 2024-09-22 18:36:35 +00:00
John Colvin f85f57d2a5 Copy DataStream WebApp 'dist' to 'wwwroot' in deploy script 2024-09-22 18:36:35 +00:00
John Colvin 4d5f87df33 Tweak DataGrid style in DataStream WebApp 2024-09-22 18:36:35 +00:00
John Colvin bc22d0f2e7 Add AwsProd config for DataStream WebApp 2024-09-22 18:36:35 +00:00
John Colvin 2020c93915 fix: Add VITE_ prefix to imported env vars
For the DataStream WebApp client
2024-09-22 18:36:35 +00:00
John Colvin e92c4a7089 Fix urls for DataStreamPublisher AwsProd.json 2024-09-22 18:36:35 +00:00
John Colvin 234d1c7b31 Add benzinga news streams to AwsProd.json
This is a config profile for DataStreamPublisher, designed to be deployed
to aws (on public internet).
2024-09-22 18:36:35 +00:00
John Colvin ec0d0a5ec4 fix build warnings: Make props nullable strings in SharedStreamConfig 2024-09-22 18:36:35 +00:00
John Colvin ad2671e199 whitespace: Format PubSubServer.json (indentation issues) 2024-09-22 18:36:35 +00:00
John Colvin 8e42651433 fix import from environment variable 2024-09-22 18:36:35 +00:00
John Colvin daee227d5b Add a few tasks to run DataStream WebApp 2024-09-22 18:36:35 +00:00
John Colvin 17b2711357 fix: Import websocket url from environment var 2024-09-22 18:36:35 +00:00
John Colvin 42022bad0b Add scripts to deploy data stream apps 2024-09-22 18:36:35 +00:00
John Colvin 15b8a2189b Add SpaServer to Production environment
For Bts.DataStreamWebApp.Server
2024-09-22 18:33:42 +00:00
John Colvin 64ce742e2a Change DirectoryBrowser requestPath to "/browse"
So we can view a psuedo file explorer for the webapp by requesting the "/browse" path
(only activated in Development environment).
2024-09-22 18:33:42 +00:00
John Colvin 38d2145a78 Publish Bts.DataStreamWebApp.Server along with PubSub & DataStream 2024-09-22 18:33:42 +00:00
John Colvin 171d786a49 cleanup: Remove unused code in Bts.DataStreamWebApp.Client 2024-09-22 18:33:42 +00:00
John Colvin 45ae320324 Capture OperationCanceledException (instead of TaskCanceledException)
The former is the base class for the latter, so this will widen the umbrella to catch a few corner cases
that would otherwise been handled by the generic "Exception" catch block. One corner case I know
of is when an op is canceled via "CancellationToken.ThrowIfCancellationRequested". In that case,
an OperationCanceledException is thrown (not a TaskCanceledException).

https://stackoverflow.com/questions/34359129/operationcanceledexception-vs-taskcanceledexception-when-task-is-canceled
2024-09-22 18:33:42 +00:00
John Colvin fc16ed7194 Hardcode webSocketUrl to aws server in DataStreamWebApp client 2024-09-22 18:33:42 +00:00
John Colvin f4584afdf0 Assert that System publishers 'doWork' is always called on same thread 2024-09-22 18:33:42 +00:00
John Colvin 4fc0eec13c fix: Make InternalPublisherQueue thread safe 2024-09-22 18:33:42 +00:00
John Colvin c5e851d0aa Improve error reporting in DataStreamWebSocketServer 2024-09-22 18:33:42 +00:00
John Colvin 402cac2d03 data stream web app client check pt 2024-09-22 18:33:42 +00:00
John Colvin 79673b7a01 fix build: Fix Spark build after making config init async 2024-09-22 18:33:42 +00:00
John Colvin 2cf7826e4d fix Change Configuration.initialize to be async 2024-09-22 18:33:42 +00:00
John Colvin 8bb67dfba2 refactor: Tweak SparkCoreServices to synchronize Configuration.init globally 2024-09-22 18:33:42 +00:00
John Colvin f922215cae refactor: Change Configuration.initialize to be async 2024-09-22 18:33:42 +00:00
John Colvin d9e804b4e1 vscode: Add 'isSparkApiEnabled' input to launch.json 2024-09-22 18:33:42 +00:00
John Colvin 8194a07a8e SparkApiService check pt 2024-09-22 18:33:42 +00:00
John Colvin d8e4c6e601 pubsub server checkpt 2024-09-22 18:33:42 +00:00
John Colvin a8d3528125 pubsub checkpt 2024-09-22 18:33:42 +00:00
John Colvin 655329fec8 datastreamwebapp server checkpt 2024-09-22 18:33:41 +00:00
John Colvin 7d7c868f87 datastream webapp client check pt 2024-09-22 18:33:41 +00:00
John Colvin c14bade720 fix add pubsub services 2024-09-22 18:33:41 +00:00
John Colvin 104de5df7b Remove unused PubSubConfig props 2024-09-22 18:33:41 +00:00
John Colvin bd52ef0a8e Fix dsqa config json 2024-09-19 20:03:31 +00:00
John Colvin a015ff98e4 fix: Always call 'handle_internal_pubs' in PubSubMessageReactor main loop 2024-09-19 20:03:31 +00:00
John Colvin 12fc295825 improvement: Accept CancellationToken, catch exceptions when running ds websocket server 2024-09-19 20:03:31 +00:00
John Colvin 587e853f21 Fix mismatched types in PubSubServerTagStatsStream schema 2024-09-19 20:03:31 +00:00
John Colvin 909c8f5bcd Add DataStreamPublisher service modules designed for Dependency Injection
* Change DataStreamPublisher json fields to match cs class property names verbatim
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
2024-09-19 20:03:31 +00:00
John Colvin 448d49f38d refactor: Decouple ViewPublisher from IPubTableBroker impl 2024-09-19 20:03:31 +00:00
John Colvin f67c24f7f5 Add PubSub service modules designed for Dependency Injection
* Get rid of JsonPropertyName attributes in PubSubConfig
  - The Dependency Injection framework (based around dotnet ApplicationBuilder)
    does not support this attribute, and maps property names verbatim to json fields.

* Change PubSubServer json fields to match cs class property names verbatim
2024-09-19 20:03:31 +00:00
John Colvin 69a72a1035 refactor: Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl 2024-09-19 20:03:31 +00:00
John Colvin 819a8f6c22 refactor: Decouple ManagedSubscriber from ISubscriberCore impl 2024-09-19 20:03:31 +00:00
John Colvin 42db20316b refactor: Only support 1 tag type in IPubSubServerTagHandler 2024-09-19 20:03:31 +00:00
John Colvin 633a8590a1 refactor: Decouple PubTableBroker from IPublisherCore impl
Reference interface rather than the PublisherCore impl.
2024-09-19 20:03:31 +00:00
John Colvin eff979fb2b improvement: Accept CancellationToken in JsonRpcRequestObservable 2024-09-19 20:03:31 +00:00
John Colvin 23771909fd fix: Dispose market data sub in various md streams 2024-09-19 20:03:31 +00:00
John Colvin 9de21a3116 cleanup: Remove unused pubsub tag handler code
* Removed 'IPubSubServerTagHandler.register'
  - All implementations of this function were empty
* Removed 'IPubSubServerTagHandler.supportedTagTypes'
  - This property wasn't referenced anywhere
* Remove IWorker inheritance from IPubSubServerTagHandler
  - No reason these interfaces have to be tied together
2024-09-19 20:03:31 +00:00
John Colvin 8d231ce46c refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
This matches the file name
2024-09-19 20:03:31 +00:00
John Colvin fc9d5ab04f Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
In config json files (not actual code change)
2024-09-19 20:03:29 +00:00
John Colvin 3562f6848e fix build: Fix prop names after refactoring PubSubRoute
* Change 'server_address' -> 'ServerAddress'
* Change 'tag_reg_exp' -> 'TagRegExp'
2024-09-19 20:02:09 +00:00
John Colvin 1511bcf252 build: Add scripts to publish data stream apps 2024-09-19 20:02:09 +00:00
John Colvin b4c902c023 Fix typo in log msg 2024-09-19 20:02:09 +00:00
John Colvin 438ea44c88 Fix build scripts so they propagate the exit code back to the shell
I noticed this was broken because launch.json configurations would still start the debugger
even when the preLaunchTask (build) failed!
2024-09-19 20:02:09 +00:00
John Colvin 75acf12ef3 vscode: Add DataStreamPublisher debug cfg to skip build 2024-09-19 20:02:09 +00:00
John Colvin be407e3e1e build: Use separate cmd to build DataStreamSuite/QueryAnalyzer.sln
The DataStreamSuite.sln builds the new webapp now, which will
(eventually) replace DSQA. We'll keep it around for now, but with a
clean separation from the main build.
2024-09-19 20:02:09 +00:00
John Colvin 2dfa44c1e4 feat: Add DataStream WebApp
Included in DataStreamSuite.sln, but not DataStreamQueryAnalyzer.sln
2024-09-19 20:02:09 +00:00
John Colvin 022cad3122 Add spark service modules designed for Dependency Injection 2024-09-19 20:02:09 +00:00
John Colvin 5c42b4fe8b build: Copy libspark to all dotnet dirs in single cmake target
This is cleaner than the previous impl, which used separate build
targets to copy libspark to "net7.0", and "net7.0-windows" (dirs).
2024-09-19 20:02:09 +00:00
John Colvin 9f81c0ab39 fix: Parse ServerAddress as normal string
The "JsonValueOrArrayConverter<T>" doesn't work the way I thought it
would. When it is associated with a property via the JsonConverter
attribute, dotnet json parsing code throws an exception if "T" isn't
assignable to the property type.
2024-09-19 20:02:09 +00:00
John Colvin c941fa78d2 fix build (api server): Change namespace to bts.websocket_core 2024-09-19 20:02:09 +00:00
John Colvin e6f2c8d1a5 Remove docker submodules again 2024-09-19 20:02:09 +00:00
John Colvin faf296254e Change externals url from ssh to https style
So we don't need ssh keys to clone it (the builtin credential manager
makes https authentication safe/easy)
2024-09-19 20:02:09 +00:00
John Colvin 41712c774f cleanup: Fix benign typos in build scripts
Change BUILD_SPARK_PY var name to BUILD_PY (because these scripts to actually build 'spark')
2024-09-19 20:02:09 +00:00
John Colvin 565c70bb92 fix build (warn): Only use Registry on windows 2024-09-19 20:02:09 +00:00
John Colvin 97bc186110 cleanup: Fix numerous build warnings in DataStreamSuite
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.

* PubSubClientMessage
  - Normalize indentation/whitespace
  - Fix some typos
  - Address nullability issues to fix build warnings
  - Modernize code

* Compression related files
  - ZipEntry
  - ZipFile
  - ZipInputStream
  - TarHeader

* RemoteMessage
  - make 'message' nullable to resolve warnings elsewhere (in calling code)

* PubSubSingleTagStateEvent.cs
* PubSubMessageReactor
* PubSubState
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SingleTagData
* ManagedTCPMessageHub
* PubsubDeserializationController
* PublisherCore
* InflaterHuffmanTree
* AsyncTCPConnection_APM
* ThreadSafeQueue
* ManagedSubscriber
* SubscriberCore
* PubSubServerManagedTCPConnection
* AsyncTCPConnection_APM
* SuperMapStream
* MapStream
* TradeBarsStream
2024-09-19 20:02:09 +00:00
John Colvin 158f4fb6c8 cleanup: Normalize indentation in TradeBarsStream 2024-09-19 20:02:09 +00:00
John Colvin c7961f53c0 cleanup: Disable nullability warnings
* BZip2InputStream
* TarInputStream
* Inflater
* InflaterDynHeader
* BZip2OutputStream
* DeflaterHuffman
* ParallelJoinTest
2024-09-19 20:02:09 +00:00
John Colvin 6e89f42bc8 cleanup: Modernize PubSubRoute code, json parsing 2024-09-19 20:02:09 +00:00
John Colvin 0ee450b12a cleanup: Remove unused DateTime grammar from exp parser 2024-09-19 20:02:09 +00:00
John Colvin b2e3a6bf82 Add JsonRpcRequestObservable, AsyncEnumerable 2024-09-19 20:02:09 +00:00
John Colvin f87aa6c4b6 Add json facilities for DataStream websocket api 2024-09-19 20:02:09 +00:00
John Colvin 40a46dd6d7 refactor: Add handy WebsocketExtensions
* IsSocketStateOpen
* IsClosedProperly
2024-09-19 20:02:09 +00:00
John Colvin 75d04b2cd2 feat: Add JsonRpc types for websocket paradigm 2024-09-19 20:02:09 +00:00
John Colvin 423e20ae41 cleanup: Remove empty file (clearly added by accident) 2024-09-19 20:02:09 +00:00
John Colvin de439579cf fix: Fix improper 'OfType' linq cast
Previously, animations were expressed through polymorphism (derived from the
base class CellAnimation), but the new impl uses composition. This 'OfType' call
was leftover from the polymorphism implementation, an attempt to linq cast
elements of base class (CellAnimation) to BackgroundAnimation.
2024-09-19 20:02:09 +00:00
John Colvin adbe72258b editorconfig: Use standard c# capitalization rules
* Use 'PascalCase' for public members (instead of 'snake_case')
* Use 'camelCase' for private members (instead of '_snake_case')
2024-09-19 20:02:09 +00:00
John Colvin 13646a5a16 refactor: Add WebsocketClient class
Basically an ISubject (I/O) wrapper for IWebsocketConnectionGateway
2024-09-19 20:02:09 +00:00
John Colvin 9a1e51eacf refactor: Change WebsocketServerConfig.port to uint32
It should always be a number between 1-65535
2024-09-19 20:02:09 +00:00
John Colvin 9982bd2203 refactor: Rename WebsocketServerConfig proto pkg 'bts.websocket_core'
This matches the namespace used in spark_websocket csproj. Eventually,
I'd like to use conventional capitalization (TitleCase namespaces/types,
etc.), but let's shelf that until we have enough time/energy to do a
more comprehensive refactoring.
2024-09-19 20:02:09 +00:00
John Colvin e7d7352941 refactor: Remove nullability constraint in GetValueOrDefault 2024-09-19 20:02:09 +00:00
John Colvin d2efe037f9 refactor: Use events to avoid circular deps for ws sync/gateway 2024-09-19 20:02:09 +00:00
John Colvin 67e6147633 refactor: Let caller specify nullability of T in json parse fn 2024-09-19 20:02:09 +00:00
John Colvin 37e5003cf9 Merge remote-tracking branch 'origin/development.ps_linux' into development 2024-09-19 05:40:59 +00:00
John Colvin 07f3868ffc Add CBOE to spark admin webapp 2024-09-18 19:03:43 +00:00
John Colvin 46f0b17138 Add TAIFEX to spark admin webapp 2024-09-18 19:03:43 +00:00
Taha 33a8e06a3a Fixing implied vols for SPX (market data was getting divide dby the fwd) 2024-09-17 22:17:31 -04:00
John Colvin 74afdde8de fix build: Add ".list" to SymbolLists config
For new exchanges, TAIFEX and CBOE
2024-09-17 13:01:05 +00:00
John Colvin 0a92ad2eb1 fix (build): Add break statement to new venue switch cases
Namely, TAIFEX and CBOE
2024-09-17 13:00:14 +00:00
John Colvin 56d207c3f1 fix: Preserve vs exp sec type (Option) when computing implied vol
Because the exp keys in the data map have sec type Option.
2024-09-16 13:46:53 -04:00
John Colvin 9172e752e3 ui: Highlight launcher buttons when selected
Namely, the expandable menu buttons in Analytics shell module
2024-09-16 13:46:53 -04:00
John Colvin adc9f2acfc Replace trivial refs to ThreadedBlackboard w/ IBus 2024-09-15 23:45:15 -04:00
John Colvin 41d17e67d2 fix: Move accounts subscription to dispatcher thread
We update a collection that is bound to the view in the callback (mutation
ops throw if performed on a different thread).
2024-09-15 23:45:15 -04:00
John Colvin 5cacff073b fix: Check 'HasTimestamp' before accessing prop in man fill row
Pending adds (i.e., unpersisted new rows) don't have timestamps.
In such cases, the 'Timestamp' getter throws an exception.
2024-09-15 23:45:15 -04:00
John Colvin c8da92c250 Cache/index options for faster lookups in SystemContractData 2024-09-15 23:45:15 -04:00
John Colvin c4e76076e3 fix: Include venue w/ symbol when publishing interp greeks 2024-09-15 23:45:14 -04:00
John Colvin 6a715b9a68 Fix typo in comment 2024-09-15 23:45:11 -04:00
John Colvin 1f28940c95 fix: SelectorList.RemoveAt doesn't need 'convert_back' delegate
So don't throw an exception if it is null.
2024-09-15 23:45:11 -04:00
John Colvin ebff40b88f fix: Don't assume venue is in spread def string
Pass it as an argument to the parse fn. This should be fine for now, but
obviously a singular venue param won't work for cross exchange spreads.
2024-09-15 23:45:11 -04:00
John Colvin a6d14a3ac3 improvement: Move system contracts sub off of dispatcher thread
Also add 200ms throttle. Contracts shouldn't change that often, so we don't need
lightening speed here. Furthermore, such changes may trigger expensive cpu ops.

Note: this change also replaces refs to ThreadedBlackboard w/ IBus (the
former is somewhat tethered to the dispatcher thread, particularly by
its timer_callback).
2024-09-15 23:45:11 -04:00
John Colvin af96bcb9e3 fix: Pass symbol/venue from vs usage cfg to AnalyticsService proto 2024-09-15 23:28:59 -04:00
John Colvin 619702cb04 cleanup: Remove unused code 2024-09-15 23:28:59 -04:00
John Colvin 5baec19aff refactor: Index vol and theos data by symbol/venue tuple
Rather than just symbol, to avoid conflicts when the same symbol exists
at multiple exchanges
2024-09-13 10:50:23 -04:00
John Colvin 39ef30c1e3 cleanup: Remove redundant type VolSurfaceSymbolStructure
There is an identical type (in terms of prop types and corresponding proto indices)
called VolSurfaceSymbolRelationship that is used in more places.
2024-09-13 10:50:23 -04:00
John Colvin c2f2d28914 Move nested vm classes into own files
VolSurfaceSettingsEditor has too much code (still). It is hard to
navigate.
2024-09-13 10:50:23 -04:00
John Colvin e06bc3bdf5 Simplify driver symbol structure in vs settings vm
The code was confusing and difficult to follow.
2024-09-13 10:50:23 -04:00
John Colvin e19c19c013 fix: Raise prop changed event when is_selected changes 2024-09-13 10:50:23 -04:00
John Colvin 8f324bc507 Add venue to vs calc fns 2024-09-13 10:50:23 -04:00
John Colvin 12107c42ce fix: Provide venue param w/ symbol when computing fut slide 2024-09-13 10:50:23 -04:00
John Colvin 91c6fb4e52 diag: Log added/removed exps when syncing vs input w/ system exps
This shouldn't happen often, but if it does, we should know about it.

PS: We recently fixed a bug caused by inequalities between
VolSurfaceExpirations in vs input, and in SystemContractData. User
inputs to the vs grid would be immediately reverted, as the exps
associated with the inputs were not found in the SystemContractData.
This bug would have been easier to track down with this logging in
place.
2024-09-13 10:50:23 -04:00
John Colvin 7044355547 Add venue to VolSurface(DisplayGrid)Expiration
So we can handle symbol/exp date conflicts for securities that differ
only by venue (exchange)
2024-09-13 10:50:23 -04:00
John Colvin 583ae65fb2 Clean up VolSurfaceExpiration
* Enable nullable lang feat
* Use file-scoped namespace
* Sort members
* Include sec type in Compare impl
2024-09-13 10:50:22 -04:00
John Colvin 552fec0edc Add null check in SelectorList ctr 2024-09-13 10:50:22 -04:00
John Colvin fcf6927fbf cleanup: Remove unused code 2024-09-13 10:50:22 -04:00
John Colvin 23f6d827d5 refactor: Replace ThreadedBlackboard w/ IBus
In SingleVolSurfaceCommittedOutputRetriever
2024-09-13 10:50:22 -04:00
John Colvin f083d0abab Clean up VolSurfaceTimeParameters
* Enable nullable lang feat
* Use file-scoped namespace
* Implement ICloneable
2024-09-13 10:50:22 -04:00
John Colvin a895e36922 cleanup: Remove unnecessary ternary op 2024-09-13 10:50:22 -04:00
John Colvin 59d7a72288 fix: Actually use (sec) type arg in to_security_expiration 2024-09-13 10:50:22 -04:00
John Colvin 7731f2f4d1 fix: Check security_type in vs exp Equals impl 2024-09-13 10:50:22 -04:00
John Colvin 41ae5897c0 fix: Maintain sec type when converting SecurityExpiration to vs exp
Avoids data loss
2024-09-13 10:50:22 -04:00
John Colvin c7949ccfba refactor: Get rid of sec type 'Option' default (magic const)
Better to be explict
2024-09-13 10:50:22 -04:00
John Colvin 567af79c7f Fix selection binding issues in vs settings editor
This specifically addresses issues in the driver symbol grids.
Unfortunately, we can't bind directly to ComboCheckBox.SelectedItems
because the view is contained in TabControl. When the selected tab
changes, the previously selected tab clears its ItemsSource (and
consequently the SelectedItems) as a side effect of being unloaded.
2024-09-13 10:50:22 -04:00
John Colvin 6bb4a7b25c refactor: Enable nullable (ref types) in ImmutableListExtensions 2024-09-13 10:50:22 -04:00
John Colvin bc7cef4c34 Fix typo in namespace bts.collection -> bts.collections 2024-09-13 10:50:22 -04:00
John Colvin ab9dc361d6 fix: Use ComboCheckBoxItem.IsSelectedProperty
Instead of Selector.IsSelectedProperty. Unlike ComboBoxItem, ComboCheckItem's
IsSelectedProperty isn't linked to Selector.IsSelectedProperty.
2024-09-13 10:50:21 -04:00
John Colvin 39da0a4dde Add currency column to mview 2024-09-13 10:50:21 -04:00
John Colvin 44e864d697 fix: Use SetCurrentValue instead of SetValue
So we propagate values to/from the binding data source,
rather than overwrite the data source itself.
2024-09-13 10:50:21 -04:00
John Colvin 0a02da3aba Add venue, sec type dropdown filters to exp params ui 2024-09-13 10:50:21 -04:00
John Colvin 836771a7f0 fix: Add null check in ComboCheckBox.SelectedItemsCollectionChanged 2024-09-13 10:50:21 -04:00
John Colvin f386badcfb fix: Disconnect old INotifyCollectionChanged source when dp changes 2024-09-13 10:50:21 -04:00
John Colvin 1d43a484db fix: Index opt/fut exps by venue in vs analytics 2024-09-13 10:50:21 -04:00
John Colvin 91042cb41e fix: Support multiple exchanges in exp params view/vm 2024-09-13 10:50:21 -04:00
John Colvin 10c0e4247d cleanup: Remove unused 'option_symbol' param from compute_atm_data 2024-09-13 10:50:21 -04:00
John Colvin 8e34264e6d fix: Filter by exch when getting atm data for sheets
This is necessary now that we support exchanges with overlapping securities
2024-09-13 10:50:21 -04:00
John Colvin 0a840d256b fix: Remove sec id prop from SecurityExpiration
This class isn't designed to represent a particular contract; it's
basically a key to get option theos.
2024-09-13 10:50:21 -04:00
John Colvin 8a69c51cac refactor: Add 'onDispatcher' param to 'ThreadedBlackboard.subscribe'
This will allow us to gradually move subscriptions off of the dispatcher
(render) thread. When the render thread is pegged, the ui is laggy and
unresponsive. The new param defaults to true, so it won't change
anything until we explicitly set it to false in 'subscribe' calls.
2024-09-13 10:50:21 -04:00
John Colvin 0262fd40bc fix: Remove redundant SparkTimer threads
This addresses some slowness on the front end. Prior to this commit,
each ThreadedBlackboard would start its own timer thread. Since multiple
ThreadedBlackboard run concurrently, this multiplied the frequency of
the SparkTimer callbacks by the number of active ThreadedBlackboards.

The fix is to start this thread in an IShellDaemon rather than the
ThreadedBlackboard. This avoids duplicate threads because the former is
guarenteed to be instantiated only once, via dependency injection.
2024-09-13 10:50:21 -04:00
John Colvin 859dfe3544 refactor: Simplify FakeMDDaemon code that delays post
Use Task.Delay/ContinueWith instead of subscribing to the SparkTimer tag.
2024-09-13 10:50:21 -04:00
John Colvin 9c624a2f79 refactor: Use IBus instead of ThreadedBlackboard
This should address some slowness/mouse lag in the front end (recently
reported by Brandon at Immutable Cap)
2024-09-13 10:50:20 -04:00
John Colvin dec1061b40 fix: Detect and log ccapi depth msg parse errors
This fixes a crash that would happen if a depth item's price or quantity
were an empty string (now we log an error and keep going).
2024-09-13 10:50:20 -04:00
John Colvin be4895faa6 fix: Actually store currency in shared mem
This field has always been in the Security proto, though it until now it
had been omitted from the shared memory representation (FixedSecurity).
2024-09-13 10:50:20 -04:00
Mohammad Arham 3c9755f0dd ADD: Added delay to databento file reader 2024-09-12 15:10:34 +00:00
Mohammad Arham 61ed847d7d Add: Replay speed added to mdreplayer 2024-09-11 06:50:37 +00:00
Mohammad Arham fc9094fea8 Added: CBOE working on databento 2024-09-11 06:17:26 +00:00
Mohammad Arham e36c0efc4b Added: GLBX working on databento 2024-09-11 06:16:42 +00:00
Mohammad Arham 4158a14cff Added: option to send timestamp in onbook IMDListener 2024-09-11 06:15:40 +00:00
Mohammad Arham 201a45cba0 taifex deo 2024-09-05 00:59:00 +00:00
Mohammad Arham ab31b9b253 Add: Multiple files working taifex 2024-09-03 15:15:55 +00:00
Adeel Khan 780ddc79a7 Add: Taifex I083 Message 2024-09-03 20:06:48 +05:00
Mohammad Arham d5a2adaed6 fix: removed errernous code 2024-09-03 13:09:54 +00:00
Mohammad Arham 008ab3f068 Merge branch 'development.new_exchanges' of bitbucket.org:bluetradesys/spark into development.new_exchanges 2024-09-02 16:31:14 -04:00
Mohammad Arham 7d926f588d FIX: typo, changed date to day 2024-09-02 16:31:10 -04:00
Mohammad Arham 707cd07876 Add: CBOE L1 books and contracts working 2024-08-28 16:22:51 +00:00
Mohammad Arham f7ecb9090c add: code to start working on cboe 2024-08-27 12:33:23 +00:00
Mohammad Arham a19e8a7315 change: changes to make taifex work 2024-08-27 07:45:59 -04:00
John Colvin cf189ecf81 refactor: Start cboe and taifex md 'start_databento'
Similar to how we start all ccapi implemented exchanges in "_start_cc"
(in MarketDataGateway)
2024-08-26 19:13:59 +00:00
John Colvin 4405b7adcb refactor: Rename cc_md_listeners -> venue_listeners
They are general purpose, not specific to ccapi.
2024-08-26 19:09:57 +00:00
John Colvin e51ee7b7b9 refactor: Wrap IDGenerator and IMarketDataListeners in shared_ptrs
Our various market data providers (ccapi, emumba, replay, onixs)
dereference pointers to common instances of these types concurrently on
different threads. Although we've correctly managed memory ownership
thus far, we really should be using smart pointers (this is why they
were made).
2024-08-26 19:03:29 +00:00
John Colvin dfaa74f426 build: Add databento dep via FetchContent
Now we actually build libdatabento in the devcontainer along with our
code (as we do with the vcpkg deps, though this is not part of vcpkg)
2024-08-26 17:13:51 +00:00
Mohammad Arham 1d2ce59009 add: added taifex entries 2024-08-22 23:13:05 +00:00
Adeel Khan 3602cc05bb added: TaifexDataHandler, temp 2024-08-23 02:06:47 +05:00
Adeel Khan ac88a81ac6 Security Extensions 2024-08-23 01:42:24 +05:00
Adeel Khan 01fa8e0c63 added: files for Taifex, Cboe 2024-08-23 00:57:40 +05:00
Taha 322c06bfbe Fixing build relted to BCG vs bts namespace 2024-08-21 16:24:22 -04:00
Taha b68fceb362 Cleaning up FileStreams 2024-08-21 16:11:57 -04:00
John Colvin 59547e0f37 fix build: Lock fmt version to 10.2.1 in vcpkg
This fixes the front-end (Windows) libspark build
2024-08-21 13:45:43 -04:00
John Colvin 3ce100a9cf build: Add databento to sparkd
* Install databento deps in docker
  - libzstd-dev
* Add databento cmake module to externals
2024-08-21 13:49:53 +00:00
John Colvin 6513fb0cd5 cleanup: Remove unused include 'date.h' from MDSource
This was an external header, and may interfere with a dependency in the
databento lib (which we are about to add).
2024-08-21 13:49:24 +00:00
Taha Afzal fd0fb1f760 Adding incremental book updates to IMarketDataListener 2024-08-12 22:49:27 -04:00
Taha Afzal 5db586c4d3 spaces, newlines? 2024-08-12 21:27:43 -04:00
Taha 31df1a3827 Added benzinga streamd to MDPub's test config 2024-07-17 14:32:57 -04:00
Taha 7e219aec7c Merging 2024-07-17 02:06:17 -04:00
Taha 37f8b98df9 Icomplete SharedStreamsConfig was breaking build, fixing 2024-07-17 02:03:43 -04:00
Taha f06beff22f Added new benzinga streams 2024-07-17 02:02:01 -04:00
Taha Afzal 6aa39a716a PubSubSingleTagState is not needed any more 2024-07-16 01:15:51 +00:00
Taha Afzal 5088b5ebe0 Fixing problems where PSS's client socket's IP was different than what the client was sending in PubSubClientSubscription 2024-07-08 23:09:53 +00:00
Taha Afzal 010dd3a26f Fixing PubsubEchoTestStream 2024-07-07 06:13:22 +00:00
Taha Afzal 1a18755176 Added pss_echo() 2024-07-06 06:53:35 +00:00
Taha Afzal 233e2576af Merging 2024-07-05 21:21:27 +00:00
Taha Afzal 543c655bca src/DataStreamSuite/PubSub/ManagedTCP/AsyncTCPConnection_AsyncWrapper.cs 2024-07-05 21:13:36 +00:00
Taha 050e21e22b Adding TableMetaDataStream 2024-07-04 23:39:04 -04:00
Taha Afzal 672ad6d237 Cleaning Config 1.0 (not completely done) 2024-06-30 23:17:20 -04:00
Taha 0a8f6e866f Added pubsub routes to the Title of DSQA 2024-06-27 14:04:07 -04:00
Taha b177ffef04 Fixing the bug where messages were being sent by the pubsubserver out of order 2024-06-27 00:17:22 -04:00
Taha 393f304be4 some buf gixes + cleanup 2024-06-26 00:23:19 -04:00
Taha 151816bf5e PubSuMessageReactor empties the message queue (instead oif one elem ata a time) in the main loop 2024-06-24 02:29:28 -04:00
Taha 5cf7eacd82 Fixing the extra .0 in the default pubsub server routeplusfixing acorner cae issue of gettign a fast streaming data cakback after an unsubscribe 2024-06-24 00:28:07 -04:00
Taha 87b6a2044c PubSubServer's linux issues fixed 2024-06-23 21:13:37 -04:00
John Colvin 087d8ab8c9 Set text=auto in gitattributes
Otherwise git can cause files to show as "permanently modified" when you
apply changes from the index (e.g., by running 'git rebase', or 'git
stash pop')
2024-06-11 17:39:12 +00:00
John Colvin b69b7b3350 vscode: Use '\' dir separator in show_last_run.cmd
This script only runs on Windows
2024-06-08 15:08:25 -05:00
John Colvin 5620f33524 vscode: Add task to view logs for last PubSubServer run 2024-06-08 14:42:13 +00:00
John Colvin f3e33fa390 vscode: Add task to run PubSubServer without building 2024-06-07 16:41:42 +00:00
John Colvin 3b631e0d5f Fix a couple more typos in logs 2024-06-07 16:35:06 +00:00
2453 changed files with 52139 additions and 26858 deletions

No files matched your search

+11
View File
@@ -87,6 +87,15 @@ RUN dpkg -i /emumba/emumba-deribit-connector-3.3.1-Linux.LICENSE_DISABLED.deb \
# Pcap
RUN apt-get update && apt-get install -y libpcap0.8-dev
# Databento (deps)
RUN apt-get update && apt-get -y install libzstd-dev
# Docker in Docker
RUN apt-get update && apt-get install -y \
docker.io \
docker-compose \
docker-compose-v2
# Set up default executables (update-alternatives)
RUN \
update-alternatives --install /usr/bin/clang clang /usr/bin/clang-$clang_major 100; \
@@ -120,3 +129,5 @@ RUN \
-G sudo \
--password "$(openssl passwd -1 "$docker_pass")" \
"$docker_user"
RUN usermod -aG docker "$docker_user"
+5 -10
View File
@@ -6,7 +6,7 @@
"memory": "4gb",
"storage": "6gb"
},
// "postCreateCommand": "/home/bts/spark/docker/sparkd/dev/devcontainer_post_create.sh",
"postCreateCommand": "/home/bts/spark/docker/sparkd/dev/devcontainer_post_create.sh",
"containerUser": "bts",
"workspaceMount": "source=${localWorkspaceFolder},target=/home/bts/spark,type=bind",
"workspaceFolder": "/home/bts/spark",
@@ -18,17 +18,12 @@
"runArgs": [
"--shm-size",
"2gb",
"-p",
"8000:8000",
"-p",
"8001:8001",
"-p",
"8002:8002",
"-p",
"8003:8003"
"--network",
"host"
],
"mounts": [
"source=${localEnv:HOME}/.bts/devcontainer-mounts/${devcontainerId},target=/home/bts,type=bind"
"source=${localEnv:HOME}/.bts/devcontainer-mounts/${devcontainerId},target=/home/bts,type=bind",
"source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind"
],
"customizations": {
"vscode": {
+30 -16
View File
@@ -207,17 +207,23 @@ dotnet_naming_symbols.parameters.applicable_accessibilities = public,internal,pr
dotnet_naming_symbols.private_symbols.applicable_kinds = property,method,field,event,delegate
dotnet_naming_symbols.private_symbols.applicable_accessibilities = protected,private
# local symbols
dotnet_naming_symbols.local_symbols.applicable_kinds = local
# parameter symbols
dotnet_naming_symbols.parameter_symbols.applicable_kinds = parameter
# Naming styles
dotnet_naming_style.snake_case.required_prefix =
dotnet_naming_style.snake_case.required_suffix =
dotnet_naming_style.snake_case.word_separator = _
dotnet_naming_style.snake_case.capitalization = all_lower
dotnet_naming_style.pascal_case.required_prefix =
dotnet_naming_style.pascal_case.required_suffix =
dotnet_naming_style.pascal_case.word_separator =
dotnet_naming_style.pascal_case.capitalization = pascal_case
dotnet_naming_style.private_snake_case.required_prefix = _
dotnet_naming_style.private_snake_case.required_suffix =
dotnet_naming_style.private_snake_case.word_separator = _
dotnet_naming_style.private_snake_case.capitalization = all_lower
dotnet_naming_style.camel_case.required_prefix =
dotnet_naming_style.camel_case.required_suffix =
dotnet_naming_style.camel_case.word_separator =
dotnet_naming_style.camel_case.capitalization = camel_case
dotnet_naming_style.user_types.required_prefix =
dotnet_naming_style.user_types.required_suffix =
@@ -229,6 +235,10 @@ dotnet_naming_style.interface.required_suffix =
dotnet_naming_style.interface.word_separator =
dotnet_naming_style.interface.capitalization = pascal_case
dotnet_naming_style.local.capitalization = camel_case
dotnet_naming_style.parameter.capitalization = camel_case
# rules
dotnet_naming_rule.user_types.symbols = user_types
dotnet_naming_rule.user_types.style = user_types
@@ -238,14 +248,18 @@ dotnet_naming_rule.interfaces.symbols = interface
dotnet_naming_rule.interfaces.style = interface_style
dotnet_naming_rule.interfaces.severity = suggestion
dotnet_naming_rule.public_members_snake_case.symbols = public_symbols
dotnet_naming_rule.public_members_snake_case.style = snake_case
dotnet_naming_rule.public_members_snake_case.severity = suggestion
dotnet_naming_rule.public_members.symbols = public_symbols
dotnet_naming_rule.public_members.style = pascal_case
dotnet_naming_rule.public_members.severity = suggestion
dotnet_naming_rule.private_members_snake_case.symbols = private_symbols
dotnet_naming_rule.private_members_snake_case.style = private_snake_case
dotnet_naming_rule.private_members_snake_case.severity = suggestion
dotnet_naming_rule.private_members.symbols = private_symbols
dotnet_naming_rule.private_members.style = camel_case
dotnet_naming_rule.private_members.severity = suggestion
dotnet_naming_rule.parameters.symbols = parameters
dotnet_naming_rule.parameters.style = snake_case
dotnet_naming_rule.variables_are_camel_case.severity = suggestion
dotnet_naming_rule.variables_are_camel_case.symbols = local_symbols
dotnet_naming_rule.variables_are_camel_case.style = local
dotnet_naming_rule.parameters.symbols = parameter_symbols
dotnet_naming_rule.parameters.style = parameter
dotnet_naming_rule.parameters.severity = suggestion
+4 -2
View File
@@ -1,2 +1,4 @@
*.cmd eol=crlf
*.sh eol=lf
*.cmd text=auto eol=crlf
*.cs text=auto eol=crlf
*.sh text=auto eol=lf
*.py text=auto eol=lf
+1 -1
View File
@@ -1,6 +1,6 @@
[submodule "externals"]
path = externals
url = git@bitbucket.org:bluetradesys/externals.git
url = https://bitbucket.org/bluetradesys/externals.git
[submodule "vcpkg"]
path = vcpkg
url = https://github.com/microsoft/vcpkg.git
+191 -33
View File
@@ -2,18 +2,72 @@
"version": "0.2.0",
"configurations": [
{
"name": "(gdb) Launch",
"type": "cppdbg",
"name": ".NET Core Attach",
"type": "coreclr",
"request": "attach"
},
{
"name": "Api Server",
"type": "coreclr",
"request": "launch",
"program": "${workspaceFolder}/build/sparkd-debug-gcc/server/sparkd",
"args": ["run"],
"program": "",
"windows": {
"preLaunchTask": "Build Api Server (Debug)",
"program": "spark_api_server.exe",
"args": [
"run",
"--sparkd",
"http://127.0.0.1:8000/config"
],
"cwd": "build/Debug/bin/net7.0",
},
"stopAtEntry": false,
"cwd": "${workspaceFolder}/build/sparkd-debug-gcc/server",
"environment": [],
"externalConsole": false,
"MIMode": "gdb",
"console": "integratedTerminal"
},
{
"name": "Api Server (Skip Build)",
"type": "coreclr",
"request": "launch",
"program": "",
"windows": {
"program": "spark_api_server.exe",
"args": [
"run",
"--sparkd",
"http://127.0.0.1:8000/config"
],
"cwd": "build/Debug/bin/net7.0",
},
"stopAtEntry": false,
"console": "integratedTerminal"
},
{
"name": "Bts.DataStreamWebApp.Server",
"type": "coreclr",
"request": "launch",
"preLaunchTask": "Build DataStream WebApp (Debug)",
"program": "${workspaceFolder}/build/Debug/bin/net7.0/Bts.DataStreamWebApp.Server.dll",
"cwd": "${workspaceFolder}/build/Debug/bin/net7.0",
"env": {
"ASPNETCORE_HTTP_PORT": "8080",
"ASPNETCORE_HTTPS_PORT": "8443",
"ASPNETCORE_ENVIRONMENT": "${input:dataStreamWebAppServerEnvironment}",
"ASPNETCORE_HOSTINGSTARTUPASSEMBLIES": "Microsoft.AspNetCore.SpaProxy",
}
},
{
"name": "Bts.DataStreamWebApp.Server (Skip Build)",
"type": "coreclr",
"request": "launch",
"program": "${workspaceFolder}/build/Debug/bin/net7.0/Bts.DataStreamWebApp.Server.dll",
"cwd": "${workspaceFolder}/build/Debug/bin/net7.0",
"env": {
"ASPNETCORE_HTTP_PORT": "8080",
"ASPNETCORE_HTTPS_PORT": "8443",
"ASPNETCORE_ENVIRONMENT": "${input:dataStreamWebAppServerEnvironment}",
"ASPNETCORE_HOSTINGSTARTUPASSEMBLIES": "Microsoft.AspNetCore.SpaProxy",
}
},
{
"name": "DataStreamPublisher",
"type": "coreclr",
@@ -26,11 +80,28 @@
"windows": {
"program": "DataStreamPublisher.exe",
},
"args": [
"--config",
"${workspaceFolder}/src/DataStreamSuite/DataStreamPublisher/Configurations/${input:dataStreamPublisherConfig}.json"
],
"cwd": "build/Debug/bin/net7.0",
"env": {
"DOTNET_ENVIRONMENT": "${input:dataStreamPublisherEnvironment}",
},
"cwd": "${workspaceFolder}/build/Debug/bin/net7.0",
"stopAtEntry": false,
"console": "integratedTerminal"
},
{
"name": "DataStreamPublisher (Skip Build)",
"type": "coreclr",
"request": "launch",
"program": "",
"linux": {
"program": "DataStreamPublisher",
},
"windows": {
"program": "DataStreamPublisher.exe",
},
"env": {
"DOTNET_ENVIRONMENT": "${input:dataStreamPublisherEnvironment}",
},
"cwd": "${workspaceFolder}/build/Debug/bin/net7.0",
"stopAtEntry": false,
"console": "integratedTerminal"
},
@@ -38,7 +109,7 @@
"name": "DataStreamQueryAnalyzer",
"type": "coreclr",
"request": "launch",
"preLaunchTask": "Build DataStream (Debug)",
"preLaunchTask": "Build DataStreamQueryAnalyzer (Debug)",
"program": "",
"windows": {
"program": "DataStreamQueryAnalyzer.exe",
@@ -66,11 +137,28 @@
"windows": {
"program": "PubSubServer.exe",
},
"args": [
"--config",
"${workspaceFolder}/src/DataStreamSuite/PubSubServer/Configurations/${input:pubSubServerConfig}.json"
],
"cwd": "build/Debug/bin/net7.0",
"env": {
"DOTNET_ENVIRONMENT": "${input:pubSubServerEnvironment}",
},
"stopAtEntry": false,
"console": "integratedTerminal"
},
{
"name": "PubSubServer (Skip Build)",
"type": "coreclr",
"request": "launch",
"program": "",
"linux": {
"program": "PubSubServer",
},
"windows": {
"program": "PubSubServer.exe",
},
"cwd": "build/Debug/bin/net7.0",
"env": {
"DOTNET_ENVIRONMENT": "${input:pubSubServerEnvironment}",
},
"stopAtEntry": false,
"console": "integratedTerminal"
},
@@ -82,6 +170,10 @@
"windows": {
"preLaunchTask": "Build Spark (Debug)",
"program": "Spark.exe",
"args": [
"--sparkd",
"http://127.0.0.1:8000/config"
],
"cwd": "build/Debug/bin/net7.0-windows",
},
"stopAtEntry": false,
@@ -106,43 +198,99 @@
},
{
"name": "Sparkd",
"type": "coreclr",
"type": "cppdbg",
"request": "launch",
"preLaunchTask": "Build Sparkd (Debug | Clang)",
"program": "",
"linux": {
"program": "sparkd",
"program": "${workspaceFolder}/build/sparkd-debug-clang/src/app_sparkd/sparkd",
},
"args": [
"run"
],
"cwd": "build/sparkd-debug-clang/server",
"cwd": "${workspaceFolder}/build/sparkd-debug-clang/src/app_sparkd",
"stopAtEntry": false,
"console": "integratedTerminal"
}
"externalConsole": false,
"MIMode": "gdb",
"setupCommands": [
{
"description": "Enable pretty-printing for gdb",
"text": "-enable-pretty-printing",
"ignoreFailures": true
},
{
"description": "Set Disassembly Flavor to Intel",
"text": "-gdb-set disassembly-flavor intel",
"ignoreFailures": true
}
]
},
{
"name": "Sparkd (Skip Build)",
"type": "cppdbg",
"request": "launch",
"program": "",
"linux": {
"program": "${workspaceFolder}/build/sparkd-debug-clang/src/app_sparkd/sparkd",
},
"args": [
"run"
],
"cwd": "${workspaceFolder}/build/sparkd-debug-clang/src/app_sparkd",
"stopAtEntry": false,
"externalConsole": false,
"MIMode": "gdb",
"setupCommands": [
{
"description": "Enable pretty-printing for gdb",
"text": "-enable-pretty-printing",
"ignoreFailures": true
},
{
"description": "Set Disassembly Flavor to Intel",
"text": "-gdb-set disassembly-flavor intel",
"ignoreFailures": true
}
]
},
],
"inputs": [
{
"id": "dataStreamPublisherConfig",
"id": "dataStreamPublisherEnvironment",
"type": "pickString",
"description": "Publisher Configuration",
"options": [
"MarketDataPublisher",
"SimpleEchoClient",
"SimpleTestDataPublisher",
"TestDataPublisher",
"Development",
"MarketData",
"Production",
"ProductionAws",
"Staging",
],
"default": "MarketDataPublisher",
},
{
"id": "pubSubServerConfig",
"id": "pubSubServerEnvironment",
"type": "pickString",
"description": "PubSub Configuration",
"options": [
"PubSubServer",
"SimpleEchoClient",
"Development",
"EchoServer",
"Production",
"ProductionAws",
"Staging",
],
"default": "PubSubServer",
"default": "Development",
},
{
"id": "dataStreamWebAppServerEnvironment",
"type": "pickString",
"description": "DataStream WebApp Environment",
"options": [
"Development",
"Production",
"ProductionAws",
"Staging",
]
},
{
"id": "sparkConfigServer",
@@ -150,5 +298,15 @@
"description": "Config Server URL",
"default": "http://localhost:8000/config",
},
{
"id": "isSparkApiEnabled",
"type": "pickString",
"description": "Enable Spark Api (connect to sparkd)",
"options": [
"True",
"False",
],
"default": "True",
}
]
}
+287 -22
View File
@@ -121,6 +121,117 @@
"color": "terminal.ansiCyan"
},
},
{
"label": "Build DataStreamQueryAnalyzer",
"group": "build",
"type": "process",
"command": "",
"windows": {
"command": "tools/build_data_stream_query_analyzer.cmd",
},
"args": [
"--configuration",
"${input:buildConfiguration}",
],
"icon": {
"id": "terminal",
"color": "terminal.ansiCyan"
},
},
{
"label": "Build DataStreamQueryAnalyzer (Debug)",
"group": "build",
"type": "process",
"command": "",
"windows": {
"command": "tools/build_data_stream_query_analyzer.cmd",
},
"args": [
"--configuration",
"Debug",
],
"icon": {
"id": "terminal",
"color": "terminal.ansiCyan"
},
},
{
"label": "Build DataStreamQueryAnalyzer (Release)",
"group": "build",
"type": "process",
"command": "",
"windows": {
"command": "tools/build_data_stream_query_analyzer.cmd",
},
"args": [
"--configuration",
"Release",
],
"icon": {
"id": "terminal",
"color": "terminal.ansiCyan"
},
},
{
"label": "Build DataStream WebApp",
"group": "build",
"type": "process",
"command": "",
"linux": {
"command": "tools/build_data_stream_web_app.sh",
},
"windows": {
"command": "tools/build_data_stream_web_app.cmd",
},
"args": [
"--configuration",
"${input:buildConfiguration}",
],
"icon": {
"id": "terminal",
"color": "terminal.ansiCyan"
},
},
{
"label": "Build DataStream WebApp (Debug)",
"group": "build",
"type": "process",
"command": "",
"linux": {
"command": "tools/build_data_stream_web_app.sh",
},
"windows": {
"command": "tools/build_data_stream_web_app.cmd",
},
"args": [
"--configuration",
"Debug",
],
"icon": {
"id": "terminal",
"color": "terminal.ansiCyan"
},
},
{
"label": "Build DataStream WebApp (Release)",
"group": "build",
"type": "process",
"command": "",
"linux": {
"command": "tools/build_data_stream_web_app.sh",
},
"windows": {
"command": "tools/build_data_stream_web_app.cmd",
},
"args": [
"--configuration",
"Release",
],
"icon": {
"id": "terminal",
"color": "terminal.ansiCyan"
},
},
{
"label": "Build Spark",
"group": "build",
@@ -189,25 +300,6 @@
"color": "terminal.ansiCyan"
},
},
{
"label": "Build Sparkd -j8",
"group": "build",
"type": "shell",
"command": "",
"linux": {
"command": "tools/build_sparkd.sh",
"args": [
"--preset",
"${input:sparkdCmakePreset}",
"-j",
"8"
],
},
"icon": {
"id": "terminal",
"color": "terminal.ansiCyan"
},
},
{
"label": "Build Sparkd (Debug)",
"group": "build",
@@ -322,6 +414,22 @@
"color": "terminal.ansiCyan"
},
},
{
"label": "Publish DataStreamWebApp",
"group": "build",
"type": "process",
"command": "",
"linux": {
"command": "tools/publish_data_stream_web_app.sh",
},
"windows": {
"command": "tools/publish_data_stream_web_app.cmd",
},
"icon": {
"id": "terminal",
"color": "terminal.ansiCyan"
},
},
{
"label": "Edit DataStream Config",
"type": "process",
@@ -372,7 +480,7 @@
},
{
"label": "Run DataStreamQueryAnalyzer",
"dependsOn": "Build DataStream",
"dependsOn": "Build DataStreamQueryAnalyzer",
"type": "process",
"command": "",
"windows": {
@@ -392,7 +500,7 @@
},
{
"label": "Run DataStreamQueryAnalyzer (Debug)",
"dependsOn": "Build DataStream (Debug)",
"dependsOn": "Build DataStreamQueryAnalyzer (Debug)",
"type": "process",
"command": "",
"windows": {
@@ -412,7 +520,7 @@
},
{
"label": "Run DataStreamQueryAnalyzer (Release)",
"dependsOn": "Build DataStream (Release)",
"dependsOn": "Build DataStreamQueryAnalyzer (Release)",
"type": "process",
"command": "",
"windows": {
@@ -499,6 +607,89 @@
"color": "terminal.ansiGreen",
},
},
{
"label": "Run DataStream WebApp (Debug)",
"dependsOn": "Build DataStream WebApp (Debug)",
"command": "",
"linux": {
"command": "Bts.DataStreamWebApp.Server",
},
"windows": {
"command": "Bts.DataStreamWebApp.Server.exe",
},
"args": [
"--config",
"${workspaceFolder}/src/DataStreamSuite/Bts.DataStreamWebApp.Server/Configurations/DevServer.json",
],
"options": {
"cwd": "${workspaceFolder}/build/Debug/bin/net7.0",
"env": {
"ASPNETCORE_HTTP_PORT": "8080",
"ASPNETCORE_HTTPS_PORT": "8443",
"ASPNETCORE_ENVIRONMENT": "Development",
"ASPNETCORE_HOSTINGSTARTUPASSEMBLIES": "Microsoft.AspNetCore.SpaProxy",
}
},
"icon": {
"id": "run",
"color": "terminal.ansiGreen",
},
},
{
"label": "Run DataStream WebApp (Release)",
"dependsOn": "Build DataStream WebApp (Release)",
"command": "",
"linux": {
"command": "Bts.DataStreamWebApp.Server",
},
"windows": {
"command": "Bts.DataStreamWebApp.Server.exe",
},
"args": [
"--config",
"${workspaceFolder}/src/DataStreamSuite/Bts.DataStreamWebApp.Server/Configurations/DevServer.json",
],
"options": {
"cwd": "${workspaceFolder}/build/Release/bin/net7.0",
"env": {
"ASPNETCORE_HTTP_PORT": "8080",
"ASPNETCORE_HTTPS_PORT": "8443",
"ASPNETCORE_ENVIRONMENT": "Production",
"ASPNETCORE_HOSTINGSTARTUPASSEMBLIES": "Microsoft.AspNetCore.SpaProxy",
}
},
"icon": {
"id": "run",
"color": "terminal.ansiGreen",
},
},
{
"label": "Run DataStream WebApp (Release | Skip Build)",
"command": "",
"linux": {
"command": "Bts.DataStreamWebApp.Server",
},
"windows": {
"command": "Bts.DataStreamWebApp.Server.exe",
},
"args": [
"--config",
"${workspaceFolder}/src/DataStreamSuite/Bts.DataStreamWebApp.Server/Configurations/DevServer.json",
],
"options": {
"cwd": "${workspaceFolder}/build/Release/bin/net7.0",
"env": {
"ASPNETCORE_HTTP_PORT": "8080",
"ASPNETCORE_HTTPS_PORT": "8443",
"ASPNETCORE_ENVIRONMENT": "Production",
"ASPNETCORE_HOSTINGSTARTUPASSEMBLIES": "Microsoft.AspNetCore.SpaProxy",
}
},
"icon": {
"id": "run",
"color": "terminal.ansiGreen",
},
},
{
"label": "Run PubSubServer",
"dependsOn": "Build DataStream",
@@ -545,6 +736,28 @@
"color": "terminal.ansiGreen",
},
},
{
"label": "Run PubSubServer (Debug | Skip Build)",
"type": "process",
"command": "",
"linux": {
"command": "PubSubServer",
},
"windows": {
"command": "PubSubServer.exe",
},
"args": [
"--config",
"${workspaceFolder}/src/DataStreamSuite/PubSubServer/Configurations/${input:pubSubServerConfig}.json",
],
"options": {
"cwd": "${workspaceFolder}/build/Debug/bin/net7.0"
},
"icon": {
"id": "run",
"color": "terminal.ansiGreen",
},
},
{
"label": "Run PubSubServer (Release)",
"dependsOn": "Build DataStream (Release)",
@@ -568,6 +781,28 @@
"color": "terminal.ansiGreen",
},
},
{
"label": "Run PubSubServer (Release | Skip Build)",
"type": "process",
"command": "",
"linux": {
"command": "PubSubServer",
},
"windows": {
"command": "PubSubServer.exe",
},
"args": [
"--config",
"${workspaceFolder}/src/DataStreamSuite/PubSubServer/Configurations/${input:pubSubServerConfig}.json",
],
"options": {
"cwd": "${workspaceFolder}/build/Release/bin/net7.0"
},
"icon": {
"id": "run",
"color": "terminal.ansiGreen",
},
},
{
"label": "Run Spark",
"dependsOn": "Build Spark",
@@ -703,6 +938,36 @@
"id": "wrench"
},
},
{
"label": "View PubSubServer Logs (Last Run)",
"type": "process",
"command": "",
"linux": {
"command": "tools/logging/show_last_run.sh",
"args": [
"${env:HOME}/.local/share/DataStream/data/logging.PubSubServer",
"Starting PubSubServer"
]
},
"windows": {
"command": "tools/logging/show_last_run.cmd",
"args": [
"DataStream",
"PubSubServer.exe",
"Starting PubSubServer"
]
},
"presentation": {
"reveal": "always",
"focus": true,
"panel": "shared",
"showReuseMessage": false,
"close": true,
},
"icon": {
"id": "wrench"
},
},
],
"inputs": [
{
+15 -46
View File
@@ -18,10 +18,10 @@ set(
# SHARED_MEMORY_PREFIX serves as a kind of namespace so multiple applications
# can run simultaneously without fighting over the same shm files (e.g., sparkd,
# api server, etc.).
set(SHARED_MEMORY_PREFIX "bts" CACHE STRING "Prefix to use when creating shared memory files")
set(SPARK_BIN "${EXTERNALS_ROOT}/bin")
set(SPARK_INCLUDE "${EXTERNALS_ROOT}/include")
set(SPARK_LIB "${EXTERNALS_ROOT}/lib/$<CONFIG>")
set(SHARED_MEMORY_PREFIX "\${BTS_SHARED_MEMORY_PREFIX}" CACHE STRING "Prefix to use when creating shared memory files")
set(EXTERNALS_BIN "${EXTERNALS_ROOT}/bin")
set(EXTERNALS_INCLUDE "${EXTERNALS_ROOT}/include")
set(EXTERNALS_LIB "${EXTERNALS_ROOT}/lib/$<CONFIG>")
set(BUILD_DOCS OFF CACHE BOOL "Build documentation")
set(BUILD_SDK OFF CACHE BOOL "Build the sdk")
set(EXPORT_MD_CONSUMER OFF CACHE BOOL "Export (extern) C functions for MD consumer interop")
@@ -33,25 +33,18 @@ else()
set(BUILD_ADMIN_WEB_APP ON CACHE BOOL "Build the Admin WebApp")
endif()
include_directories(BEFORE SYSTEM ${SPARK_INCLUDE} ${EXTERNALS_ROOT}/include/zlib)
link_directories(${SPARK_LIB})
include_directories(BEFORE SYSTEM ${EXTERNALS_INCLUDE} ${EXTERNALS_ROOT}/include/zlib)
link_directories(${EXTERNALS_LIB})
add_compile_definitions(SHARED_MEMORY_PREFIX="${SHARED_MEMORY_PREFIX}")
add_compile_definitions(APP_DATA_DIR="${APP_DATA_DIR}")
if(DEFINED APP_DATA_DIR_OVERRIDE)
add_compile_definitions(APP_DATA_DIR_OVERRIDE="${APP_DATA_DIR_OVERRIDE}")
if(DEFINED APP_DATA_DIR)
add_compile_definitions(APP_DATA_DIR="${APP_DATA_DIR}")
endif()
add_compile_definitions(EXPORT_MD_CONSUMER="${EXPORT_MD_CONSUMER}")
if(WIN32)
# subdirs
add_subdirectory(src/lib)
add_subdirectory(src/lib_spark)
else()
# make sparkd look for libspark.so in its directory
set(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE)
set(CMAKE_INSTALL_RPATH ".")
set(CMAKE_INSTALL_PREFIX "${CMAKE_SOURCE_DIR}/install")
set(SPARK_INSTALL_PREFIX "${CMAKE_INSTALL_PREFIX}/$<CONFIG>")
if(BUILD_SPARKD)
# Note: All macros that begin w/ 'CCAPI' are ccapi's macros (defined and/or
# read by their code). We override some below.
@@ -111,10 +104,14 @@ else()
# subdirs
if (BUILD_SPARKD)
add_subdirectory(src/server/sparkd)
add_subdirectory(src/lib_system)
add_subdirectory(src/lib_http)
add_subdirectory(src/lib_domain)
add_subdirectory(src/lib_python_md)
add_subdirectory(src/app_sparkd)
add_subdirectory(src/data_files)
endif()
add_subdirectory(src/lib)
add_subdirectory(src/lib_spark)
if (BUILD_ADMIN_WEB_APP)
add_subdirectory(src/Spark.Admin.WebApp)
endif()
@@ -124,32 +121,4 @@ else()
if(BUILD_SDK)
add_subdirectory(src/sdk)
endif()
if(BUILD_SPARKD)
add_custom_command(
OUTPUT ${CMAKE_BINARY_DIR}/server
COMMAND mkdir -p ${CMAKE_BINARY_DIR}/server
COMMAND ln ARGS -sf ${CMAKE_BINARY_DIR}/src/server/sparkd/sparkd ${CMAKE_BINARY_DIR}/server
COMMAND ln ARGS -sf ${CMAKE_BINARY_DIR}/data_files ${CMAKE_BINARY_DIR}/server
COMMAND ln ARGS -sf ${CMAKE_BINARY_DIR}/src/lib/libspark.so ${CMAKE_BINARY_DIR}/server
COMMAND
ln ARGS -sf
${EXTERNALS_ROOT}/lib/Release/libOnixS.IceImpactMulticastPriceFeedHandlerCpp.so.3.16.0.0
${CMAKE_BINARY_DIR}/server
COMMAND ln ARGS -sf ${EXTERNALS_ROOT}/lib/Release/libACE-5.7.5.so ${CMAKE_BINARY_DIR}/server
COMMAND mkdir ARGS -p ${CMAKE_BINARY_DIR}/server/plugins
COMMENT "Creating server runtime layout"
DEPENDS sparkd data_files_dir)
add_custom_target(server_layout ALL DEPENDS ${CMAKE_BINARY_DIR}/server)
install(
DIRECTORY "${CMAKE_BINARY_DIR}/server/plugins"
DESTINATION "${SPARK_INSTALL_PREFIX}/bin")
install(
FILES
"${SPARK_LIB}/libACE-5.7.5.so"
"${SPARK_LIB}/libOnixS.IceImpactMulticastPriceFeedHandlerCpp.so.3.16.0.0"
DESTINATION "${SPARK_INSTALL_PREFIX}/bin")
endif()
endif()
+27 -45
View File
@@ -20,8 +20,6 @@
"SPARK_CMAKE_CXX_FLAGS": "/MP /WX /wd4250 /wd4251 /wd4244 /wd4267 /wd4355 /wd4275 /wd4345 /bigobj /EHsc /DCURL_STATICLIB /D_WINSOCK_DEPRECATED_NO_WARNINGS /D__TBB_NO_IMPLICIT_LINKAGE /DBOOST_ALL_NO_LIB /std:c++20 /D_SCL_SECURE_NO_WARNINGS /D_CRT_SECURE_NO_WARNINGS /D_WIN32_WINNT=0x0600"
},
"cacheVariables": {
"APP_DATA_DIR": "${dollar}{APPDATA}/bts",
"APP_DATA_DIR_OVERRIDE": "${dollar}{SPARK_DATA_DIR}",
"CMAKE_CXX_FLAGS": "$env{SPARK_CMAKE_CXX_FLAGS}",
"CMAKE_CXX_FLAGS_DEBUG": "$env{SPARK_CMAKE_CXX_FLAGS} /MTd",
"CMAKE_CXX_FLAGS_RELEASE": "$env{SPARK_CMAKE_CXX_FLAGS} /MT",
@@ -31,13 +29,11 @@
"CMAKE_TOOLCHAIN_FILE": "${sourceDir}/vcpkg/scripts/buildsystems/vcpkg.cmake",
"EXTERNALS_ROOT": "${sourceDir}/externals/windows",
"CONFIG_DEFAULTS_PATH": "config/profiles/default/*.conf",
"DOTNET_BINARY_DIR": "${sourceDir}/build/$<CONFIG>/bin/net7.0",
"DOTNET_WINDOWS_BINARY_DIR": "${sourceDir}/build/$<CONFIG>/bin/net7.0-windows",
"DOTNET_BINARY_DIR": "${sourceDir}/build/$<CONFIG>/bin/net7.0;${sourceDir}/build/$<CONFIG>/bin/net7.0-windows",
"EXPORT_MD_CONSUMER": {
"type": "BOOL",
"value": "ON"
},
"SHARED_MEMORY_PREFIX": "spark"
}
}
},
@@ -53,16 +49,11 @@
"binaryDir": "${sourceDir}/build/${presetName}",
"cacheVariables": {
"platform": "",
"APP_DATA_DIR": "${dollar}{HOME}/sparkdata",
"APP_DATA_DIR_OVERRIDE": "${dollar}{SPARKD_DATA_DIR}",
"CMAKE_BUILD_WITH_INSTALL_RPATH": {
"type": "BOOL",
"value": "ON"
},
"CMAKE_CXX_STANDARD": "20",
"CMAKE_INSTALL_RPATH": ".",
"CMAKE_INSTALL_PREFIX": "${sourceDir}/install/${presetName}/sparkd",
"CMAKE_INSTALL_PREFIX": "${sourceDir}/install/${presetName}",
"CMAKE_PREFIX_PATH": "${sourceDir}/cmake/;${sourceDir}/.venv/lib/python3.10/site-packages/pybind11/share/cmake/pybind11",
"CMAKE_EXE_LINKER_FLAGS": "-Wno-undef",
"CMAKE_TOOLCHAIN_FILE": "${sourceDir}/vcpkg/scripts/buildsystems/vcpkg.cmake",
"EXTERNALS_ROOT": "${sourceDir}/externals/linux",
"BUILD_SDK": {
"type": "BOOL",
@@ -71,8 +62,7 @@
"BUILD_ADMIN_WEB_APP": {
"type": "BOOL",
"value": "ON"
},
"SHARED_MEMORY_PREFIX": "sparkd"
}
}
},
@@ -87,7 +77,7 @@
"value": "ON"
},
"CCAPI_LOG_LEVEL": "Info",
"CONFIG_DEFAULTS_PATH": "config/profiles/md_dev/*.conf;config/profiles/default/*.conf"
"CONFIG_DEFAULTS_PATH": "spark/config/profiles/md_dev/*.conf;spark/config/profiles/default/*.conf"
}
},
@@ -98,7 +88,7 @@
"cacheVariables": {
"CMAKE_BUILD_TYPE": "Release",
"CCAPI_LOG_LEVEL": "Warn",
"CONFIG_DEFAULTS_PATH": "config/profiles/md_prod/*.conf;config/profiles/default/*.conf"
"CONFIG_DEFAULTS_PATH": "spark/config/profiles/md_prod/*.conf;spark/config/profiles/default/*.conf"
}
},
@@ -153,11 +143,7 @@
{
"name": "spark-api-server-windows",
"inherits": "spark",
"displayName": "Spark Api Server (Windows)",
"cacheVariables": {
"APP_DATA_DIR_OVERRIDE": "${dollar}{SPARK_API_SERVER_DATA_DIR}",
"SHARED_MEMORY_PREFIX": "spark_api_server"
}
"displayName": "Spark Api Server (Windows)"
},
{
@@ -166,7 +152,6 @@
"displayName": "Spark Api Server (Linux)",
"generator": "Ninja Multi-Config",
"cacheVariables": {
"APP_DATA_DIR_OVERRIDE": "${dollar}{SPARK_API_SERVER_DATA_DIR}",
"BUILD_ADMIN_WEB_APP": {
"type": "BOOL",
"value": "OFF"
@@ -179,9 +164,12 @@
"CMAKE_CXX_COMPILER": "clang++",
"CMAKE_CXX_FLAGS_DEBUG": "-g -fPIC -Wall -pedantic -Wextra -Werror -Wno-missing-braces -Wno-ignored-qualifiers --std=c++20 -fdiagnostics-color -DDEBUG",
"CMAKE_CXX_FLAGS_RELEASE": "-fPIC -Wall -pedantic -Wextra -Werror -Wno-missing-braces -Wno-ignored-qualifiers --std=c++20 -fdiagnostics-color -fno-omit-frame-pointer -DNDEBUG -O3",
"CMAKE_INSTALL_PREFIX": "${sourceDir}/install/${presetName}",
"CMAKE_EXPORT_COMPILE_COMMANDS": {
"type": "BOOL",
"value": "ON"
},
"CONFIG_DEFAULTS_PATH": "config/profiles/default/*.conf",
"DOTNET_BINARY_DIR": "${sourceDir}/build/$<CONFIG>/bin/net7.0",
"DOTNET_BINARY_DIR": "${sourceDir}/build/Spark.Api.Server/$<CONFIG>/bin/net7.0",
"EXPORT_MD_CONSUMER": {
"type": "BOOL",
"value": "ON"
@@ -219,10 +207,8 @@
"description": "Data Stream utils ported over from Edge",
"binaryDir": "${sourceDir}/build/${presetName}",
"cacheVariables": {
"APP_DATA_DIR": "${dollar}{APPDATA}/bts/DataStream",
"APP_DATA_DIR_OVERRIDE": "${dollar}{DATA_STREAM_DATA_DIR}",
"CONFIG_DEFAULTS_PATH": "config/profiles/default/*.conf;config/profiles/data_stream/*.conf",
"SHARED_MEMORY_PREFIX": "data_stream"
"DOTNET_BINARY_DIR": "${sourceDir}/build/$<CONFIG>/bin/net7.0;${sourceDir}/build/$<CONFIG>/bin/net7.0-windows"
}
},
@@ -238,8 +224,6 @@
"inherits": "sparkd",
"generator": "Ninja Multi-Config",
"cacheVariables": {
"APP_DATA_DIR": "${dollar}{HOME}/.local/share/DataStream",
"APP_DATA_DIR_OVERRIDE": "${dollar}{DATA_STREAM_DATA_DIR}",
"BUILD_ADMIN_WEB_APP": {
"type": "BOOL",
"value": "OFF"
@@ -256,14 +240,13 @@
"CMAKE_CXX_COMPILER": "clang++",
"CMAKE_CXX_FLAGS_DEBUG": "-g -fPIC -Wall -pedantic -Wextra -Werror -Wno-missing-braces -Wno-ignored-qualifiers --std=c++20 -fdiagnostics-color -DDEBUG",
"CMAKE_CXX_FLAGS_RELEASE": "-fPIC -Wall -pedantic -Wextra -Werror -Wno-missing-braces -Wno-ignored-qualifiers --std=c++20 -fdiagnostics-color -fno-omit-frame-pointer -DNDEBUG -O3",
"CMAKE_INSTALL_PREFIX": "${sourceDir}/install/${presetName}",
"CONFIG_DEFAULTS_PATH": "config/profiles/default/*.conf;config/profiles/data_stream/*.conf",
"DOTNET_BINARY_DIR": "${sourceDir}/build/$<CONFIG>/bin/net7.0",
"CONFIG_PROFILES": "default;data_stream",
"DOTNET_BINARY_DIR": "${sourceDir}/build/$<CONFIG>/bin/net7.0;${sourceDir}/build/$<CONFIG>/bin/net8.0",
"EXPORT_MD_CONSUMER": {
"type": "BOOL",
"value": "ON"
},
"SHARED_MEMORY_PREFIX": "data_stream"
}
}
}
],
@@ -302,16 +285,7 @@
"lhs": "${hostSystemName}",
"rhs": "Linux"
},
"hidden": true,
"targets": [
"data_files_dir",
"admin_web_app_bundle",
"install",
"libspark",
"rapidjson_project",
"server_layout",
"sparkd"
]
"hidden": true
},
{
@@ -321,6 +295,14 @@
"inherits": "sparkd"
},
{
"name": "sparkd-debug-clang-j6",
"configurePreset": "sparkd-debug-clang",
"displayName": "Sparkd (Debug | Clang | jobs=6)",
"inherits": "sparkd",
"jobs": 6
},
{
"name": "sparkd-debug-gcc",
"configurePreset": "sparkd-debug-gcc",
+7
View File
@@ -0,0 +1,7 @@
include_guard(GLOBAL)
macro(export_to_parent)
foreach(var IN ITEMS ${ARGN})
set(${var} ${${var}} PARENT_SCOPE)
endforeach()
endmacro()
+133
View File
@@ -0,0 +1,133 @@
include_guard(DIRECTORY)
include(${CMAKE_SOURCE_DIR}/cmake/ExportToParent.cmake)
include(${CMAKE_SOURCE_DIR}/cmake/InstallBasic.cmake)
include(${CMAKE_SOURCE_DIR}/cmake/ProtobufGenerateCpp.cmake)
include(${CMAKE_SOURCE_DIR}/cmake/SetFirstDefined.cmake)
include(${CMAKE_SOURCE_DIR}/cmake/safe_dep_glob.cmake)
function(init_library)
set(_options SHARED STATIC)
set(_one_value_args
NAME
SOURCE_DIR
INCLUDE_DIR
)
set(_multi_value_args)
cmake_parse_arguments(_ARGS "${_options}" "${_one_value_args}" "${_multi_value_args}" ${ARGN})
if (_ARGS_SHARED AND _ARGS_STATIC)
message(FATAL_ERROR "Invalid args passed to init_library: cannot specify STATIC and SHARED")
endif()
set_first_defined(_name _ARGS_NAME PROJECT_NAME)
set_first_defined(_source_dir _ARGS_SOURCE_DIR _name)
set_first_defined(_include_dir _ARGS_INCLUDE_DIR _source_dir)
# Cpp sources
file(
GLOB_RECURSE ${_name}_SOURCES
RELATIVE ${CMAKE_CURRENT_SOURCE_DIR}
${_source_dir}/*.cpp
)
# Note: cpp files inside 'tests' dir(s) are considered tests.
# Remove tests from main sources, and add them to test sources
foreach(item ${${_name}_SOURCES})
if(${item} MATCHES "tests/.*\.cpp$")
list(APPEND ${_name}_TEST_SOURCES ${item})
endif()
endforeach()
foreach(item ${${_name}_TEST_SOURCES})
list(REMOVE_ITEM ${_name}_SOURCES ${item})
endforeach()
# Proto sources
file(
GLOB_RECURSE ${_name}_PROTO_SOURCES
RELATIVE ${CMAKE_CURRENT_SOURCE_DIR}
${_source_dir}/protobufs/*.proto
)
update_deps_file("${${_name}_PROTO_SOURCES};")
protobuf_generate_cpp(
CXX_SOURCES ${_name}_CXX_PROTO_SOURCES
CXX_HEADERS ${_name}_CXX_PROTO_HEADERS
CXX_OUT_DIR ${CMAKE_CURRENT_BINARY_DIR}
PROTO_INCLUDE_DIRS
${EXTERNALS_INCLUDE}
${CMAKE_CURRENT_SOURCE_DIR}
PROTO_SOURCES
${${_name}_PROTO_SOURCES}
)
export_to_parent(
${_name}_SOURCES
${_name}_TEST_SOURCES
${_name}_PROTO_SOURCES
${_name}_CXX_PROTO_SOURCES
${_name}_CXX_PROTO_HEADERS
)
# Add main lib
if (_ARGS_STATIC)
add_library(${_name} STATIC ${${_name}_SOURCES} ${${_name}_CXX_PROTO_SOURCES})
elseif(_ARGS_SHARED)
add_library(${_name} SHARED ${${_name}_SOURCES} ${${_name}_CXX_PROTO_SOURCES})
else()
add_library(${_name} ${${_name}_SOURCES} ${${_name}_CXX_PROTO_SOURCES})
endif()
target_include_directories(
${_name}
PUBLIC
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>
)
# Include protos (if any)
if (${_name}_PROTO_SOURCES)
target_include_directories(
${_name}
PUBLIC
$<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}>
)
endif()
target_link_directories(
${_name}
PUBLIC
$<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}>
$<INSTALL_INTERFACE:${CMAKE_INSTALL_LIBDIR}>
)
# Install main lib
foreach(_hdr ${${_name}_CXX_PROTO_HEADERS})
file(RELATIVE_PATH _relative_path ${CMAKE_CURRENT_BINARY_DIR} ${_hdr})
install(FILES ${_hdr} DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/${_relative_path})
endforeach()
install_basic(
NAME ${_name}
INCLUDE_DIR ${_include_dir}
)
# Add tests executable (if there are any)
if (${_name}_TEST_SOURCES)
add_executable(${_name}-tests ${${_name}_TEST_SOURCES})
get_target_property(_lib_type ${_name} TYPE)
if (_lib_type STREQUAL "SHARED_LIBRARY")
# Set RPATH so the tests exe knows where to find the shared lib.
set_target_properties(
${_name}-tests
PROPERTIES
BUILD_RPATH "$ORIGIN"
INSTALL_RPATH "$ORIGIN/../${CMAKE_INSTALL_LIBDIR}"
)
endif()
# Link main lib
target_link_directories(
${_name}-tests
PRIVATE
$<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}>
$<INSTALL_INTERFACE:${CMAKE_INSTALL_LIBDIR}>
)
target_link_libraries(${_name}-tests PRIVATE ${_name})
# Catch2
find_package(Catch2 CONFIG REQUIRED COMPONENTS Catch2WithMain)
target_link_libraries(${_name}-tests PRIVATE Catch2::Catch2WithMain)
endif()
endfunction()
+39
View File
@@ -0,0 +1,39 @@
include_guard(DIRECTORY)
include(GNUInstallDirs)
include(${CMAKE_SOURCE_DIR}/cmake/PackBasic.cmake)
include(${CMAKE_SOURCE_DIR}/cmake/SetFirstDefined.cmake)
function(install_basic)
set(_options)
set(_one_value_args NAME INCLUDE_DIR)
set(_multi_value_args)
cmake_parse_arguments(_ARGS "${_options}" "${_one_value_args}" "${_multi_value_args}" ${ARGN})
set_first_defined(_name _ARGS_NAME PROJECT_NAME)
set_first_defined(_include_dir _ARGS_INCLUDE_DIR _name)
file(
GLOB_RECURSE ${_name}_HEADERS
RELATIVE ${CMAKE_CURRENT_SOURCE_DIR}
${_include_dir}/*.h)
target_sources(${_name}
PUBLIC
FILE_SET HEADERS FILES ${${_name}_HEADERS})
install(
TARGETS ${_name}
EXPORT ${_name}Targets
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
FILE_SET HEADERS DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}
)
install(
EXPORT ${_name}Targets
FILE ${_name}Targets.cmake
NAMESPACE ${_name}::
DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${_name})
pack_basic()
endfunction()
+44
View File
@@ -0,0 +1,44 @@
include_guard(DIRECTORY)
include(CMakePackageConfigHelpers)
include(${CMAKE_SOURCE_DIR}/cmake/SetFirstDefined.cmake)
function(pack_basic)
set(_options)
set(_one_value_args NAME)
set(_multi_value_args)
cmake_parse_arguments(_ARGS "${_options}" "${_one_value_args}" "${_multi_value_args}" ${ARGN})
set_first_defined(_name _ARGS_NAME PROJECT_NAME)
find_package(Git QUIET)
if(GIT_FOUND)
execute_process(
COMMAND ${GIT_EXECUTABLE} describe --tags --always --dirty
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
OUTPUT_VARIABLE PROJECT_VERSION
OUTPUT_STRIP_TRAILING_WHITESPACE
)
endif()
if(NOT GIT_FOUND OR PROJECT_VERSION STREQUAL "")
set(PROJECT_VERSION "0.0.0")
endif()
write_basic_package_version_file(
"${CMAKE_CURRENT_BINARY_DIR}/${_name}ConfigVersion.cmake"
VERSION ${PROJECT_VERSION}
COMPATIBILITY AnyNewerVersion
)
configure_package_config_file(
"${CMAKE_CURRENT_SOURCE_DIR}/Config.cmake.in"
"${CMAKE_CURRENT_BINARY_DIR}/${_name}Config.cmake"
INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${_name}
)
install(
FILES
"${CMAKE_CURRENT_BINARY_DIR}/${_name}Config.cmake"
"${CMAKE_CURRENT_BINARY_DIR}/${_name}ConfigVersion.cmake"
DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${_name})
endfunction()
-145
View File
@@ -1,145 +0,0 @@
set(Protobuf_PROTOC_EXECUTABLE "${SPARK_BIN}/protoc")
set(PROTOBUF_INCLUDE_DIRS "${SPARK_INCLUDE}")
set(PROTOBUF_STATIC_LIB "libprotobuf")
function(Protobuf_Generate_Cpp SRCS HDRS)
set(${SRCS})
set(${HDRS})
foreach(FIL ${ARGN})
get_filename_component(ABS_FIL ${FIL} ABSOLUTE)
get_filename_component(FIL_WE ${FIL} NAME_WE)
get_filename_component(FIL_DIR ${FIL} DIRECTORY)
if(FIL_DIR)
set(FIL_WE "${FIL_DIR}/${FIL_WE}")
endif()
list(APPEND ${SRCS} "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.cc")
list(APPEND ${HDRS} "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.h")
add_custom_command(
OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.cc"
"${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.h"
COMMAND
${Protobuf_PROTOC_EXECUTABLE} ARGS --cpp_out=${CMAKE_BINARY_DIR}/src -I
${PROTOBUF_INCLUDE_DIRS} -I ${CMAKE_SOURCE_DIR}/src ${ABS_FIL}
DEPENDS ${ABS_FIL}
COMMENT "Running C++ protocol buffer compiler on ${FIL}"
VERBATIM)
endforeach()
set_source_files_properties(${${SRCS}} ${${HDRS}} PROPERTIES GENERATED TRUE)
# remove some warnings from protobuf generated files
if(WIN32)
set_source_files_properties(${${SRCS}} PROPERTIES COMPILE_FLAGS
"/wd4309 /wd4838")
else()
set_source_files_properties(
${${SRCS}}
PROPERTIES
COMPILE_FLAGS
"-Wno-unused-parameter -Wno-effc++ -Wno-pedantic -Wno-sign-compare -Wno-uninitialized"
)
endif()
set(${SRCS}
${${SRCS}}
PARENT_SCOPE)
set(${HDRS}
${${HDRS}}
PARENT_SCOPE)
endfunction()
function(Service_Compiler_Generate_Cpp SRCS HDRS)
set(${SRCS})
set(${HDRS})
foreach(FIL ${ARGN})
get_filename_component(ABS_FIL ${FIL} ABSOLUTE)
get_filename_component(FIL_WE ${FIL} NAME_WE)
get_filename_component(FIL_DIR ${FIL} DIRECTORY)
if(FIL_DIR)
set(FIL_WE "${FIL_DIR}/${FIL_WE}")
endif()
list(APPEND ${SRCS} "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.cc")
list(APPEND ${HDRS} "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.h")
list(APPEND ${SRCS} "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.client.pb.cc")
list(APPEND ${HDRS} "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.client.pb.h")
list(APPEND ${SRCS} "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.server.pb.cc")
list(APPEND ${HDRS} "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.server.pb.h")
set(TEMPLATES
${CMAKE_CURRENT_SOURCE_DIR}/api/service_compiler/templates/cpp/client.pb.cc
${CMAKE_CURRENT_SOURCE_DIR}/api/service_compiler/templates/cpp/client.pb.h
${CMAKE_CURRENT_SOURCE_DIR}/api/service_compiler/templates/cpp/server.pb.cc
${CMAKE_CURRENT_SOURCE_DIR}/api/service_compiler/templates/cpp/server.pb.h
)
if(WIN32)
add_custom_command(
OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.cc"
"${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.h"
"${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.client.pb.cc"
"${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.client.pb.h"
"${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.server.pb.cc"
"${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.server.pb.h"
COMMAND SET PYTHONPATH=${SPARK_PYTHON}
COMMAND ${CMAKE_SOURCE_DIR}/.venv/Scripts/activate.bat
COMMAND
${Protobuf_PROTOC_EXECUTABLE} ARGS --cpp_out=${CMAKE_BINARY_DIR}/src
--plugin=protoc-gen-service=${CMAKE_SOURCE_DIR}/src/lib/api/service_compiler/plugin.cmd
--service_out=lang=cpp:${CMAKE_BINARY_DIR}/src -I
${PROTOBUF_INCLUDE_DIRS} -I ${CMAKE_SOURCE_DIR}/src ${ABS_FIL}
DEPENDS ${ABS_FIL} ${TEMPLATES}
${CMAKE_CURRENT_SOURCE_DIR}/api/service_compiler/plugin.py
COMMENT "Running C++ service compiler on ${FIL}"
VERBATIM)
else()
add_custom_command(
OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.cc"
"${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.h"
"${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.client.pb.cc"
"${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.client.pb.h"
"${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.server.pb.cc"
"${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.server.pb.h"
COMMAND ${CMAKE_SOURCE_DIR}/tools/venv_exec.sh
ARGS
${Protobuf_PROTOC_EXECUTABLE}
--cpp_out=${CMAKE_BINARY_DIR}/src
--plugin=protoc-gen-service=${CMAKE_SOURCE_DIR}/src/lib/api/service_compiler/plugin.py
--service_out=lang=cpp,make_file_name=${ABS_FIL}:${CMAKE_BINARY_DIR}/src
-I ${PROTOBUF_INCLUDE_DIRS} -I ${CMAKE_SOURCE_DIR}/src ${ABS_FIL}
DEPENDS ${ABS_FIL} ${TEMPLATES}
${CMAKE_CURRENT_SOURCE_DIR}/api/service_compiler/plugin.py
COMMENT "Running C++ service compiler on ${FIL}"
VERBATIM)
endif()
endforeach()
# remove some warnings from protobuf generated files
if(WIN32)
set_source_files_properties(${${SRCS}} PROPERTIES COMPILE_FLAGS
"/wd4309 /wd4838")
else()
set_source_files_properties(
${${SRCS}}
PROPERTIES
COMPILE_FLAGS
"-Wno-unused-parameter -Wno-effc++ -Wno-pedantic -Wno-sign-compare -Wno-uninitialized"
)
endif()
set(${SRCS}
${${SRCS}}
PARENT_SCOPE)
set(${HDRS}
${${HDRS}}
PARENT_SCOPE)
endfunction()
+76
View File
@@ -0,0 +1,76 @@
include_guard(GLOBAL)
function(protobuf_generate_cpp)
set(_options)
set(_one_value_args
CXX_SOURCES
CXX_HEADERS
CXX_OUT_DIR
PROTOC
)
set(_multi_value_args
PROTO_INCLUDE_DIRS
PROTO_SOURCES
)
cmake_parse_arguments(_ARGS "${_options}" "${_one_value_args}" "${_multi_value_args}" ${ARGN})
if (NOT DEFINED _ARGS_PROTOC)
set(_ARGS_PROTOC "${EXTERNALS_BIN}/protoc")
endif()
set(${_ARGS_CXX_SOURCES})
set(${_ARGS_CXX_HEADERS})
set(_proto_include_cli_args)
foreach(_dir ${_ARGS_PROTO_INCLUDE_DIRS})
list(APPEND _proto_include_cli_args -I ${_dir})
endforeach()
foreach(_fil ${_ARGS_PROTO_SOURCES})
get_filename_component(_abs_fil ${_fil} ABSOLUTE)
get_filename_component(_fil_we ${_fil} NAME_WE)
get_filename_component(_fil_dir ${_fil} DIRECTORY)
if(_fil_dir)
set(_fil_we "${_fil_dir}/${_fil_we}")
endif()
list(APPEND ${_ARGS_CXX_SOURCES} "${_ARGS_CXX_OUT_DIR}/${_fil_we}.pb.cc")
list(APPEND ${_ARGS_CXX_HEADERS} "${_ARGS_CXX_OUT_DIR}/${_fil_we}.pb.h")
message(STATUS "add_custom_command for ${_fil} -> ${_ARGS_CXX_OUT_DIR}/${_fil_we}.pb.h")
add_custom_command(
OUTPUT
"${_ARGS_CXX_OUT_DIR}/${_fil_we}.pb.cc"
"${_ARGS_CXX_OUT_DIR}/${_fil_we}.pb.h"
COMMAND
${_ARGS_PROTOC}
ARGS
--cpp_out=${_ARGS_CXX_OUT_DIR} ${_proto_include_cli_args} ${_fil}
DEPENDS ${_abs_fil}
COMMENT "Running C++ protocol buffer compiler on ${_fil}"
VERBATIM
)
endforeach()
set_source_files_properties(
${${_ARGS_CXX_SOURCES}}
${${_ARGS_CXX_HEADERS}}
PROPERTIES
GENERATED TRUE
)
# remove some warnings from protobuf generated files
if(WIN32)
set_source_files_properties(
${${_ARGS_CXX_SOURCES}}
PROPERTIES
COMPILE_FLAGS "/wd4309 /wd4838"
)
else()
set_source_files_properties(
${${_ARGS_CXX_SOURCES}}
PROPERTIES
COMPILE_FLAGS "-Wno-unused-parameter -Wno-effc++ -Wno-pedantic -Wno-sign-compare -Wno-uninitialized"
)
endif()
set(${_ARGS_CXX_SOURCES} ${${_ARGS_CXX_SOURCES}} PARENT_SCOPE)
set(${_ARGS_CXX_HEADERS} ${${_ARGS_CXX_HEADERS}} PARENT_SCOPE)
endfunction()
+58
View File
@@ -0,0 +1,58 @@
function(protobuf_generate_cpp_config)
set(_options)
set(_one_value_args
PROTO_SOURCES_HEADER
PROTO_DEFAULTS_HEADER
)
set(_multi_value_args
PROTO_SOURCES
PROTO_DEFAULTS
)
cmake_parse_arguments(_ARGS "${_options}" "${_one_value_args}" "${_multi_value_args}" ${ARGN})
# Configure the 'config_proto_files' header. It provides an array of strings populated with the
# files in PROTO_SOURCES. At runtime, we use this array to generate a special 'config' proto
# message with fields for each message defined in said files. A global instance of this 'config'
# proto is then constructed, and populated with [compile-time] defaults, and [runtime] overrides
# (i.e., in spark.conf). Calls to `bts::config::get<T>()` retrieve the value from the field
# associated with type `T` in this global instance.
message(STATUS "PROTO_SOURCES: ${_ARGS_PROTO_SOURCES}")
message(STATUS "PROTO_SOURCES_HEADER: ${_ARGS_PROTO_SOURCES_HEADER}")
message(STATUS "PROTO_DEFAULTS_HEADER: ${_ARGS_PROTO_DEFAULTS_HEADER}")
if (_ARGS_PROTO_SOURCES_HEADER)
set(PROTO_SOURCES)
set(PROTO_SOURCES_CXX_ARRAY_ITEMS "")
foreach(_proto_path ${_ARGS_PROTO_SOURCES})
list(APPEND PROTO_SOURCES ${_proto_path})
string(APPEND PROTO_SOURCES_CXX_ARRAY_ITEMS "\"${_proto_path}\",\n")
endforeach()
list(LENGTH PROTO_SOURCES PROTO_SOURCES_LENGTH)
message(STATUS "CONFIGURING PROTO_SOURCES_HEADER): ${_ARGS_PROTO_SOURCES_HEADER}")
configure_file(${CMAKE_SOURCE_DIR}/cmake/config_proto_files.h.in ${_ARGS_PROTO_SOURCES_HEADER})
endif()
# Generate static defaults. Ideally, this would be a single string, but we use an array because
# the compiler complains if the string is longer than 65k.
set(_cfg_defaults_str "const char * config_static_defaults[] = { ")
foreach(_cfg_file_path ${_ARGS_PROTO_DEFAULTS})
file(READ ${_cfg_file_path} _cfg_file_content)
string(REPLACE "\"" "\\\"" _cfg_file_content_escaped ${_cfg_file_content})
string(REPLACE "\n" ";" _cfg_lines ${_cfg_file_content_escaped})
foreach(_line ${_cfg_lines})
string(APPEND _cfg_defaults_str "\n \"${_line}\",")
endforeach(_line)
endforeach(_cfg_file_path)
string(APPEND _cfg_defaults_str "};")
set(_old_cfg_defaults_str "")
if(EXISTS ${_ARGS_PROTO_DEFAULTS_HEADER})
file(READ ${_ARGS_PROTO_DEFAULTS_HEADER} _old_cfg_defaults_str)
endif()
if(NOT _cfg_defaults_str STREQUAL _old_cfg_defaults_str)
file(WRITE "${_ARGS_PROTO_DEFAULTS_HEADER}" "${_cfg_defaults_str}")
endif()
endfunction()
+49
View File
@@ -0,0 +1,49 @@
include_guard(DIRECTORY)
function(protobuf_generate_fs)
set(_options)
set(_one_value_args TARGET_NAME OUT_DIR PROTOC)
set(_multi_value_args PROTO_INCLUDE_DIRS PROTO_SOURCES)
cmake_parse_arguments(_ARGS "${_options}" "${_one_value_args}" "${_multi_value_args}" ${ARGN})
if (NOT DEFINED _ARGS_TARGET_NAME)
set(_ARGS_TARGET_NAME "${PROJECT_NAME}_generate_fs_protos")
endif()
if (NOT DEFINED _ARGS_PROTOC)
set(_ARGS_PROTOC "${EXTERNALS_BIN}/protoc")
endif()
set(_proto_include_cli_args)
foreach(_dir ${_ARGS_PROTO_INCLUDE_DIRS})
list(APPEND _proto_include_cli_args -I ${_dir})
endforeach()
set(_outputs)
foreach(_proto_source ${_ARGS_PROTO_SOURCES})
# get_filename_component(_abs_proto_source ${_proto_source} ABSOLUTE)
get_filename_component(_proto_source_we ${_proto_source} NAME_WE)
# get_filename_component(_proto_source_dir ${_proto_source} DIRECTORY)
list(APPEND _outputs ${_ARGS_OUT_DIR}/${_proto_source_we}.fable.fs)
endforeach()
set(_venv_exec ${CMAKE_SOURCE_DIR}/tools/venv_exec.sh)
set(_plugin_py ${CMAKE_SOURCE_DIR}/src/lib_spark/spark/api/service_compiler/plugin.py)
set(_template_fs ${CMAKE_SOURCE_DIR}/src/lib_spark/spark/api/service_compiler/templates/fs/fable.fs)
add_custom_command(
OUTPUT ${_outputs}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
COMMAND ${_venv_exec}
${_ARGS_PROTOC}
--plugin=protoc-gen-service=${_plugin_py}
--service_out=lang=fs,strip_out_path=1:${_ARGS_OUT_DIR}
${_proto_include_cli_args}
${_ARGS_PROTO_SOURCES}
DEPENDS
${_venv_exec}
${_plugin_py}
${_template_fs}
${CMAKE_CURRENT_SOURCE_DIR}/package-lock.json
${_ARGS_PROTO_SOURCES}
)
add_custom_target(${_ARGS_TARGET_NAME} ALL DEPENDS "${_outputs}")
endfunction()
+33
View File
@@ -0,0 +1,33 @@
include_guard(DIRECTORY)
function(protobuf_generate_js)
set(_options)
set(_one_value_args TARGET_NAME OUTPUT)
set(_multi_value_args PROTO_INCLUDE_DIRS PROTO_SOURCES)
cmake_parse_arguments(_ARGS "${_options}" "${_one_value_args}" "${_multi_value_args}" ${ARGN})
if (NOT DEFINED _ARGS_TARGET_NAME)
set(_ARGS_TARGET_NAME "${PROJECT_NAME}_generate_js_protos")
endif()
set(_proto_include_cli_args)
foreach(_dir ${_ARGS_PROTO_INCLUDE_DIRS})
list(APPEND _proto_include_cli_args -p ${_dir})
endforeach()
add_custom_command(
OUTPUT ${_ARGS_OUTPUT}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
COMMAND npx
--no-install pbjs
-t json-module
-w commonjs
-o ${_ARGS_OUTPUT}
${_proto_include_cli_args}
${_ARGS_PROTO_SOURCES}
DEPENDS
${CMAKE_CURRENT_SOURCE_DIR}/package-lock.json
${_ARGS_PROTO_SOURCES}
)
add_custom_target(${_ARGS_TARGET_NAME} ALL DEPENDS ${_ARGS_OUTPUT})
endfunction()
+10
View File
@@ -0,0 +1,10 @@
include_guard(GLOBAL)
macro(set_first_defined RESULT)
foreach(_arg ${ARGN})
if (DEFINED ${_arg})
set(${RESULT} ${${_arg}})
break()
endif()
endforeach()
endmacro()
+51
View File
@@ -0,0 +1,51 @@
include_guard(DIRECTORY)
include(${CMAKE_SOURCE_DIR}/cmake/TernaryOperator.cmake)
function(sym_link_dir)
set(_options)
set(_one_value_args TARGET_NAME SOURCE DEST COMMENT)
set(_multi_value_args)
cmake_parse_arguments(_ARGS "${_options}" "${_one_value_args}" "${_multi_value_args}" ${ARGN})
add_custom_command(
OUTPUT ${_ARGS_DEST}
COMMAND ln ARGS -sf ${_ARGS_SOURCE} ${_ARGS_DEST}
)
set_ternary(_comment _ARGS_COMMENT "${_ARGS_COMMENT}" "linking dir ${_ARGS_SOURCE} -> ${_ARGS_DEST}")
add_custom_target(
${_ARGS_TARGET_NAME} ALL
DEPENDS ${_ARGS_DEST}
COMMENT "${_comment}"
)
endfunction()
function(sym_link_file)
set(_options)
set(_one_value_args TARGET_NAME SOURCE DEST INSTALL_DEST)
set(_multi_value_args)
cmake_parse_arguments(_ARGS "${_options}" "${_one_value_args}" "${_multi_value_args}" ${ARGN})
add_custom_command(
OUTPUT ${_ARGS_DEST}
COMMAND ln ARGS -s ${_ARGS_SOURCE} ${_ARGS_DEST}
)
add_custom_target(
${_ARGS_TARGET_NAME} ALL
DEPENDS ${_ARGS_DEST}
)
if (DEFINED _ARGS_INSTALL_DEST)
install(
FILES
${_ARGS_SOURCE}
DESTINATION ${_ARGS_INSTALL_DEST}
)
endif()
endfunction()
function(sym_link_local_lib LIB_NAME)
set(_lib_file_name "${CMAKE_SHARED_LIBRARY_PREFIX}${LIB_NAME}.so")
file(
CREATE_LINK
"${CMAKE_BINARY_DIR}/src/lib_${LIB_NAME}/${_lib_file_name}"
"${CMAKE_CURRENT_BINARY_DIR}/${_lib_file_name}"
SYMBOLIC
)
endfunction()
+9
View File
@@ -0,0 +1,9 @@
include_guard(GLOBAL)
macro(set_ternary RESULT CONDITION TRUE_VALUE FALSE_VALUE)
if(${CONDITION})
set(${RESULT} ${TRUE_VALUE})
else()
set(${RESULT} ${FALSE_VALUE})
endif()
endmacro()
+11
View File
@@ -0,0 +1,11 @@
#pragma once
#include <array>
namespace bts::@PROJECT_NAME@::protobufs {
inline std::array<const char *, @PROTO_SOURCES_LENGTH@> config_proto_files = {
@PROTO_SOURCES_CXX_ARRAY_ITEMS@
};
} // namespace bts::@PROJECT_NAME@::protobufs
+51
View File
@@ -0,0 +1,51 @@
services:
sparkd:
build:
dockerfile: Dockerfile
context: ./sparkd
command: [ "./sparkd", "run" ]
ports: [ "8000:8000", "8001:8001", "8002:8002" ]
pubsub:
build:
dockerfile: pubsub.Dockerfile
context: ./DataStream
command: [ "./PubSubServer" ]
environment:
- DOTNET_ENVIRONMENT=Production
ports:
- 22019:22019
md-publisher:
build:
dockerfile: md-publisher.Dockerfile
context: ./DataStream
depends_on:
- pubsub
- sparkd
command:
[
"dockerize",
"-wait",
"tcp://pubsub:22019",
"-wait",
"tcp://sparkd:8000",
"-timeout",
"10s",
"./DataStreamPublisher"
]
environment:
- DOTNET_ENVIRONMENT=MarketData
webapp:
ports:
- 8080:8080
- 8443:8443
build:
dockerfile: webapp.Dockerfile
context: ./DataStream
command: [ "./Bts.DataStreamWebApp.Server" ]
environment:
- ASPNETCORE_HTTP_PORT=8080
- ASPNETCORE_HTTPS_PORT=8443
- ASPNETCORE_ENVIRONMENT=Staging
+72
View File
@@ -0,0 +1,72 @@
services:
sparkd:
build:
dockerfile: src/server/sparkd/Dockerfile
command: [ "./sparkd", "run" ]
ports: [ "8000:8000", "8001:8001", "8002:8002" ]
pubsub:
environment:
- DOTNET_ENVIRONMENT=Staging
build:
dockerfile: src/DataStreamSuite/PubSubServer/Dockerfile
ports:
- 22019:22019
command: [ "./PubSubServer" ]
md-publisher:
build:
dockerfile: src/DataStreamSuite/DataStreamPublisher/Dockerfile
depends_on:
- pubsub
- sparkd
command:
[
"dockerize",
"-wait",
"tcp://pubsub:22019",
"-wait",
"tcp://sparkd:8000",
"-timeout",
"10s",
"./DataStreamPublisher"
]
environment:
- DOTNET_ENVIRONMENT=MarketData
webapp:
ports:
- 8080:8080
- 8443:8443
build:
dockerfile: src/DataStreamSuite/Bts.DataStreamWebApp.Server/Dockerfile
command: [ "./Bts.DataStreamWebApp.Server" ]
environment:
- ASPNETCORE_ENVIRONMENT=Staging
api-server:
profiles:
- api-server
depends_on:
- sparkd
ports:
- 8008:8008
- 9443:9443
build:
dockerfile: src/client/spark_api_server/Dockerfile
command: [ "./spark_api_server", "run" ]
environment:
- ASPNETCORE_ENVIRONMENT=Staging
sparkd-ping-test:
depends_on:
- sparkd
profiles:
- ping-test
build:
context: .
dockerfile_inline: |
FROM bluetradingsystems/linux_dotnet7
USER "root"
RUN apt-get update && apt-get -y install iputils-ping
command: [ "ping", "sparkd" ]
+1 -1
View File
@@ -53,7 +53,7 @@ The status callback is called when the market for this security opens, closes, o
// Subscribe to security trading status
void IAlgoServices::subscribe_status(SecurityKey);
// You implement
virtual void on_status(SecurityKey, co2::TradingStatus) override;
virtual void on_status(SecurityKey, domain::protobufs::TradingStatus) override;
```
# Child Orders
Submodule docker/linux deleted from 56c61231f3.
+39
View File
@@ -0,0 +1,39 @@
FROM ubuntu:22.04
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
&& apt-get install -y pkg-config \
&& apt-get install -y locales \
&& apt-get install -y ca-certificates \
&& apt-get install -y libssh2-1-dev \
&& apt-get install -y software-properties-common \
&& apt-get install -y sudo
# GCC
ARG gcc_major=13
RUN add-apt-repository ppa:ubuntu-toolchain-r/test \
&& apt-get update \
&& apt-get install -y g++-$gcc_major
RUN apt-get update \
&& apt-get install -y dotnet7
RUN locale-gen en_US.UTF-8
ARG docker_user="bts"
ARG docker_uid="1000"
ARG docker_pass="bts"
RUN \
groupadd -g "$docker_uid" "$docker_user" \
&& useradd \
-m \
-u "$docker_uid" \
-g "$docker_uid" \
-G sudo \
--password "$(openssl passwd -1 "$docker_pass")" \
"$docker_user"
USER bts
WORKDIR /home/bts
Submodule docker/windows deleted from de7d40c09d.
@@ -8,7 +8,6 @@
public int Integer;
public double Real;
public string Identifier;
public DateTime DateTime;
public ImmutableList<IArithmeticExpression> Arguments;
public ImmutableList<Assignment> Assignments;
public IArithmeticExpression Argument;
@@ -1,4 +1,6 @@
using System;
using System.Runtime.Versioning;
using Microsoft.Win32;
namespace BCG
@@ -10,6 +12,7 @@ namespace BCG
rk.SetValue (name, value.ToString ());
}
[SupportedOSPlatform("windows")]
public static double getValue (RegistryKey rk, string name)
{
object v = rk.GetValue (name);
@@ -1,19 +1,11 @@
namespace Bts.BooleanExpressions;
using bts;
class BooleanExpression : IBooleanExpression {
readonly Func<IDictionary<string, object?>, bool> evaluate;
public BooleanExpression(Func<IDictionary<string, object?>, bool> evaluate) =>
this.evaluate = evaluate;
public bool Evaluate(IDictionary<string, object?> env) {
try {
return evaluate(env);
} catch (Exception ex) {
Log.assert(false, $"BooleanExpression.Evaluate failed: {ex}");
throw;
}
}
public bool Evaluate(IDictionary<string, object?> env) =>
evaluate(env);
}
@@ -47,11 +47,6 @@ IsOfType '/'
%%
{StringConstant} {
yylval.String = yytext.Substring(1, yytext.Length - 2).Replace(@"\'", "'");
return (int)Token.STRING_CONSTANT;
}
{BooleanConstant} {
yylval.Boolean = yytext.ToLower() switch {
"true" => true,
@@ -62,6 +57,11 @@ IsOfType '/'
return (int)Token.BOOLEAN_CONSTANT;
}
{StringConstant} {
yylval.String = yytext.Substring(1, yytext.Length - 2).Replace(@"\'", "'");
return (int)Token.STRING_CONSTANT;
}
{NullConstant} {
yylval.Null = DBNull.Value;
return (int)Token.NULL_CONSTANT;
@@ -0,0 +1 @@
VITE_DATASTREAM_WEBSOCKET_URL=wss://localhost:443
@@ -0,0 +1 @@
VITE_DATASTREAM_WEBSOCKET_URL=wss://localhost:443
@@ -0,0 +1,32 @@
module.exports = {
root: true,
env: { browser: true, es2020: true },
extends: [
'eslint:recommended',
'plugin:react/recommended',
'plugin:react/jsx-runtime',
'plugin:react-hooks/recommended',
],
ignorePatterns: ['dist', '.eslintrc.cjs'],
parserOptions: { ecmaVersion: 'latest', sourceType: 'module' },
settings: { react: { version: '18.3.1' } },
plugins: ['react-refresh'],
rules: {
"import/order": [
"error",
{
"groups": [
"builtin",
"external",
"internal",
["parent", "sibling", "index"]
],
"newlines-between": "always"
}
],
'react-refresh/only-export-components': [
'warn',
{ allowConstantExport: true },
],
},
}
@@ -0,0 +1,24 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-ssr
*.local
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?
@@ -0,0 +1,11 @@
<Project Sdk="Microsoft.VisualStudio.JavaScript.Sdk/0.5.128-alpha">
<PropertyGroup>
<StartupCommand>npm run dev</StartupCommand>
<JavaScriptTestRoot>src\</JavaScriptTestRoot>
<JavaScriptTestFramework>Jest</JavaScriptTestFramework>
<!-- Allows the build (or compile) script located on package.json to run on Build -->
<ShouldRunBuildScript>true</ShouldRunBuildScript>
<!-- Folder where production build objects will be placed -->
<PublishAssetsDirectory>$(MSBuildProjectDirectory)\distill</PublishAssetsDirectory>
</PropertyGroup>
</Project>
@@ -0,0 +1,8 @@
# React + Vite
This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.
Currently, two official plugins are available:
- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react/README.md) uses [Babel](https://babeljs.io/) for Fast Refresh
- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react-swc) uses [SWC](https://swc.rs/) for Fast Refresh
@@ -0,0 +1,13 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/ico" href="/logo.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>BTS Cloud Streams</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="/src/main.jsx"></script>
</body>
</html>
@@ -0,0 +1,10 @@
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<clear />
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
</packageSources>
<disabledPackageSources>
<clear />
</disabledPackageSources>
</configuration>
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,44 @@
{
"name": "bts.datastreamwebapp.client",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"lint": "eslint . --ext js,jsx --report-unused-disable-directives --max-warnings 0",
"preview": "vite preview"
},
"dependencies": {
"@angular/material": "^18.0.4",
"@emotion/react": "^11.11.4",
"@emotion/styled": "^11.11.5",
"@fontsource/noto-sans": "^5.0.22",
"@material/data-table": "^15.0.0-canary.7f224ddd4.0",
"@mui/icons-material": "^5.15.20",
"@mui/material": "^5.15.20",
"@mui/x-data-grid": "^7.7.1",
"bulma": "^1.0.1",
"classnames": "^2.5.1",
"react": "^18.3.1",
"react-data-grid": "^7.0.0-beta.44",
"react-data-table-component": "^7.6.2",
"react-dom": "^18.3.1",
"react-json-view-lite": "^1.4.0",
"react-pro-sidebar": "^1.1.0",
"react-use-websocket": "^4.8.1",
"rpc-websockets": "^7.11.2"
},
"devDependencies": {
"@types/react": "^18.3.3",
"@types/react-dom": "^18.3.0",
"@vitejs/plugin-react": "^4.3.1",
"eslint": "^8.57.0",
"eslint-plugin-import": "^2.29.1",
"eslint-plugin-react": "^7.34.2",
"eslint-plugin-react-hooks": "^4.6.2",
"eslint-plugin-react-refresh": "^0.4.7",
"react-mui-sidebar": "^1.3.5",
"vite": "^5.2.13"
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 8.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 107 KiB

@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" class="iconify iconify--logos" width="31.88" height="32" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 257"><defs><linearGradient id="IconifyId1813088fe1fbc01fb466" x1="-.828%" x2="57.636%" y1="7.652%" y2="78.411%"><stop offset="0%" stop-color="#41D1FF"></stop><stop offset="100%" stop-color="#BD34FE"></stop></linearGradient><linearGradient id="IconifyId1813088fe1fbc01fb467" x1="43.376%" x2="50.316%" y1="2.242%" y2="89.03%"><stop offset="0%" stop-color="#FFEA83"></stop><stop offset="8.333%" stop-color="#FFDD35"></stop><stop offset="100%" stop-color="#FFA800"></stop></linearGradient></defs><path fill="url(#IconifyId1813088fe1fbc01fb466)" d="M255.153 37.938L134.897 252.976c-2.483 4.44-8.862 4.466-11.382.048L.875 37.958c-2.746-4.814 1.371-10.646 6.827-9.67l120.385 21.517a6.537 6.537 0 0 0 2.322-.004l117.867-21.483c5.438-.991 9.574 4.796 6.877 9.62Z"></path><path fill="url(#IconifyId1813088fe1fbc01fb467)" d="M185.432.063L96.44 17.501a3.268 3.268 0 0 0-2.634 3.014l-5.474 92.456a3.268 3.268 0 0 0 3.997 3.378l24.777-5.718c2.318-.535 4.413 1.507 3.936 3.838l-7.361 36.047c-.495 2.426 1.782 4.5 4.151 3.78l15.304-4.649c2.372-.72 4.652 1.36 4.15 3.788l-11.698 56.621c-.732 3.542 3.979 5.473 5.943 2.437l1.313-2.028l72.516-144.72c1.215-2.423-.88-5.186-3.54-4.672l-25.505 4.922c-2.396.462-4.435-1.77-3.759-4.114l16.646-57.705c.677-2.35-1.37-4.583-3.769-4.113Z"></path></svg>

After

Width:  |  Height:  |  Size: 1.5 KiB

@@ -0,0 +1,40 @@
import { Sidebar, Menu, MenuItem, Submenu, Logo } from "react-mui-sidebar";
import './App.css';
const AnimatedSideBar = () => {
return (
<Sidebar className={"sidebar-container"} width={"270px"}>
<Logo img="https://adminmart.com/wp-content/uploads/2024/03/logo-admin-mart-news.png">
AdminMart
</Logo>
<Menu subHeading="HOME">
<MenuItem link="/" badge="true">
Modern
</MenuItem>
<MenuItem>eCommerce</MenuItem>
<MenuItem>Analytical</MenuItem>
</Menu>
<Menu subHeading="APPS">
<MenuItem>Chat</MenuItem>
<MenuItem>Calendar</MenuItem>
</Menu>
<Menu subHeading="OTHERS">
<Submenu title="Menu Level">
<MenuItem>Post</MenuItem>
<MenuItem>Details</MenuItem>
<Submenu title="Level 2">
<MenuItem>new</MenuItem>
<MenuItem>Hello</MenuItem>
</Submenu>
</Submenu>
<MenuItem>Chip</MenuItem>
<MenuItem target="_blank" link="google.com">
External Link
</MenuItem>
</Menu>
</Sidebar>
);
};
export default AnimatedSideBar;
@@ -0,0 +1,101 @@
@use "@material/checkbox";
@use "@material/icon-button";
@use "@material/data-table/data-table";
.root-div {
display: flex;
flex-direction: row;
align-items: flex-start;
}
.main-content-div {
display: flex;
flex-direction: row;
}
.title-div {
width: 100%;
}
.container {
display: flex;
}
.container-column {
display: flex;
flex-direction: column;
}
.container-row {
display: flex;
flex-direction: row;
align-items: stretch;
}
box.query-input-form {
/* p: 2, display: "flex", flexFlow: "column", alignItems: "stretch" */
display: flex;
flex-flow: column;
align-items: stretch;
}
/* li.ps-menuitem-root:hover {
background-color: #ff9922;
} */
.container-row-centered {
display: flex;
flex-direction: row;
justify-content: center;
align-items: center;
}
h1#main-header {
padding: 4px 8px 4px 8px;
text-align: center;
}
td:hover {
background-color: cadetblue;
}
td#cell-decimal {
text-align: right;
}
td#cell-integer {
text-align: right;
}
td#cell-string {
text-align: center;
}
td#cell-date-time {
text-align: center;
}
/* Sidebar.css */
/* .sidebar-container {
display: flex;
flex-direction: column;
}
.sidebar {
position: relative;
top: 0;
left: 0;
height: 100vh;
background-color: #333;
color: white;
overflow: hidden;
transition: transform 0.3s ease;
}
.sidebar.closed {
width: 0;
}
.sidebar.open {
width: 250px;
} */
@@ -0,0 +1,173 @@
import { ThemeProvider, CssBaseline, GlobalStyles } from '@mui/material';
import QueryStatusView from './QueryStatusView';
import useMediaQuery from '@mui/material/useMediaQuery';
import QueryControl from './QueryControl';
import DataTable from './DataTable';
import { useState, useEffect, useMemo } from 'react';
import useWebSocket, { ReadyState } from 'react-use-websocket';
import Sidebar from './SideBar';
import getTheme from './Theme';
import '@fontsource/noto-sans';
import './App.css';
import './SideBar.css';
const webSocketUrl = import.meta.env.VITE_DATASTREAM_WEBSOCKET_URL;
// Always use the same subscription id (1), because we only show 1 table at a time.
// The server will automatically dispose the previous subscription and start a new one
// if we're currently subscribed to another query.
const subscriptionId = "1";
var rpcReqIdCounter = 1;
function subscribeReq(query) {
return {
jsonrpc: "2.0",
id: rpcReqIdCounter++,
method: "subscribe",
params: {
subscription_id: subscriptionId,
query: query
}
};
}
function unsubscribeReq() {
return {
jsonrpc: "2.0",
id: rpcReqIdCounter++,
method: "unsubscribe",
params: {
subscription_id: subscriptionId,
}
};
}
const App = () => {
const prefersDarkMode = useMediaQuery('(prefers-color-scheme: dark)');
const theme = getTheme({ mode: prefersDarkMode ? 'dark' : 'light' });
const globalStyles = {
':root': {
'--primary-color': theme.palette.primary.main,
'--primary-dark-color': theme.palette.primary.dark,
'--secondary-color': theme.palette.secondary.main,
'--secondary-dark-color': theme.palette.secondary.dark,
'--background-color': theme.palette.background.default,
},
};
const [blueTable, setBlueTable] = useState({ columns: null, rows: null });
const [queryError, setQueryError] = useState(null);
const [queryState, setQueryState] = useState("Unsubscribed");
const queryStateDetails = useMemo(() => {
if (queryState == "Unsubscribed") {
return "Enter query above, then click the play button to subscribe";
}
if (queryState == "Subscribed") {
return null;
}
}, [queryState]);
const [sidebarIsOpen, setSidebarIsOpen] = useState(false);
const [cannedQuery, setCannedQuery] = useState(null);
const onCannedQueryClicked = (cannedQuery) => {
console.info(`Executing canned query: ${JSON.stringify(cannedQuery)}`);
const queryText = cannedQuery.query.join("\n");
setCannedQuery({ queryText: queryText, clickTime: Date.now() });
subscribe(queryText);
};
const { sendJsonMessage, getWebSocket, readyState } = useWebSocket(
webSocketUrl,
{
onOpen: (e) => console.info(`WebSocket open. ${JSON.stringify(e)}`),
onClose: (e) => console.info(`WebSocket closed. ${JSON.stringify(e)}`),
onError: (e) => console.error(`WebSocket error: ${JSON.stringify(e)}`),
shouldReconnect: () => true,
});
useEffect(() => {
console.info(`WebSocket readyState: ${readyState}`)
if (readyState != ReadyState.OPEN) {
return;
}
const ws = getWebSocket();
ws.onmessage = e => {
console.info(`Received message: ${e.data}`);
const msg = JSON.parse(e.data);
setQueryError(msg.error);
if (msg.error != null) {
setQueryState("Error");
} else if (msg.id != null) {
setQueryState(msg.result.status);
} else {
console.assert(msg.params.subscription_id == subscriptionId);
console.info(`Received rows: ${msg.params.table.all}`);
setBlueTable({ columns: msg.params.table.columns, rows: msg.params.table.all });
}
};
}, [getWebSocket, readyState]);
const subscribe = (query) => {
const req = subscribeReq(query);
console.info(`Subscribing: ${JSON.stringify(subscribeReq(query))}`);
sendJsonMessage(req);
};
const unsubscribe = () => {
const req = unsubscribeReq();
console.info(`Unsubscribing: ${JSON.stringify(req)}`);
sendJsonMessage(req);
};
const onOpened = () => {
setSidebarIsOpen(true);
};
const onClosed = () => {
setSidebarIsOpen(false);
};
const sidebarWidth = sidebarIsOpen ? '250px' : '65px';
return (
<ThemeProvider theme={theme}>
<CssBaseline />
<GlobalStyles styles={globalStyles} />
<div className='root-div'>
<div style={{
display: 'flex',
flexDirection: 'column',
alignItems: 'flex-start',
flexWrap: 'nowrap',
width: sidebarWidth,
transition: 'width 0.3s ease',
}}>
<Sidebar onSelection={onCannedQueryClicked} onOpened={onOpened} onClosed={onClosed} />
</div>
<div style={{
display: 'flex',
flexDirection: 'column',
left: sidebarWidth,
width: '100%',
transition: 'width 0.3s ease',
}}>
<div style={{ width: '100%', display: 'flex', flexDirection: 'column', alignItems: 'flex-start', margin: theme.spacing(2, 0, 0, 0) }}>
<div style={{ width: '90%', rightMargin: '100px' }}>
<QueryControl
queryState={queryState}
queryText={cannedQuery?.queryText}
queryTextResetTime={cannedQuery?.clickTime}
subscribe={subscribe}
unsubscribe={unsubscribe} />
</div>
<div style={{ width: '90%', rightMargin: '100px' }}>
<QueryStatusView queryState={queryState} details={queryStateDetails} error={queryError} />
<DataTable columns={blueTable?.columns ?? []} rows={blueTable?.rows ?? []} />
</div>
</div>
</div>
</div>
</ThemeProvider>
);
};
export default App;
@@ -0,0 +1,53 @@
@use "@material/checkbox";
@use "@material/icon-button";
@use "@material/data-table/data-table";
.data-grid-container {
overflow: auto;
white-space: nowrap;
}
.data-grid {
max-height: 1000px;
}
div[role=columnheader] {
border-top: 0px;
border-bottom: 0px;
background-color: var(--primary-dark-color);
}
/* div[role=columnheader] {
border-top: 0px;
border-bottom: 0px;
background-color: var(--primary-dark-color);
}
.rdg-row {
border: 0px;
border-top: 0px;
border-bottom: 0px;
background-color: rgba(var(--primary-dark-color), 0.8);
color: var(--primary-color);
}
.rdg-row-odd {
border: 0px;
border-top: 0px;
border-bottom: 0px;
background-color: var(--primary-dark-color);
color: var(--primary-color);
}
.rdg-cell {
border-bottom: 0px;
}
.rdg-cell.even {
border-bottom: 5px;
}
.rdg-cell.odd {
border-bottom: 5px;
} */
@@ -0,0 +1,32 @@
@use "@material/checkbox";
@use "@material/icon-button";
@use "@material/data-table/data-table";
th {
font-size: large;
padding: 15px;
}
td {
padding: 15px;
}
td:hover {
background-color: cadetblue;
}
td#cell-decimal {
text-align: right;
}
td#cell-integer {
text-align: right;
}
td#cell-string {
text-align: center;
}
td#cell-date-time {
text-align: center;
}
@@ -0,0 +1,74 @@
import PropTypes from 'prop-types';
import { useTheme } from '@mui/material/styles';
import { DataGrid } from '@mui/x-data-grid';
import './DataGrid.css'
import { useMemo, useState } from 'react';
export default function DataTable({ columns, rows }) {
const theme = useTheme();
const [columnWidths, setColumnWidths] = useState({});
const handleColumnResize = (params) => {
const newWidth = params.width;
setColumnWidths((prevWidths) => ({
...prevWidths,
[params.colDef.field]: params.width,
}));
};
const getGridColDef = useMemo(() => {
return col => {
return {
field: col.name,
headerName: col.name,
width: columnWidths[col.name],
};
}
}, [columnWidths]);
return (
<DataGrid
className='data-grid-container'
autosizeOnMount={false}
columns={columns?.map(getGridColDef) ?? []}
onColumnResize={handleColumnResize}
rows={rows ?? []}
sx={{
margin: theme.spacing(2, 0, 0, 0),
borderColor: 'primary.main',
backgroundColor: 'primary.dark',
color: 'text.primary',
'& .MuiDataGrid-columnHeader': {
color: 'primary.main',
fontWeight: 700,
backgroundColor: 'primary.dark'
},
'& .MuiDataGrid-filler': {
color: 'primary.main',
fontWeight: 700,
backgroundColor: 'primary.dark'
},
'& .MuiDataGrid-cell': {
color: 'text.primary',
},
'& .MuiDataGrid-cell:hover': {
color: 'primary.main',
},
'& .MuiDataGrid-row': {
'--DataGrid-rowBorderColor': 'secondary.dark'
}
}}
/>
);
}
DataTable.propTypes = {
columns: PropTypes.arrayOf(
PropTypes.shape({
name: PropTypes.string.isRequired,
type: PropTypes.string.isRequired,
})
).isRequired,
rows: PropTypes.arrayOf(PropTypes.array).isRequired,
apiRef: PropTypes.object,
};
@@ -0,0 +1,119 @@
import PropTypes from 'prop-types';
import DataTable from './DataTable';
import { useRef, useState, useEffect } from 'react';
import { useGridApiRef } from '@mui/x-data-grid';
import Box from '@mui/material/Box';
import Button from '@mui/material/Button';
import Grid from '@mui/material/Grid';
import Paper from '@mui/material/Paper';
import PlayArrowIcon from '@mui/icons-material/PlayArrow';
import StopIcon from '@mui/icons-material/Stop';
import TextField from '@mui/material/TextField';
import Tooltip from '@mui/material/Tooltip';
const QueryControl = (props) => {
const { queryState, queryText, queryTextResetTime, subscribe, unsubscribe } = props;
// const theme = useTheme();
let textInput = useRef();
const [queryTextChangeTime, setQueryTextChangeTime] = useState(null);
useEffect(() => {
if (queryText == null || textInput.current == null) {
return;
}
// Init text field (unless a non-empty string is already present, or the user has modified
// the query - we don't want to overwrite their changes).
if (queryTextChangeTime == null || queryTextResetTime > queryTextChangeTime) {
console.info(`Initializing query text: ${queryText}`)
textInput.current.value = queryText;
}
}, [queryText, queryTextChangeTime, queryTextResetTime]);
const onClearClick = () => {
textInput.current.value = '';
};
const onRunClick = () => {
subscribe(textInput.current.value);
};
const onStopClick = () => {
unsubscribe();
};
const onKeyDown = (e) => {
// Detect keyboard shortcuts.
// Ctrl+Enter : Run
if (e.ctrlKey && e.key === 'Enter') {
onRunClick();
}
};
const onChange = (e) => {
// So we know not to overwrite the changes with the canned query on re-render
setQueryTextChangeTime(Date.now());
};
return (
<Box sx={{ display: "flex", flexFlow: "column" }}>
<Paper elevation={3} sx={{ color: 'primary.main' }}>
<Box sx={{ p: 2 }}>
<TextField
fullWidth
multiline
rows={10}
variant="outlined"
onChange={onChange}
onKeyDown={onKeyDown}
inputRef={textInput}
placeholder="Enter Query..."
/>
</Box>
<Box sx={{ p: 2, borderTop: 1 }}>
<Grid container spacing={2} justifyContent="flex-start">
<Grid item xs>
<Tooltip title="Clear Query Text">
<Button onClick={onClearClick} sx={{ padding: '11px', color: 'text.dark' }}>
Clear
</Button>
</Tooltip>
</Grid>
<Grid item>
<Tooltip title="Run (Ctrl+Enter)">
<Button onClick={onRunClick} sx={{
color: `queryStartButton.${queryState}.color`,
bgcolor: `queryStartButton.${queryState}.backgroundColor`,
padding: '11px'
}}>
<PlayArrowIcon />
</Button>
</Tooltip>
</Grid>
<Grid item>
<Tooltip title="Stop">
<Button onClick={onStopClick} sx={{
color: `queryStopButton.${queryState}.color`,
bgcolor: `queryStopButton.${queryState}.backgroundColor`,
padding: '11px'
}}>
<StopIcon />
</Button>
</Tooltip>
</Grid>
</Grid>
</Box>
</Paper>
</Box>
);
};
QueryControl.propTypes = {
columns: PropTypes.array,
rows: PropTypes.array,
isRunning: PropTypes.bool,
queryText: PropTypes.string,
queryTextResetTime: PropTypes.number,
queryState: PropTypes.string,
subscribe: PropTypes.func,
unsubscribe: PropTypes.func,
};
export default QueryControl;
@@ -0,0 +1,73 @@
import { useState } from 'react';
import PropTypes from 'prop-types';
import { Box, Typography, Collapse } from '@mui/material';
import Alert from '@mui/material/Alert';
import AlertTitle from '@mui/material/AlertTitle';
import { useTheme } from '@mui/material/styles';
import { alpha } from '@mui/system';
const QueryStatusView = ({ queryState, details, error }) => {
const theme = useTheme();
const [open, setOpen] = useState(false);
let header = queryState;
let headerContent = null;
let detailElements = [];
let severity = queryState == "Subscribed" ? "success" : "info";
let cursor = details == null ? "default" : "pointer";
if (error != null) {
severity = "error";
cursor = "pointer";
header = `Error Code: ${error.code}`;
headerContent = error.message;
detailElements.push(<Typography variant="body1">{error.data.error_message}</Typography>);
detailElements.push(<Typography variant="subtitle1">Stack Trace:</Typography>);
(error.data.stack_trace ?? []).forEach(line => detailElements.push(
<Typography variant="body2" component="pre" sx={{ whiteSpace: 'pre-wrap' }}>
{line}
</Typography>));
} else if (details != null) {
detailElements.push(<Typography variant="body1">{details}</Typography>);
}
return (
<Box sx={{ margin: theme.spacing(2, 0, 0, 0) }}>
<Alert
severity={severity}
onClick={() => setOpen(!open)}
sx={{
margin: theme.spacing(2, 0, 0, 0),
cursor: cursor,
}}>
<AlertTitle>{header}</AlertTitle>
{headerContent}
</Alert>
{detailElements.length == 0
? null
: <Collapse in={open}>
<Box
sx={{
backgroundColor: alpha(theme.palette[severity].light, 0.75),
color: theme.palette[severity].contrastText,
padding: theme.spacing(2),
marginTop: theme.spacing(0),
borderRadius: theme.shape.borderRadius,
}}
>
{detailElements}
</Box>
</Collapse>}
</Box>
);
};
QueryStatusView.propTypes = {
queryState: PropTypes.string,
details: PropTypes.string,
error: PropTypes.shape({
code: PropTypes.number,
message: PropTypes.string,
data: PropTypes.object,
})
};
export default QueryStatusView;
@@ -0,0 +1,27 @@
import 'react-data-grid/lib/styles.css';
import PropTypes from 'prop-types';
import DataGrid from 'react-data-grid';
import './DataGrid.css';
function getGridColDef(col) {
return {
key: col.name,
name: col.name,
};
}
export default function ReactDataGrid({columns, rows}) {
return (
<DataGrid className={"data-grid"} rows={rows} columns={columns.map(getGridColDef)} />
)
}
ReactDataGrid.propTypes = {
columns: PropTypes.arrayOf(
PropTypes.shape({
name: PropTypes.string.isRequired,
type: PropTypes.string.isRequired,
})
).isRequired,
rows: PropTypes.arrayOf(PropTypes.array).isRequired,
};
@@ -0,0 +1,98 @@
/* Sidebar.css */
.app-container {
display: flex;
width: 100%;
overflow: hidden;
flex-direction: row;
justify-content: flex-start;
align-items: flex-start;
}
.sidebar {
display: flex;
flex-direction: row;
justify-content: flex-start;
align-items: flex-start;
background-color: transparent;
color: white;
overflow: hidden;
transition: width 0.3s ease;
}
.sidebar.open {
width: 100%;
}
.sidebar.closed {
width: 0%;
}
.sidebar-toggle-button {
border: 0px;
border-radius: 50%;
box-shadow: rgba(9, 30, 66, 0.08) 0px 0px 0px 1px, rgba(9, 30, 66, 0.08) 0px 2px 4px 1px;
color: var(--primary-color);
cursor: pointer;
height: 24px;
opacity: 1;
outline: 0px;
padding: 0px;
background-color: transparent;
transition: background-color 100ms linear 0s, color 100ms linear 0s, opacity 300ms cubic-bezier(0.2, 0, 0, 1) 0s, transform 300ms cubic-bezier(0.2, 0, 0, 1) 0s;
width: 20%;
}
.sidebar-toggle-button:hover {
transition: background-color 100ms linear 0s, color 100ms linear 0s, opacity 300ms cubic-bezier(0.2, 0, 0, 1) 0s, transform 300ms cubic-bezier(0.2, 0, 0, 1) 0s;
}
.main-content.open {
width: 165px;
transition: width 0.3s ease;
background-color: transparent;
}
.main-content.closed {
width: 0%;
/* min-width: 25px; */
transition: width 0.3s ease;
background-color: transparent;
}
.sidebar-menu-item {
width: 100%;
box-sizing: border-box;
}
.sidebar-menu-item.open {
width: 100%;
/* justify-content: flex-start; */
transition: width 0.3s ease;
}
.sidebar-menu-item.closed {
width: 0%;
min-width: 0px;
/* justify-content: flex-start; */
transition: width 0.3s ease;
}
.sidebar-menu-item-content.open {
width: 100%;
/* justify-content: flex-start; */
transition: width 0.3s ease;
}
.sidebar-menu-item-content.closed {
width: 0px;
/* justify-content: flex-start; */
transition: width 0.3s ease;
}
/* Ensure span can shrink */
.button-text {
display: block; /* Make span a block element */
width: 100%; /* Ensure it takes the full width of its parent */
overflow: hidden; /* Hide any overflow content */
white-space: nowrap; /* Prevent text wrapping */
}
@@ -0,0 +1,109 @@
import { useState, useEffect } from 'react';
import Button from '@mui/material/Button';
import { useTheme } from '@mui/material/styles';
import PropTypes from 'prop-types';
import Tooltip from '@mui/material/Tooltip';
import ChevronLeftRounded from "@mui/icons-material/ChevronLeftRounded"
import ChevronRightRounded from '@mui/icons-material/ChevronRightRounded';
import FileCopyIcon from '@mui/icons-material/FileCopy';
import './SideBar.css';
const Sidebar = ({ onSelection, onOpened, onClosed }) => {
const theme = useTheme();
const [isOpen, setIsOpen] = useState(false);
const [cannedQueries, setCannedQueries] = useState([]);
const Icon = isOpen ? ChevronLeftRounded : ChevronRightRounded;
const toggleIsOpen = () => {
if (isOpen) {
setIsOpen(false);
if (onClosed != null) {
onClosed();
}
} else {
setIsOpen(true);
if (onOpened != null) {
onOpened();
}
}
};
const onMenuItemClick = (event, cannedQuery) => {
console.info(`menu item clicked: ${cannedQuery.name}`);
onSelection(cannedQuery);
};
// const onMenuItemClick = async (event, cannedQuery) => {
// try {
// await navigator.clipboard.writeText(cannedQuery.query.join("\n"));
// alert(`Copied ${cannedQuery.name} to clipboard`);
// } catch (err) {
// console.error('Failed to copy query text: ', err);
// }
// };
useEffect(() => {
const fetchCannedQueries = async () => {
try {
const response = await fetch('api/cannedqueries');
const data = await response.json();
setCannedQueries(data);
} catch (error) {
console.error('Error fetching WebSocket config:', error);
}
}
fetchCannedQueries();
}, []);
const mainContentClass = `main-content ${isOpen ? 'open' : 'closed'}`;
const menuItemClass = `sidebar-menu-item ${isOpen ? 'open' : 'closed'}`;
const menuItemContentClass = `sidebar-menu-item-content ${isOpen ? 'open' : 'closed'}`;
const toggleButtonTitle = `${isOpen ? 'Hide' : 'Show'} Common Queries`;
return (
<div
className={"sidebar"}
style={{
display: 'flex',
flexDirection: 'row',
justifyContent: 'flexStart',
alignItems: 'flexStart',
margin: theme.spacing(2, 0, 0, 0)
}}
>
<div className={mainContentClass}>
{cannedQueries.map(cq =>
<Button key={cq.name} className={menuItemClass} onClick={(event) => onMenuItemClick(event, cq)}>
<div className={menuItemContentClass}>
<span className='button-text'>{cq.name}</span>
</div>
</Button>)
}
{/* {cannedQueries.map(cq =>
<Button
variant="contained"
color="primary"
startIcon={<FileCopyIcon />}
onClick={onMenuItemClick}
>
<div className={menuItemContentClass}>
<span className='button-text'>{cq.name}</span>
</div>
</Button>)} */}
</div>
<Tooltip title={toggleButtonTitle}>
<Button onClick={toggleIsOpen}>
<Icon className="w-5 h-5" />
</Button>
</Tooltip>
</div>
);
};
export default Sidebar;
Sidebar.propTypes = {
onSelection: PropTypes.func,
onOpened: PropTypes.func,
onClosed: PropTypes.func,
};
@@ -0,0 +1,112 @@
import { createTheme } from '@mui/material/styles';
import PropTypes from 'prop-types';
import '@fontsource/noto-sans';
import './App.css';
const getLightPalette = () => {
return {
mode: 'light',
primary: {
light: '#6573c3',
main: '#3f51b5',
dark: '#2c387e',
contrastText: '#ffffff',
},
secondary: {
light: '#f73378',
main: '#f50057',
dark: '#ab003c',
contrastText: '#ffffff',
},
text: {
primary: '#000000c4',
secondary: '#00000086',
disabled: '#0000005b',
hint: '#0000005b',
}
};
};
const getDarkPalette = () => {
return {
mode: 'dark',
primary: {
light: '#bcc5ff',
main: '#33baf9',
dark: '#202334',
contrastText: '#78a2ac',
},
secondary: {
light: '#9fd0b8',
main: '#5eb289',
dark: '#ffffff12',
contrastText: '#c8d4f7',
},
action: {
hoverOpacity: 0.08
},
text: {
primary: '#eaf8f8',
dark: '#eaf8f8a0'
},
queryStartButton: {
Subscribed: {
color: '#33baf9',
backgroundColor: '#ffffff40',
},
Unsubscribed: {
color: '#ffffff40',
backgroundColor: '#ffffff00',
}
},
queryStopButton: {
Subscribed: {
color: '#ff4444a0',
backgroundColor: '#ffffff40',
},
Unsubscribed: {
color: '#ff444450',
backgroundColor: '#ffffff00',
}
},
error: {
light: '#e57373',
main: '#f44336',
dark: '#d32f2f',
contrastText: '#fff',
},
warning: {
light: '#ffb74d',
main: '#ff9800',
dark: '#f57c00',
contrastText: '#000',
},
info: {
light: '#64b5f6',
main: '#2196f3',
dark: '#303030',
contrastText: '#fff',
},
success: {
light: '#81c784',
main: '#4caf50',
dark: '#388e3c',
contrastText: '#000',
},
};
};
const getTheme = ({ mode }) => {
return createTheme({
typography: {
fontFamily: '"Helvetica"'
},
palette: mode == 'dark' ? getDarkPalette() : getLightPalette()
});
};
export default getTheme;
getTheme.propTypes = {
mode: PropTypes.string.isRequired,
};
@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" class="iconify iconify--logos" width="35.93" height="32" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 228"><path fill="#00D8FF" d="M210.483 73.824a171.49 171.49 0 0 0-8.24-2.597c.465-1.9.893-3.777 1.273-5.621c6.238-30.281 2.16-54.676-11.769-62.708c-13.355-7.7-35.196.329-57.254 19.526a171.23 171.23 0 0 0-6.375 5.848a155.866 155.866 0 0 0-4.241-3.917C100.759 3.829 77.587-4.822 63.673 3.233C50.33 10.957 46.379 33.89 51.995 62.588a170.974 170.974 0 0 0 1.892 8.48c-3.28.932-6.445 1.924-9.474 2.98C17.309 83.498 0 98.307 0 113.668c0 15.865 18.582 31.778 46.812 41.427a145.52 145.52 0 0 0 6.921 2.165a167.467 167.467 0 0 0-2.01 9.138c-5.354 28.2-1.173 50.591 12.134 58.266c13.744 7.926 36.812-.22 59.273-19.855a145.567 145.567 0 0 0 5.342-4.923a168.064 168.064 0 0 0 6.92 6.314c21.758 18.722 43.246 26.282 56.54 18.586c13.731-7.949 18.194-32.003 12.4-61.268a145.016 145.016 0 0 0-1.535-6.842c1.62-.48 3.21-.974 4.76-1.488c29.348-9.723 48.443-25.443 48.443-41.52c0-15.417-17.868-30.326-45.517-39.844Zm-6.365 70.984c-1.4.463-2.836.91-4.3 1.345c-3.24-10.257-7.612-21.163-12.963-32.432c5.106-11 9.31-21.767 12.459-31.957c2.619.758 5.16 1.557 7.61 2.4c23.69 8.156 38.14 20.213 38.14 29.504c0 9.896-15.606 22.743-40.946 31.14Zm-10.514 20.834c2.562 12.94 2.927 24.64 1.23 33.787c-1.524 8.219-4.59 13.698-8.382 15.893c-8.067 4.67-25.32-1.4-43.927-17.412a156.726 156.726 0 0 1-6.437-5.87c7.214-7.889 14.423-17.06 21.459-27.246c12.376-1.098 24.068-2.894 34.671-5.345a134.17 134.17 0 0 1 1.386 6.193ZM87.276 214.515c-7.882 2.783-14.16 2.863-17.955.675c-8.075-4.657-11.432-22.636-6.853-46.752a156.923 156.923 0 0 1 1.869-8.499c10.486 2.32 22.093 3.988 34.498 4.994c7.084 9.967 14.501 19.128 21.976 27.15a134.668 134.668 0 0 1-4.877 4.492c-9.933 8.682-19.886 14.842-28.658 17.94ZM50.35 144.747c-12.483-4.267-22.792-9.812-29.858-15.863c-6.35-5.437-9.555-10.836-9.555-15.216c0-9.322 13.897-21.212 37.076-29.293c2.813-.98 5.757-1.905 8.812-2.773c3.204 10.42 7.406 21.315 12.477 32.332c-5.137 11.18-9.399 22.249-12.634 32.792a134.718 134.718 0 0 1-6.318-1.979Zm12.378-84.26c-4.811-24.587-1.616-43.134 6.425-47.789c8.564-4.958 27.502 2.111 47.463 19.835a144.318 144.318 0 0 1 3.841 3.545c-7.438 7.987-14.787 17.08-21.808 26.988c-12.04 1.116-23.565 2.908-34.161 5.309a160.342 160.342 0 0 1-1.76-7.887Zm110.427 27.268a347.8 347.8 0 0 0-7.785-12.803c8.168 1.033 15.994 2.404 23.343 4.08c-2.206 7.072-4.956 14.465-8.193 22.045a381.151 381.151 0 0 0-7.365-13.322Zm-45.032-43.861c5.044 5.465 10.096 11.566 15.065 18.186a322.04 322.04 0 0 0-30.257-.006c4.974-6.559 10.069-12.652 15.192-18.18ZM82.802 87.83a323.167 323.167 0 0 0-7.227 13.238c-3.184-7.553-5.909-14.98-8.134-22.152c7.304-1.634 15.093-2.97 23.209-3.984a321.524 321.524 0 0 0-7.848 12.897Zm8.081 65.352c-8.385-.936-16.291-2.203-23.593-3.793c2.26-7.3 5.045-14.885 8.298-22.6a321.187 321.187 0 0 0 7.257 13.246c2.594 4.48 5.28 8.868 8.038 13.147Zm37.542 31.03c-5.184-5.592-10.354-11.779-15.403-18.433c4.902.192 9.899.29 14.978.29c5.218 0 10.376-.117 15.453-.343c-4.985 6.774-10.018 12.97-15.028 18.486Zm52.198-57.817c3.422 7.8 6.306 15.345 8.596 22.52c-7.422 1.694-15.436 3.058-23.88 4.071a382.417 382.417 0 0 0 7.859-13.026a347.403 347.403 0 0 0 7.425-13.565Zm-16.898 8.101a358.557 358.557 0 0 1-12.281 19.815a329.4 329.4 0 0 1-23.444.823c-7.967 0-15.716-.248-23.178-.732a310.202 310.202 0 0 1-12.513-19.846h.001a307.41 307.41 0 0 1-10.923-20.627a310.278 310.278 0 0 1 10.89-20.637l-.001.001a307.318 307.318 0 0 1 12.413-19.761c7.613-.576 15.42-.876 23.31-.876H128c7.926 0 15.743.303 23.354.883a329.357 329.357 0 0 1 12.335 19.695a358.489 358.489 0 0 1 11.036 20.54a329.472 329.472 0 0 1-11 20.722Zm22.56-122.124c8.572 4.944 11.906 24.881 6.52 51.026c-.344 1.668-.73 3.367-1.15 5.09c-10.622-2.452-22.155-4.275-34.23-5.408c-7.034-10.017-14.323-19.124-21.64-27.008a160.789 160.789 0 0 1 5.888-5.4c18.9-16.447 36.564-22.941 44.612-18.3ZM128 90.808c12.625 0 22.86 10.235 22.86 22.86s-10.235 22.86-22.86 22.86s-22.86-10.235-22.86-22.86s10.235-22.86 22.86-22.86Z"></path></svg>

After

Width:  |  Height:  |  Size: 4.0 KiB

@@ -0,0 +1,51 @@
.dark-high-contrast {
--md-sys-color-primary: rgb(251 250 255);
--md-sys-color-surface-tint: rgb(165 200 255);
--md-sys-color-on-primary: rgb(0 0 0);
--md-sys-color-primary-container: rgb(173 204 255);
--md-sys-color-on-primary-container: rgb(0 0 0);
--md-sys-color-secondary: rgb(250 250 255);
--md-sys-color-on-secondary: rgb(0 0 0);
--md-sys-color-secondary-container: rgb(159 208 255);
--md-sys-color-on-secondary-container: rgb(0 0 0);
--md-sys-color-tertiary: rgb(255 249 251);
--md-sys-color-on-tertiary: rgb(0 0 0);
--md-sys-color-tertiary-container: rgb(222 193 231);
--md-sys-color-on-tertiary-container: rgb(0 0 0);
--md-sys-color-error: rgb(255 249 249);
--md-sys-color-on-error: rgb(0 0 0);
--md-sys-color-error-container: rgb(255 186 177);
--md-sys-color-on-error-container: rgb(0 0 0);
--md-sys-color-background: rgb(17 19 24);
--md-sys-color-on-background: rgb(225 226 233);
--md-sys-color-surface: rgb(17 19 24);
--md-sys-color-on-surface: rgb(255 255 255);
--md-sys-color-surface-variant: rgb(67 71 78);
--md-sys-color-on-surface-variant: rgb(251 250 255);
--md-sys-color-outline: rgb(200 202 212);
--md-sys-color-outline-variant: rgb(200 202 212);
--md-sys-color-shadow: rgb(0 0 0);
--md-sys-color-scrim: rgb(0 0 0);
--md-sys-color-inverse-surface: rgb(225 226 233);
--md-sys-color-inverse-on-surface: rgb(0 0 0);
--md-sys-color-inverse-primary: rgb(0 42 83);
--md-sys-color-primary-fixed: rgb(219 231 255);
--md-sys-color-on-primary-fixed: rgb(0 0 0);
--md-sys-color-primary-fixed-dim: rgb(173 204 255);
--md-sys-color-on-primary-fixed-variant: rgb(0 22 49);
--md-sys-color-secondary-fixed: rgb(214 233 255);
--md-sys-color-on-secondary-fixed: rgb(0 0 0);
--md-sys-color-secondary-fixed-dim: rgb(159 208 255);
--md-sys-color-on-secondary-fixed-variant: rgb(0 24 42);
--md-sys-color-tertiary-fixed: rgb(249 222 255);
--md-sys-color-on-tertiary-fixed: rgb(0 0 0);
--md-sys-color-tertiary-fixed-dim: rgb(222 193 231);
--md-sys-color-on-tertiary-fixed-variant: rgb(33 14 42);
--md-sys-color-surface-dim: rgb(17 19 24);
--md-sys-color-surface-bright: rgb(55 57 62);
--md-sys-color-surface-container-lowest: rgb(12 14 19);
--md-sys-color-surface-container-low: rgb(25 28 32);
--md-sys-color-surface-container: rgb(29 32 36);
--md-sys-color-surface-container-high: rgb(40 42 47);
--md-sys-color-surface-container-highest: rgb(50 53 58);
}
@@ -0,0 +1,51 @@
.dark-medium-contrast {
--md-sys-color-primary: rgb(173 204 255);
--md-sys-color-surface-tint: rgb(165 200 255);
--md-sys-color-on-primary: rgb(0 22 49);
--md-sys-color-primary-container: rgb(112 146 198);
--md-sys-color-on-primary-container: rgb(0 0 0);
--md-sys-color-secondary: rgb(159 208 255);
--md-sys-color-on-secondary: rgb(0 24 42);
--md-sys-color-secondary-container: rgb(100 149 193);
--md-sys-color-on-secondary-container: rgb(0 0 0);
--md-sys-color-tertiary: rgb(222 193 231);
--md-sys-color-on-tertiary: rgb(33 14 42);
--md-sys-color-tertiary-container: rgb(162 136 171);
--md-sys-color-on-tertiary-container: rgb(0 0 0);
--md-sys-color-error: rgb(255 186 177);
--md-sys-color-on-error: rgb(55 0 1);
--md-sys-color-error-container: rgb(255 84 73);
--md-sys-color-on-error-container: rgb(0 0 0);
--md-sys-color-background: rgb(17 19 24);
--md-sys-color-on-background: rgb(225 226 233);
--md-sys-color-surface: rgb(17 19 24);
--md-sys-color-on-surface: rgb(251 250 255);
--md-sys-color-surface-variant: rgb(67 71 78);
--md-sys-color-on-surface-variant: rgb(200 202 212);
--md-sys-color-outline: rgb(160 163 171);
--md-sys-color-outline-variant: rgb(128 131 139);
--md-sys-color-shadow: rgb(0 0 0);
--md-sys-color-scrim: rgb(0 0 0);
--md-sys-color-inverse-surface: rgb(225 226 233);
--md-sys-color-inverse-on-surface: rgb(40 42 47);
--md-sys-color-inverse-primary: rgb(35 73 120);
--md-sys-color-primary-fixed: rgb(212 227 255);
--md-sys-color-on-primary-fixed: rgb(0 17 40);
--md-sys-color-primary-fixed-dim: rgb(165 200 255);
--md-sys-color-on-primary-fixed-variant: rgb(10 55 100);
--md-sys-color-secondary-fixed: rgb(206 229 255);
--md-sys-color-on-secondary-fixed: rgb(0 18 34);
--md-sys-color-secondary-fixed-dim: rgb(154 203 250);
--md-sys-color-on-secondary-fixed-variant: rgb(0 57 91);
--md-sys-color-tertiary-fixed: rgb(247 216 255);
--md-sys-color-on-tertiary-fixed: rgb(28 9 37);
--md-sys-color-tertiary-fixed-dim: rgb(218 189 226);
--md-sys-color-on-tertiary-fixed-variant: rgb(67 46 76);
--md-sys-color-surface-dim: rgb(17 19 24);
--md-sys-color-surface-bright: rgb(55 57 62);
--md-sys-color-surface-container-lowest: rgb(12 14 19);
--md-sys-color-surface-container-low: rgb(25 28 32);
--md-sys-color-surface-container: rgb(29 32 36);
--md-sys-color-surface-container-high: rgb(40 42 47);
--md-sys-color-surface-container-highest: rgb(50 53 58);
}
@@ -0,0 +1,51 @@
.dark {
--md-sys-color-primary: rgb(165 200 255);
--md-sys-color-surface-tint: rgb(165 200 255);
--md-sys-color-on-primary: rgb(0 49 94);
--md-sys-color-primary-container: rgb(34 72 118);
--md-sys-color-on-primary-container: rgb(212 227 255);
--md-sys-color-secondary: rgb(154 203 250);
--md-sys-color-on-secondary: rgb(0 51 83);
--md-sys-color-secondary-container: rgb(12 74 114);
--md-sys-color-on-secondary-container: rgb(206 229 255);
--md-sys-color-tertiary: rgb(218 189 226);
--md-sys-color-on-tertiary: rgb(61 41 70);
--md-sys-color-tertiary-container: rgb(85 63 93);
--md-sys-color-on-tertiary-container: rgb(247 216 255);
--md-sys-color-error: rgb(255 180 171);
--md-sys-color-on-error: rgb(105 0 5);
--md-sys-color-error-container: rgb(147 0 10);
--md-sys-color-on-error-container: rgb(255 218 214);
--md-sys-color-background: rgb(17 19 24);
--md-sys-color-on-background: rgb(225 226 233);
--md-sys-color-surface: rgb(17 19 24);
--md-sys-color-on-surface: rgb(225 226 233);
--md-sys-color-surface-variant: rgb(67 71 78);
--md-sys-color-on-surface-variant: rgb(195 198 207);
--md-sys-color-outline: rgb(141 145 153);
--md-sys-color-outline-variant: rgb(67 71 78);
--md-sys-color-shadow: rgb(0 0 0);
--md-sys-color-scrim: rgb(0 0 0);
--md-sys-color-inverse-surface: rgb(225 226 233);
--md-sys-color-inverse-on-surface: rgb(46 48 53);
--md-sys-color-inverse-primary: rgb(60 96 144);
--md-sys-color-primary-fixed: rgb(212 227 255);
--md-sys-color-on-primary-fixed: rgb(0 28 58);
--md-sys-color-primary-fixed-dim: rgb(165 200 255);
--md-sys-color-on-primary-fixed-variant: rgb(34 72 118);
--md-sys-color-secondary-fixed: rgb(206 229 255);
--md-sys-color-on-secondary-fixed: rgb(0 29 50);
--md-sys-color-secondary-fixed-dim: rgb(154 203 250);
--md-sys-color-on-secondary-fixed-variant: rgb(12 74 114);
--md-sys-color-tertiary-fixed: rgb(247 216 255);
--md-sys-color-on-tertiary-fixed: rgb(39 20 48);
--md-sys-color-tertiary-fixed-dim: rgb(218 189 226);
--md-sys-color-on-tertiary-fixed-variant: rgb(85 63 93);
--md-sys-color-surface-dim: rgb(17 19 24);
--md-sys-color-surface-bright: rgb(55 57 62);
--md-sys-color-surface-container-lowest: rgb(12 14 19);
--md-sys-color-surface-container-low: rgb(25 28 32);
--md-sys-color-surface-container: rgb(29 32 36);
--md-sys-color-surface-container-high: rgb(40 42 47);
--md-sys-color-surface-container-highest: rgb(50 53 58);
}
@@ -0,0 +1,51 @@
.light-high-contrast {
--md-sys-color-primary: rgb(0 34 70);
--md-sys-color-surface-tint: rgb(60 96 144);
--md-sys-color-on-primary: rgb(255 255 255);
--md-sys-color-primary-container: rgb(29 68 114);
--md-sys-color-on-primary-container: rgb(255 255 255);
--md-sys-color-secondary: rgb(0 36 61);
--md-sys-color-on-secondary: rgb(255 255 255);
--md-sys-color-secondary-container: rgb(3 70 110);
--md-sys-color-on-secondary-container: rgb(255 255 255);
--md-sys-color-tertiary: rgb(46 26 55);
--md-sys-color-on-tertiary: rgb(255 255 255);
--md-sys-color-tertiary-container: rgb(81 59 89);
--md-sys-color-on-tertiary-container: rgb(255 255 255);
--md-sys-color-error: rgb(78 0 2);
--md-sys-color-on-error: rgb(255 255 255);
--md-sys-color-error-container: rgb(140 0 9);
--md-sys-color-on-error-container: rgb(255 255 255);
--md-sys-color-background: rgb(249 249 255);
--md-sys-color-on-background: rgb(25 28 32);
--md-sys-color-surface: rgb(249 249 255);
--md-sys-color-on-surface: rgb(0 0 0);
--md-sys-color-surface-variant: rgb(224 226 236);
--md-sys-color-on-surface-variant: rgb(32 36 43);
--md-sys-color-outline: rgb(63 67 74);
--md-sys-color-outline-variant: rgb(63 67 74);
--md-sys-color-shadow: rgb(0 0 0);
--md-sys-color-scrim: rgb(0 0 0);
--md-sys-color-inverse-surface: rgb(46 48 53);
--md-sys-color-inverse-on-surface: rgb(255 255 255);
--md-sys-color-inverse-primary: rgb(228 236 255);
--md-sys-color-primary-fixed: rgb(29 68 114);
--md-sys-color-on-primary-fixed: rgb(255 255 255);
--md-sys-color-primary-fixed-dim: rgb(0 45 88);
--md-sys-color-on-primary-fixed-variant: rgb(255 255 255);
--md-sys-color-secondary-fixed: rgb(3 70 110);
--md-sys-color-on-secondary-fixed: rgb(255 255 255);
--md-sys-color-secondary-fixed-dim: rgb(0 47 77);
--md-sys-color-on-secondary-fixed-variant: rgb(255 255 255);
--md-sys-color-tertiary-fixed: rgb(81 59 89);
--md-sys-color-on-tertiary-fixed: rgb(255 255 255);
--md-sys-color-tertiary-fixed-dim: rgb(57 37 66);
--md-sys-color-on-tertiary-fixed-variant: rgb(255 255 255);
--md-sys-color-surface-dim: rgb(217 218 224);
--md-sys-color-surface-bright: rgb(249 249 255);
--md-sys-color-surface-container-lowest: rgb(255 255 255);
--md-sys-color-surface-container-low: rgb(243 243 250);
--md-sys-color-surface-container: rgb(237 237 244);
--md-sys-color-surface-container-high: rgb(231 232 238);
--md-sys-color-surface-container-highest: rgb(225 226 233);
}
@@ -0,0 +1,51 @@
.light-medium-contrast {
--md-sys-color-primary: rgb(29 68 114);
--md-sys-color-surface-tint: rgb(60 96 144);
--md-sys-color-on-primary: rgb(255 255 255);
--md-sys-color-primary-container: rgb(83 118 167);
--md-sys-color-on-primary-container: rgb(255 255 255);
--md-sys-color-secondary: rgb(3 70 110);
--md-sys-color-on-secondary: rgb(255 255 255);
--md-sys-color-secondary-container: rgb(71 121 163);
--md-sys-color-on-secondary-container: rgb(255 255 255);
--md-sys-color-tertiary: rgb(81 59 89);
--md-sys-color-on-tertiary: rgb(255 255 255);
--md-sys-color-tertiary-container: rgb(133 108 141);
--md-sys-color-on-tertiary-container: rgb(255 255 255);
--md-sys-color-error: rgb(140 0 9);
--md-sys-color-on-error: rgb(255 255 255);
--md-sys-color-error-container: rgb(218 52 46);
--md-sys-color-on-error-container: rgb(255 255 255);
--md-sys-color-background: rgb(249 249 255);
--md-sys-color-on-background: rgb(25 28 32);
--md-sys-color-surface: rgb(249 249 255);
--md-sys-color-on-surface: rgb(25 28 32);
--md-sys-color-surface-variant: rgb(224 226 236);
--md-sys-color-on-surface-variant: rgb(63 67 74);
--md-sys-color-outline: rgb(91 95 103);
--md-sys-color-outline-variant: rgb(119 123 131);
--md-sys-color-shadow: rgb(0 0 0);
--md-sys-color-scrim: rgb(0 0 0);
--md-sys-color-inverse-surface: rgb(46 48 53);
--md-sys-color-inverse-on-surface: rgb(240 240 247);
--md-sys-color-inverse-primary: rgb(165 200 255);
--md-sys-color-primary-fixed: rgb(83 118 167);
--md-sys-color-on-primary-fixed: rgb(255 255 255);
--md-sys-color-primary-fixed-dim: rgb(58 93 141);
--md-sys-color-on-primary-fixed-variant: rgb(255 255 255);
--md-sys-color-secondary-fixed: rgb(71 121 163);
--md-sys-color-on-secondary-fixed: rgb(255 255 255);
--md-sys-color-secondary-fixed-dim: rgb(43 96 137);
--md-sys-color-on-secondary-fixed-variant: rgb(255 255 255);
--md-sys-color-tertiary-fixed: rgb(133 108 141);
--md-sys-color-on-tertiary-fixed: rgb(255 255 255);
--md-sys-color-tertiary-fixed-dim: rgb(107 84 116);
--md-sys-color-on-tertiary-fixed-variant: rgb(255 255 255);
--md-sys-color-surface-dim: rgb(217 218 224);
--md-sys-color-surface-bright: rgb(249 249 255);
--md-sys-color-surface-container-lowest: rgb(255 255 255);
--md-sys-color-surface-container-low: rgb(243 243 250);
--md-sys-color-surface-container: rgb(237 237 244);
--md-sys-color-surface-container-high: rgb(231 232 238);
--md-sys-color-surface-container-highest: rgb(225 226 233);
}
@@ -0,0 +1,51 @@
.light {
--md-sys-color-primary: rgb(60 96 144);
--md-sys-color-surface-tint: rgb(60 96 144);
--md-sys-color-on-primary: rgb(255 255 255);
--md-sys-color-primary-container: rgb(212 227 255);
--md-sys-color-on-primary-container: rgb(0 28 58);
--md-sys-color-secondary: rgb(46 98 140);
--md-sys-color-on-secondary: rgb(255 255 255);
--md-sys-color-secondary-container: rgb(206 229 255);
--md-sys-color-on-secondary-container: rgb(0 29 50);
--md-sys-color-tertiary: rgb(110 86 118);
--md-sys-color-on-tertiary: rgb(255 255 255);
--md-sys-color-tertiary-container: rgb(247 216 255);
--md-sys-color-on-tertiary-container: rgb(39 20 48);
--md-sys-color-error: rgb(186 26 26);
--md-sys-color-on-error: rgb(255 255 255);
--md-sys-color-error-container: rgb(255 218 214);
--md-sys-color-on-error-container: rgb(65 0 2);
--md-sys-color-background: rgb(249 249 255);
--md-sys-color-on-background: rgb(25 28 32);
--md-sys-color-surface: rgb(249 249 255);
--md-sys-color-on-surface: rgb(25 28 32);
--md-sys-color-surface-variant: rgb(224 226 236);
--md-sys-color-on-surface-variant: rgb(67 71 78);
--md-sys-color-outline: rgb(116 119 127);
--md-sys-color-outline-variant: rgb(195 198 207);
--md-sys-color-shadow: rgb(0 0 0);
--md-sys-color-scrim: rgb(0 0 0);
--md-sys-color-inverse-surface: rgb(46 48 53);
--md-sys-color-inverse-on-surface: rgb(240 240 247);
--md-sys-color-inverse-primary: rgb(165 200 255);
--md-sys-color-primary-fixed: rgb(212 227 255);
--md-sys-color-on-primary-fixed: rgb(0 28 58);
--md-sys-color-primary-fixed-dim: rgb(165 200 255);
--md-sys-color-on-primary-fixed-variant: rgb(34 72 118);
--md-sys-color-secondary-fixed: rgb(206 229 255);
--md-sys-color-on-secondary-fixed: rgb(0 29 50);
--md-sys-color-secondary-fixed-dim: rgb(154 203 250);
--md-sys-color-on-secondary-fixed-variant: rgb(12 74 114);
--md-sys-color-tertiary-fixed: rgb(247 216 255);
--md-sys-color-on-tertiary-fixed: rgb(39 20 48);
--md-sys-color-tertiary-fixed-dim: rgb(218 189 226);
--md-sys-color-on-tertiary-fixed-variant: rgb(85 63 93);
--md-sys-color-surface-dim: rgb(217 218 224);
--md-sys-color-surface-bright: rgb(249 249 255);
--md-sys-color-surface-container-lowest: rgb(255 255 255);
--md-sys-color-surface-container-low: rgb(243 243 250);
--md-sys-color-surface-container: rgb(237 237 244);
--md-sys-color-surface-container-high: rgb(231 232 238);
--md-sys-color-surface-container-highest: rgb(225 226 233);
}
@@ -0,0 +1,107 @@
:root {
font-family: Inter, system-ui, Avenir, Helvetica, Arial, sans-serif;
line-height: 1.5;
font-weight: 400;
color-scheme: light dark;
color: rgba(255, 255, 255, 0.87);
background-color: #242424;
font-synthesis: none;
text-rendering: optimizeLegibility;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
}
a {
font-weight: 500;
color: #646cff;
text-decoration: inherit;
}
a:hover {
color: #535bf2;
}
body {
margin: 0;
display: flex;
place-items: center;
min-width: 320px;
min-height: 100vh;
}
button {
border-radius: 8px;
border: 1px solid transparent;
padding: 0.6em 1.2em;
font-size: 1em;
font-weight: 500;
font-family: inherit;
background-color: #1a1a1a;
cursor: pointer;
transition: border-color 0.25s;
}
button:hover {
border-color: #646cff;
}
button:focus,
button:focus-visible {
outline: 4px auto -webkit-focus-ring-color;
}
button#query-button {
height: 5em;
vertical-align: top;
}
button#play-button {
padding: '10px 20px';
font-size: '16px';
cursor: 'pointer';
background-color: '#28a745';
color: '#fff';
border: 'none';
border-radius: '5px';
};
button#pause-button {
background-color: #ffc107;
}
button#stop-button {
background-color: #dc3545;
}
h1 {
font-size: 3.2em;
line-height: 1.1;
}
p#footer {
float: none;
clear: both;
vertical-align: bottom;
}
table#query-output {
float: none;
clear: both;
}
textarea#query-input {
height: 5em;
width: 50em;
}
@media (prefers-color-scheme: light) {
:root {
color: #213547;
background-color: #ffffff;
}
a:hover {
color: #747bff;
}
button {
background-color: #f9f9f9;
}
}
@@ -0,0 +1,7 @@
import React from 'react';
import ReactDOM from 'react-dom/client';
import App from './App';
ReactDOM.createRoot(document.getElementById('root')).render(
<App />
)
@@ -0,0 +1,68 @@
import { fileURLToPath, URL } from 'node:url';
import { defineConfig, loadEnv } from 'vite';
import plugin from '@vitejs/plugin-react';
import fs from 'fs';
import path from 'path';
import child_process from 'child_process';
import process, { env } from 'process';
const baseFolder =
process.env.APPDATA !== undefined && process.env.APPDATA !== ''
? `${process.env.APPDATA}/ASP.NET/https`
: `${process.env.HOME}/.aspnet/https`;
const certificateName = "bts.datastreamwebapp.client";
const certFilePath = path.join(baseFolder, `${certificateName}.pem`);
const keyFilePath = path.join(baseFolder, `${certificateName}.key`);
if (!fs.existsSync(certFilePath) || !fs.existsSync(keyFilePath)) {
if (0 !== child_process.spawnSync('dotnet', [
'dev-certs',
'https',
'--export-path',
certFilePath,
'--format',
'Pem',
'--no-password',
], { stdio: 'inherit', }).status) {
throw new Error("Could not create certificate.");
}
}
export default defineConfig(({ $$command, mode, $$isSsrBuild, $$isPreview }) => {
const target = env.ASPNETCORE_HTTPS_PORT
? `https://localhost:${env.ASPNETCORE_HTTPS_PORT}`
: env.ASPNETCORE_URLS
? env.ASPNETCORE_URLS.split(';')[0]
: 'https://localhost:8443';
return {
plugins: [plugin()],
resolve: {
alias: {
'@': fileURLToPath(new URL('./src', import.meta.url))
}
},
build: {
minify: false,
sourcemap: true,
outDir: `../../../build/Bts.DataStreamWebApp.Client/${mode}/dist`
},
server: {
proxy: {
'^/api': {
target,
secure: false
},
'^/ws': {
target,
secure: false
}
},
port: 5173,
https: {
key: fs.readFileSync(keyFilePath),
cert: fs.readFileSync(certFilePath),
}
}
};
});
@@ -0,0 +1,40 @@
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net7.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<InvariantGlobalization>true</InvariantGlobalization>
<SpaRoot>..\Bts.DataStreamWebApp.Client</SpaRoot>
<SpaProxyLaunchCommand>npm run dev</SpaProxyLaunchCommand>
<SpaProxyServerUrl>https://localhost:5173</SpaProxyServerUrl>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="CommandLineParser" Version="2.9.1" />
<PackageReference Include="Microsoft.AspNetCore.SpaProxy" Version="7.0.20" />
<PackageReference Include="Microsoft.AspNetCore.SpaServices.Extensions" Version="7.0.20" />
<PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
<PackageReference Include="Swashbuckle.AspNetCore" Version="6.4.0" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\Bts.DataStreamWebApp.Client\Bts.DataStreamWebApp.Client.esproj">
<ReferenceOutputAssembly>false</ReferenceOutputAssembly>
</ProjectReference>
<ProjectReference Include="..\DataStream\DataStream.csproj" />
<ProjectReference Include="..\SharedStreams\SharedStreams.csproj" />
<ProjectReference Include="..\..\client\spark_api\spark_api.csproj" />
<ProjectReference Include="..\..\client\spark_core\spark_core.csproj" />
<ProjectReference Include="..\..\client\spark_websocket\spark_websocket.csproj" />
</ItemGroup>
<ItemGroup>
<ConfigurationProfile Include="appsettings.Development.json" />
<ConfigurationProfile Include="appsettings.Production.json" />
<ConfigurationProfile Include="appsettings.ProductionAws.json" />
<ConfigurationProfile Include="appsettings.Staging.json" />
<ConfigurationProfile Include="appsettings.json" />
</ItemGroup>
</Project>
@@ -0,0 +1,5 @@
namespace Bts.DataStreamWebApp.Server;
public class CannedQueriesConfig {
public List<CannedQuery>? Items { get; set; }
}
@@ -0,0 +1,6 @@
namespace Bts.DataStreamWebApp.Server;
public class CannedQuery {
public string? Name { get; set; }
public List<string>? Query { get; set; }
}
@@ -0,0 +1,18 @@
namespace Bts.DataStreamWebApp.Server;
using Microsoft.AspNetCore.Mvc;
[ApiController]
[Route("api/[controller]")]
public class CannedQueriesController : ControllerBase {
readonly IConfiguration config;
public CannedQueriesController(IConfiguration config) =>
this.config = config;
[HttpGet(Name = "GetCannedQueries")]
public IEnumerable<CannedQuery>? Get() {
var cannedQueries = config.GetSection("CannedQueries").Get<List<CannedQuery>>()?.ToArray() ?? Array.Empty<CannedQuery>();
return cannedQueries;
}
}
@@ -0,0 +1,22 @@
// using Bts.DataStreamWebApp.Server;
// using Microsoft.AspNetCore.Mvc;
// [ApiController]
// [Route("api/[controller]")]
// public class ConfigController : ControllerBase {
// private readonly IConfiguration _configuration;
// public ConfigController(IConfiguration configuration) {
// _configuration = configuration;
// }
// [HttpGet("websocket")]
// public ActionResult<DataStreamWebSocketConfig> GetWebSocketConfig() {
// var endpoint = $"{_configuration["Kestrel:Endpoints:Http:Url"]}/ws";
// var webSocketConfig = new DataStreamWebSocketConfig {
// Endpoint = endpoint // or Https based on your setup
// };
// return Ok(webSocketConfig);
// }
// }
@@ -0,0 +1,30 @@
namespace Bts.DataStreamWebApp.Server.Controllers;
using Microsoft.AspNetCore.Mvc;
[Route("/ws")]
public class DataStreamSubscriptionController : ControllerBase {
readonly ILogger logger;
public DataStreamSubscriptionController(ILogger<DataStreamSubscriptionController> logger) {
this.logger = logger;
logger.LogInformation("Creating {ControllerType}", GetType().Name);
}
[Route("/ws")]
public async Task Get(CancellationToken cancellationToken) {
if (!HttpContext.WebSockets.IsWebSocketRequest) {
logger.LogError("Expected websocket request on path '{}'", HttpContext.Request.Path);
return;
}
logger.LogInformation("Establishing websocket connection...");
try {
using var webSocket = await HttpContext.WebSockets.AcceptWebSocketAsync();
logger.LogInformation("Websocket connection established.");
using var server = new DataStreamWebSocketServer(webSocket, logger);
await server.Run(cancellationToken);
} catch (Exception ex) {
logger.LogError("DataStreamSubscriptionController.Run threw exception: {Exception}", ex);
}
}
}
@@ -0,0 +1,133 @@
namespace Bts.DataStreamWebApp.Server.Controllers;
using BCG;
using bts.websocket_core.json_rpc;
using Bts.DataStreamCore;
using Bts.DataStreamWebApp.Server.Messages;
using System;
using System.Net.WebSockets;
using System.Text;
using System.Text.Json;
using System.Text.Json.Serialization;
class DataStreamSubscription : IDisposable, IHostedService {
public readonly ViewStream ViewStream;
public DataPacket? LastFired;
readonly ILogger logger;
readonly WebSocket webSocket;
public DataStreamSubscription(string id, JsonRpcId requestId, ViewStream viewStream, WebSocket webSocket, ILogger logger) {
ViewStream = viewStream;
RequestId = requestId;
Id = id;
this.webSocket = webSocket;
this.logger = logger;
}
public string Id { get; }
public JsonRpcId RequestId { get; }
public SubscriptionStatus Status { get; private set; }
public void Start() {
if (Status == SubscriptionStatus.Subscribed) {
logger.LogWarning("Ignoring redundant call to 'Start'. Id = {Id} Status = {Status}", Id, Status);
return;
}
logger.LogInformation("Subscribing to ViewStream. SubscriptionId = {SubscriptionId}", Id);
Status = SubscriptionStatus.Subscribed;
ViewStream.onChange += onChange;
}
public Task StartAsync(CancellationToken cancellationToken) {
cancellationToken.ThrowIfCancellationRequested();
Start();
return Task.CompletedTask;
}
public Task Clear() {
if (LastFired is null) {
return Task.CompletedTask;
}
logger.LogInformation("Clearing subscription (firing DataPacket to delete all rows). Id = {Id} Status = {Status}", Id, Status);
return sendDataPacket("clear", new(new BlueTable(LastFired.schema), new BlueTable(LastFired.schema), LastFired.newRows));
}
public void Stop() {
if (Status == SubscriptionStatus.Unsubscribed) {
logger.LogWarning("Ignoring redundant call to 'Stop'. Id = {Id} Status = {Status}", Id, Status);
return;
}
logger.LogInformation("Unsubscribing from ViewStream. SubscriptionId = {SubscriptionId}", Id);
Status = SubscriptionStatus.Unsubscribed;
ViewStream.onChange -= onChange;
}
public Task StopAsync(CancellationToken cancellationToken) {
cancellationToken.ThrowIfCancellationRequested();
Stop();
return Task.CompletedTask;
}
public void Dispose() =>
Stop();
async void onChange(object newValue) {
if (newValue is not DataPacket dp) {
logger.LogError("Expected DataPacket in stream callback (actually received {}).", newValue?.GetType());
return;
}
if (webSocket.State != WebSocketState.Open) {
logger.LogError(
"Unsubscribing from stream early because WebSocket state = {WebSocketState}", webSocket.State);
Dispose();
return;
}
await sendDataPacket("subscribe", dp);
}
async Task sendDataPacket(string method, DataPacket dp) {
var serializerOpts = new JsonSerializerOptions {
NumberHandling = JsonNumberHandling.AllowNamedFloatingPointLiterals
};
serializerOpts.Converters.Add(new DataPacketJsonConverter() {
IncludeCurrent = true,
UseObjectRows = true,
});
var res = JsonRpcRequest.CreateNotification(method, new DataStreamSubscriptionUpdate() {
SubscriptionId = Id,
Table = dp
});
var str = serialize(res, serializerOpts);
if (str is null) {
logger.LogError("Unsubscribing from stream early because we failed to serialize an outgoing DataPacket");
Dispose();
return;
}
logger.LogInformation(
"Sending DataPacket. NumRows = {NumRows} NumCols = {NumCols} MessageLength = {MessageLength}",
dp.newRows.Count,
dp.schema.columnNames.Length,
str.Length);
var bytes = Encoding.UTF8.GetBytes(str);
try {
await webSocket.SendAsync(bytes, WebSocketMessageType.Text, true, CancellationToken.None);
LastFired = dp;
} catch (WebSocketException ex) {
logger.LogError($"sendDataPacket failed on webSocket.SendAsync: {ex}");
}
}
string? serialize<T>(T message, JsonSerializerOptions? opts) {
opts ??= JsonSerializerOptions.Default;
try {
return JsonSerializer.Serialize(message, opts);
} catch (Exception ex) {
logger.LogError("Failed to serialize message of type {MessageType}: {Exception}", typeof(T), ex);
}
return null;
}
}
@@ -0,0 +1,13 @@
namespace Bts.DataStreamWebApp.Server;
using BCG;
using System.Text.Json.Serialization;
public class DataStreamSubscriptionUpdate {
[JsonPropertyName("subscription_id")]
public string? SubscriptionId { get; set; }
[JsonPropertyName("table")]
public DataPacket? Table { get; set; }
}
@@ -0,0 +1,89 @@
namespace Bts.DataStreamWebApp.Server;
using Bts.PubSub;
using CommandLine;
using System.Text.Json;
using System.Text.Json.Serialization;
class DataStreamWebAppServerConfig {
/// <summary>
/// The login user for sparkd
/// </summary>
[JsonPropertyName("user")]
[Option("user", HelpText = "The sparkd login user")]
public string? User { get; set; }
/// <summary>
/// The login password for sparkd
/// </summary>
[JsonPropertyName("password")]
[Option("pass", HelpText = "The sparkd login password")]
public string? Password { get; set; }
/// <summary>
/// The config server url for sparkd
/// </summary>
[JsonPropertyName("sparkd")]
[Option("sparkd", HelpText = "The sparkd config server")]
public string? Sparkd { get; set; }
[JsonPropertyName("pubsub")]
public PubSubConfig? PubSub { get; set; }
[Option("content-root-path")]
[JsonPropertyName("content_root_path")]
public string? ContentRootPath { get; set; }
[Option("web-root-path")]
[JsonPropertyName("web_root_path")]
public string? WebRootPath { get; set; }
[Option("environment-name")]
[JsonPropertyName("environment_name")]
public string? EnvironmentName { get; set; }
public DirectoryBrowserOptions? DirectoryBrowser { get; set; }
[Option(
"config",
MetaValue = "PATH",
HelpText = "Path to json file representing these options (DataStreamPublisherConfig)")]
[JsonIgnore]
public string? ConfigFile { get; set; }
public static DataStreamWebAppServerConfig Default() =>
new() {
User = "bts",
PubSub = PubSubConfig.Default(),
};
public void MergeWith(DataStreamWebAppServerConfig other) {
User ??= other.User;
Password ??= other.Password;
PubSub ??= other.PubSub;
Sparkd ??= other.Sparkd;
ContentRootPath ??= other.ContentRootPath;
WebRootPath ??= other.WebRootPath;
ConfigFile ??= other.ConfigFile;
}
/// <summary>
/// Merges options with those defined in `ConfigFile` (does not overwrite
/// non-null existing values).
/// </summary>
/// <exception cref="InvalidOperationException">
/// JsonSerializer.Deserialize returns null
/// </exception>
public void MergeWithConfigFile() {
if (ConfigFile is null) {
return;
}
var cfgString = File.ReadAllText(ConfigFile);
var parsedCfg = JsonSerializer.Deserialize<DataStreamWebAppServerConfig>(cfgString) ??
throw new InvalidOperationException($"Failed to parse json from {ConfigFile}");
MergeWith(parsedCfg);
ConfigFile = null;
}
}
@@ -0,0 +1,5 @@
namespace Bts.DataStreamWebApp.Server;
public class DataStreamWebSocketConfig {
public string? Endpoint { get; set; }
}
@@ -0,0 +1,148 @@
namespace Bts.DataStreamWebApp.Server.Controllers;
using bts.websocket_core.json_rpc;
using Bts.DataStreamWebApp.Server.Messages;
using System.Net.WebSockets;
using System.Text;
using System.Text.Json;
using System.Text.Json.Nodes;
public class DataStreamWebSocketServer : IDisposable, IAsyncDisposable {
readonly ILogger logger;
readonly WebSocket webSocket;
readonly Dictionary<string, DataStreamSubscription> subscriptions = new();
public DataStreamWebSocketServer(WebSocket webSocket, ILogger logger) {
this.webSocket = webSocket;
this.logger = logger;
}
public async Task<int> Run(CancellationToken cancellationToken) {
logger.LogInformation("DataStreamWebSocketServer running");
await foreach (var req in JsonRpcRequestAsyncEnumerable.Create(webSocket, logger, cancellationToken)) {
await (req.Method switch {
"subscribe" => onSubscribe(req, cancellationToken),
"unsubscribe" => onUnsubscribe(req.Deserialize<UnsubscribeRequest>(), cancellationToken),
_ => Task.CompletedTask,
});
}
logger.LogInformation("DataStreamWebSocketServer stopped");
return 0;
}
/// <summary>
/// Disposes internal subscriptions. To additionally notify the client of the subscription status
/// change(s), use <see cref="DisposeAsync"/>.
/// </summary>
public void Dispose() {
GC.SuppressFinalize(this);
disposeSubscriptions();
}
/// <summary>
/// Disposes internal subscriptions (like <see cref="Dispose"/>), then sends 'subscription status change'
/// notifications to the client (unlike <see cref="Dispose"/>).
/// </summary>
public async ValueTask DisposeAsync() {
GC.SuppressFinalize(this);
var finalSubs = subscriptions.Values.Select(s => (s.RequestId, s.Id)).ToList();
disposeSubscriptions();
// Send a 'subscription status change' msg to the client for each one.
foreach (var (requestId, id) in finalSubs) {
await sendSubscriptionStatus(requestId, id, "unsubscribe", SubscriptionStatus.Unsubscribed, CancellationToken.None);
}
}
async Task onSubscribe(JsonRpcRequest<JsonNode> req, CancellationToken cancellationToken) {
ArgumentNullException.ThrowIfNull(req.Id);
ArgumentException.ThrowIfNullOrEmpty(req.Method);
if (!SubscribeRequest.Parse(req, out var subReq, out var parseErr, out var errCode)) {
logger.LogError("Invalid 'subscribe' request: {Reason}", parseErr);
var rpcError = JsonRpcResponse.CreateError<SubscriptionError>(
req.Id, new() { Code = (long)errCode, Message = "Failed to parse subscription request", Data = parseErr });
await send(rpcError, cancellationToken);
return;
}
var subId = subReq.SubscriptionId ?? throw new InvalidOperationException("SubscriptionId is null (parse false)");
disposeSubscription(subId);
if (!subReq.ParseQuery(out var vs, out parseErr, out errCode)) {
logger.LogError("Invalid 'subscribe' request: {Reason}", parseErr);
var rpcError = JsonRpcResponse.CreateError<SubscriptionError>(
req.Id, new() { Code = (long)errCode, Message = "Failed to parse subscription request", Data = parseErr });
await send(rpcError, cancellationToken);
return;
}
if (subscriptions.TryGetValue(subId, out var oldSub)) {
logger.LogInformation("Disposing existing subscription with id {SubscriptionId}", subId);
oldSub.Dispose();
}
var sub = new DataStreamSubscription(subId, req.Id.Value, vs, webSocket, logger);
subscriptions[subId] = sub;
await sendSubscriptionStatus(req.Id.Value, subId, req.Method, SubscriptionStatus.Subscribed, cancellationToken);
vs.ErrorChanged += OnStreamError;
await sub.StartAsync(cancellationToken);
async void OnStreamError(Exception exception) {
logger.LogError("Error in ViewStream (disposing subscription): {Exception}", exception);
subscriptions.Remove(sub.Id);
sub.Dispose();
var error = new SubscriptionError() {
ErrorMessage = "Error in ViewStream (disposing subscription)"
};
if (exception is not null) {
error.ErrorMessage += ". " + exception.Message;
error.StackTrace = exception.StackTrace?.Split("\n")?.ToList();
}
var rpcError = JsonRpcResponse.CreateError<SubscriptionError>(
req.Id, new() { Code = (long)ErrorCode.ViewStreamError, Message = "Error in ViewStream (disposing subscription)", Data = error });
await send(rpcError, cancellationToken);
}
}
async Task onUnsubscribe(JsonRpcRequest<UnsubscribeRequest> req, CancellationToken cancellationToken) {
ArgumentNullException.ThrowIfNull(req.Id);
ArgumentException.ThrowIfNullOrEmpty(req.Method);
var subId = req.Params?.SubscriptionId ?? string.Empty;
if (!disposeSubscription(subId)) {
logger.LogInformation("Unable to dispose subscription. Subscription with id {Id} not found.", subId);
}
await sendSubscriptionStatus(req.Id.Value, subId, "unsubscribe", SubscriptionStatus.Unsubscribed, cancellationToken);
}
bool disposeSubscription(string subId) {
if (!subscriptions.TryGetValue(subId, out var sub)) {
return false;
}
logger.LogInformation("Disposing subscription with id {Id}", subId);
sub.Dispose();
subscriptions.Remove(subId);
return true;
}
Task sendSubscriptionStatus(JsonRpcId reqId, string subId, string method, SubscriptionStatus status, CancellationToken cancellationToken) {
logger.LogInformation("Sending {Method} ack. SubscriptionId = {SubscriptionId}", method, subId);
var response = JsonRpcResponse.Create(reqId, new SubscriptionStatusMessage() {
SubscriptionId = subId,
Status = status,
});
return send(response, cancellationToken);
}
Task send<T>(T message, CancellationToken cancellationToken) {
if (message is null) {
return Task.CompletedTask;
}
var str = JsonSerializer.Serialize(message);
var bytes = Encoding.UTF8.GetBytes(str);
logger.LogTrace("Sending data. Length = {Length}", bytes.Length);
return webSocket.SendAsync(bytes, WebSocketMessageType.Text, true, cancellationToken);
}
void disposeSubscriptions() {
logger.LogInformation("Disposing DataStreamWebSocketServer subscriptions. Count: {Count}", subscriptions.Count);
subscriptions.Values.DisposeAll();
subscriptions.Clear();
}
}
@@ -0,0 +1,11 @@
FROM bluetradingsystems/linux_dotnet7
ARG SOURCE_DIR="build/Release/bin/net7.0/linux-x64/publish"
ARG WEB_ROOT="build/Bts.DataStreamWebApp.Client/production/dist"
ARG CONF_FILE="src/DataStreamSuite/Bts.DataStreamWebApp.Server/spark.conf"
RUN dotnet dev-certs https --trust
COPY --chown=bts:bts "${SOURCE_DIR}" "bin"
COPY --chown=bts:bts "${WEB_ROOT}" "bin/wwwroot"
COPY --chown=bts:bts "${CONF_FILE}" ".local/share/bts/Bts.DataStreamWebApp.Server/conf/spark.conf"
WORKDIR "/home/bts/bin"
@@ -0,0 +1,5 @@
namespace Bts.DataStreamWebApp.Server;
public class CannedQueriesResponse {
public List<CannedQuery>? Items { get; set; }
}
@@ -0,0 +1,7 @@
namespace Bts.DataStreamWebApp.Server.Messages;
enum ErrorCode {
None = 0,
InvalidSubscriptionRequest,
ViewStreamError,
}
@@ -0,0 +1,72 @@
namespace Bts.DataStreamWebApp.Server.Messages;
using BCG;
using bts.websocket_core.json_rpc;
using Bts.DataStreamExpressions;
using System.Diagnostics.CodeAnalysis;
using System.Text.Json;
using System.Text.Json.Nodes;
using System.Text.Json.Serialization;
public class SubscribeRequest {
[JsonPropertyName("query")]
public string? Query { get; set; }
[JsonPropertyName("subscription_id")]
public string? SubscriptionId { get; set; }
internal static bool Parse(JsonRpcRequest<JsonNode> req, [NotNullWhen(true)]out SubscribeRequest? subscription, [NotNullWhen(false)]out SubscriptionError? error, out ErrorCode errorCode) {
subscription = null;
errorCode = ErrorCode.None;
var paramsJsonObj = req.Params?.AsObject();
error = null;
if (paramsJsonObj is null) {
errorCode = ErrorCode.InvalidSubscriptionRequest;
error = new() { ErrorMessage = "'params' object not found" };
return false;
}
subscription = paramsJsonObj.Deserialize<SubscribeRequest>();
if (subscription is null) {
errorCode = ErrorCode.InvalidSubscriptionRequest;
error = new() { ErrorMessage = "'params' object invalid" };
return false;
}
return true;
}
internal bool ParseQuery([NotNullWhen(true)] out ViewStream? stream, [NotNullWhen(false)] out SubscriptionError? error, out ErrorCode errorCode) {
stream = null;
errorCode = ErrorCode.None;
error = null;
if (Query is null) {
errorCode = ErrorCode.InvalidSubscriptionRequest;
error = new() { ErrorMessage = "'params.query' field not found (expected a string)" };
return false;
}
var exp = DataStreamExpressionsInterpreter.Evaluate(Query);
if (exp is null) {
errorCode = ErrorCode.InvalidSubscriptionRequest;
error = new() { ErrorMessage = "'params.query' expression has invalid syntax" };
return false;
}
Exception? exception = null;
try {
stream = exp.getView(new Dictionary<string, StatementExpression>(0));
} catch (Exception ex) {
exception = ex;
}
if (stream is null) {
errorCode = ErrorCode.InvalidSubscriptionRequest;
error = new() { ErrorMessage = "Failed to create ViewStream from 'params.query' expression (check for typos in arguments/typenames)" };
if (exception is not null) {
error.ErrorMessage += ". " + exception.Message;
error.StackTrace = exception.StackTrace?.Split("\n")?.ToList();
}
return false;
}
return true;
}
}
@@ -0,0 +1,11 @@
namespace Bts.DataStreamWebApp.Server.Messages;
using System.Text.Json.Serialization;
public class SubscriptionError {
[JsonPropertyName("error_message")]
public string? ErrorMessage { get; set; }
[JsonPropertyName("stack_trace")]
public List<string>? StackTrace { get; set; }
}
@@ -0,0 +1,18 @@
namespace Bts.DataStreamWebApp.Server.Messages;
using System.Text.Json.Serialization;
enum SubscriptionStatus {
None,
Subscribed,
Unsubscribed,
}
class SubscriptionStatusMessage {
[JsonPropertyName("subscription_id")]
public string? SubscriptionId { get; set; }
[JsonPropertyName("status")]
[JsonConverter(typeof(JsonStringEnumConverter))]
public SubscriptionStatus Status { get; set; } = SubscriptionStatus.None;
}
@@ -0,0 +1,8 @@
namespace Bts.DataStreamWebApp.Server.Messages;
using System.Text.Json.Serialization;
public class UnsubscribeRequest {
[JsonPropertyName("subscription_id")]
public string? SubscriptionId { get; set; }
}
@@ -0,0 +1,82 @@
using Bts.DataStreamCore.Services;
using Bts.PubSub.Services;
using Spark.Api.Services;
using Spark.Core.Services;
using System.Reactive.Linq;
using System.Reflection;
var builder = WebApplication.CreateBuilder(new WebApplicationOptions {
Args = args,
ApplicationName = "Bts.DataStreamWebApp.Server",
});
ConfigureApp(builder.Configuration, builder.Environment.EnvironmentName);
ConfigureServices(builder.Services, builder.Configuration);
var app = builder.Build();
var wsOpts = new WebSocketOptions {
KeepAliveInterval = TimeSpan.FromMinutes(2)
};
if (app.Environment.IsDevelopment()) {
app.UseDirectoryBrowser("/browse");
app.UseSwagger();
app.UseSwaggerUI();
}
app.UseHttpsRedirection();
app.UseWebSockets(wsOpts);
app.UseDefaultFiles();
app.UseStaticFiles();
app.UseAuthorization();
app.MapControllers();
try {
Console.Write($"Starting Application");
app.Run();
Console.Write($"Application exited gracefully.");
} catch (Exception ex) {
Console.Error.Write($"Application crashed: {ex}");
return 1;
}
return 0;
static void ConfigureApp(ConfigurationManager configBuilder, string environment) {
// We prefix appsettings with the exe name because multiple exes will usually be in the build dir
// (this way, each exe can still load its own appsettings files from the current directory).
var exeName = Assembly.GetCallingAssembly().GetName().Name;
AddJsonFile($"{exeName}.appsettings.json");
AddJsonFile($"{exeName}.appsettings.{environment}.json");
var cfg = configBuilder.GetSection("config")?.Value;
if (!SetBasePath(cfg)) {
AddJsonFile(cfg);
}
bool SetBasePath(string? path) {
if (!Directory.Exists(path)) {
return false;
}
configBuilder.SetBasePath(path);
return true;
}
bool AddJsonFile(string? path) {
if (!File.Exists(path)) {
return false;
}
configBuilder.AddJsonFile(path);
return true;
}
}
static void ConfigureServices(IServiceCollection services, IConfiguration config) {
services.AddDirectoryBrowser()
.AddControllers().Services
.AddEndpointsApiExplorer()
.AddSwaggerGen();
services.AddSpaStaticFiles();
services.AddSpark(config.GetSection("Spark"))
.AddSparkApi(config.GetSection("SparkApi"))
.AddMasterWorker(config.GetSection("MasterWorker"))
.AddViewStreamDatabase()
.AddPubSubClient(config.GetSection("PubSub"));
}
@@ -0,0 +1,29 @@
{
"$schema": "http://json.schemastore.org/launchsettings.json",
"profiles": {
"http": {
"commandName": "Project",
"dotnetRunMessages": true,
"launchBrowser": true,
"launchUrl": "swagger",
"applicationUrl": "http://localhost:8080",
"environmentVariables": {
"ASPNETCORE_HTTP_PORT": "8080",
"ASPNETCORE_ENVIRONMENT": "Development",
"ASPNETCORE_HOSTINGSTARTUPASSEMBLIES": "Microsoft.AspNetCore.SpaProxy",
}
},
"https": {
"commandName": "Project",
"dotnetRunMessages": true,
"launchBrowser": true,
"applicationUrl": "http://localhost:8080;https://localhost:8443",
"environmentVariables": {
"ASPNETCORE_HTTP_PORT": "8080",
"ASPNETCORE_HTTPS_PORT": "8443",
"ASPNETCORE_ENVIRONMENT": "Development",
"ASPNETCORE_HOSTINGSTARTUPASSEMBLIES": "Microsoft.AspNetCore.SpaProxy",
}
},
},
}
@@ -0,0 +1,62 @@
{
"Kestrel": {
"Endpoints": {
"Http": {
"Url": "http://localhost:8080"
},
"Https": {
"Url": "https://localhost:8443"
}
}
},
"PubSub": {
"Routing": {
"PublisherRoutes": [
{
"Name": "default",
"ServerAddress": "bts-cloud://localhost:22019",
"TagRegExp": ".*"
}
],
"SubscriberRoutes": [
{
"Name": "default",
"ServerAddress": "bts-cloud://localhost:22019",
"TagRegExp": ".*"
}
]
}
},
"CannedQueries": [
{
"Name": "BBO",
"Query": [
"bbo"
]
},
{
"Name": "Trades",
"Query": [
"timesales"
]
},
{
"Name": "Instruments",
"Query": [
"instruments"
]
},
{
"Name": "Books",
"Query": [
"books"
]
},
{
"Name": "Benzinga News",
"Query": [
"benzinga_news"
]
}
]
}
@@ -0,0 +1,8 @@
{
"Spark": {
"ConfigServer": "http://sparkd:8000/config"
},
"SparkApi": {
"IsEnabled": true
}
}
@@ -0,0 +1,115 @@
{
"MasterWorker": {
"Interval": "00:00:00.1"
},
"ViewStreamDatabase": {
"Interval": "00:00:00.1"
},
"Spark": {
"ConfigServer": "http://172.30.2.181:8000/config"
},
"SparkApi": {
"Credentials": {
"User": "bts",
"Password": ""
},
"EnableAdmin": true,
"EnableOrders": true,
"EnablePosition": true
},
"PubSub": {
"HeartbeatInterval": 15000,
"MissingHeartbeatsBeforeGc": 50,
"ServerPort": 22019,
"Routing": {
"PublisherRoutes": [
{
"Name": "default",
"ServerAddress": "bts-cloud://172.30.2.181:22019",
"TagRegExp": ".*"
}
],
"SubscriberRoutes": [
{
"Name": "default",
"ServerAddress": "bts-cloud://172.30.2.181:22019",
"TagRegExp": ".*"
}
]
},
"TagTypeToAssemblyMap": {
"Entries": [
{
"HandlerClass": "BCG.PubSubServerTagHandler",
"Assembly": "DataStreamCore.dll"
}
]
},
"TagHandlerFactory": {
"Handlers": [
{
"TagType": "DataStream",
"Assembly": "DataStreamCore",
"Class": "bts.PubsubServerStreamTagHandler"
},
{
"TagType": "System_Sub",
"Assembly": "PubSub",
"Class": "BCG.SystemSubscriptionMapPublisher"
},
{
"TagType": "System_Stats",
"Assembly": "PubSub",
"Class": "BCG.SystemStatsPublisher"
},
{
"TagType": ".*",
"Assembly": "DataStreamCore",
"Class": "bts.PubsubServerGenericDataHandler"
}
]
}
},
"Kestrel": {
"Endpoints": {
"Http": {
"Url": "http://172.30.2.181:8080"
},
"Https": {
"Url": "https://172.30.2.181:8443"
}
}
},
"CannedQueries": [
{
"Name": "BBO",
"Query": [
"bbo"
]
},
{
"Name": "Trades",
"Query": [
"timesales"
]
},
{
"Name": "Instruments",
"Query": [
"instruments"
]
},
{
"Name": "Books",
"Query": [
"books"
]
},
{
"Name": "Benzinga News",
"Query": [
"benzinga_news"
]
}
]
}
@@ -0,0 +1,34 @@
{
"CannedQueries": [
{
"Name": "BBO",
"Query": [
"bbo"
]
},
{
"Name": "Trades",
"Query": [
"timesales"
]
},
{
"Name": "Instruments",
"Query": [
"instruments"
]
},
{
"Name": "Books",
"Query": [
"books"
]
},
{
"Name": "Benzinga News",
"Query": [
"benzinga_news"
]
}
]
}
@@ -0,0 +1,85 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
},
"AllowedHosts": "*",
"Kestrel": {
"Endpoints": {
"Http": {
"Url": "http://0.0.0.0:8080"
},
"Https": {
"Url": "https://0.0.0.0:8443"
}
}
},
"MasterWorker": {
"Interval": "00:00:00.1"
},
"ViewStreamDatabase": {
"Interval": "00:00:00.1"
},
"SparkApi": {
"IsEnabled": false,
"Credentials": {
"User": "bts",
"Password": ""
}
},
"PubSub": {
"HeartbeatInterval": 15000,
"MissingHeartbeatsBeforeGc": 50,
"ServerPort": 22019,
"Routing": {
"PublisherRoutes": [
{
"Name": "default",
"ServerAddress": "bts-cloud://pubsub:22019",
"TagRegExp": ".*"
}
],
"SubscriberRoutes": [
{
"Name": "default",
"ServerAddress": "bts-cloud://pubsub:22019",
"TagRegExp": ".*"
}
]
},
"TagTypeToAssemblyMap": {
"Entries": [
{
"HandlerClass": "BCG.PubSubServerTagHandler",
"Assembly": "DataStreamCore.dll"
}
]
},
"TagHandlerFactory": {
"Handlers": [
{
"TagType": "DataStream",
"Assembly": "DataStreamCore",
"Class": "bts.PubsubServerStreamTagHandler"
},
{
"TagType": "System_Sub",
"Assembly": "PubSub",
"Class": "BCG.SystemSubscriptionMapPublisher"
},
{
"TagType": "System_Stats",
"Assembly": "PubSub",
"Class": "BCG.SystemStatsPublisher"
},
{
"TagType": ".*",
"Assembly": "DataStreamCore",
"Class": "bts.PubsubServerGenericDataHandler"
}
]
}
}
}
@@ -0,0 +1,15 @@
Logging {
value {
console {
default_level: "Info"
}
syslog {
default_level: "Info"
}
file {
default_level: "Info"
}
initial_queue_size: 120001
max_queue_growth: 4
}
}
@@ -0,0 +1,168 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<title></title>
<style>
table {
border: 0
}
.commslog-data {
font-family: Consolas, Courier New, Courier, monospace;
}
.commslog-server {
background-color: red;
color: white
}
.commslog-client {
background-color: green;
color: white
}
</style>
</head>
<body>
<h1>ALT WebSocket Sample Application</h1>
<p id="stateLabel">Ready to connect...</p>
<div>
<label for="connectionUrl">WebSocket Server URL:</label>
<input id="connectionUrl" />
<button id="connectButton" type="submit">Connect</button>
</div>
<p></p>
<div>
<label for="sendMessage">Message to send:</label>
<input id="sendMessage" disabled />
<button id="sendButton" type="submit" disabled>Send</button>
<button id="closeButton" disabled>Close Socket</button>
</div>
<h2>Communication Log</h2>
<table style="width: 800px">
<thead>
<tr>
<td style="width: 100px">From</td>
<td style="width: 100px">To</td>
<td>Data</td>
</tr>
</thead>
<tbody id="commsLog">
</tbody>
</table>
<script>
var connectionUrl = document.getElementById("connectionUrl");
var connectButton = document.getElementById("connectButton");
var stateLabel = document.getElementById("stateLabel");
var sendMessage = document.getElementById("sendMessage");
var sendButton = document.getElementById("sendButton");
var commsLog = document.getElementById("commsLog");
var closeButton = document.getElementById("closeButton");
var socket;
var scheme = document.location.protocol === "https:" ? "wss" : "ws";
var port = document.location.port ? (":" + document.location.port) : "";
connectionUrl.value = scheme + "://" + document.location.hostname + port + "/ws" ;
function updateState() {
function disable() {
sendMessage.disabled = true;
sendButton.disabled = true;
closeButton.disabled = true;
}
function enable() {
sendMessage.disabled = false;
sendButton.disabled = false;
closeButton.disabled = false;
}
connectionUrl.disabled = true;
connectButton.disabled = true;
if (!socket) {
disable();
} else {
switch (socket.readyState) {
case WebSocket.CLOSED:
stateLabel.innerHTML = "Closed";
disable();
connectionUrl.disabled = false;
connectButton.disabled = false;
break;
case WebSocket.CLOSING:
stateLabel.innerHTML = "Closing...";
disable();
break;
case WebSocket.CONNECTING:
stateLabel.innerHTML = "Connecting...";
disable();
break;
case WebSocket.OPEN:
stateLabel.innerHTML = "Open";
enable();
break;
default:
stateLabel.innerHTML = "Unknown WebSocket State: " + htmlEscape(socket.readyState);
disable();
break;
}
}
}
closeButton.onclick = function () {
if (!socket || socket.readyState !== WebSocket.OPEN) {
alert("socket not connected");
}
socket.close(1000, "Closing from client");
};
sendButton.onclick = function () {
if (!socket || socket.readyState !== WebSocket.OPEN) {
alert("socket not connected");
}
var data = sendMessage.value;
socket.send(data);
commsLog.innerHTML += '<tr>' +
'<td class="commslog-client">Client</td>' +
'<td class="commslog-server">Server</td>' +
'<td class="commslog-data">' + htmlEscape(data) + '</td></tr>';
};
connectButton.onclick = function() {
stateLabel.innerHTML = "Connecting";
socket = new WebSocket(connectionUrl.value);
socket.onopen = function (event) {
updateState();
commsLog.innerHTML += '<tr>' +
'<td colspan="3" class="commslog-data">Connection opened</td>' +
'</tr>';
};
socket.onclose = function (event) {
updateState();
commsLog.innerHTML += '<tr>' +
'<td colspan="3" class="commslog-data">Connection closed. Code: ' + htmlEscape(event.code) + '. Reason: ' + htmlEscape(event.reason) + '</td>' +
'</tr>';
};
socket.onerror = updateState;
socket.onmessage = function (event) {
commsLog.innerHTML += '<tr>' +
'<td class="commslog-server">Server</td>' +
'<td class="commslog-client">Client</td>' +
'<td class="commslog-data">' + htmlEscape(event.data) + '</td></tr>';
};
};
function htmlEscape(str) {
return str.toString()
.replace(/&/g, '&amp;')
.replace(/"/g, '&quot;')
.replace(/'/g, '&#39;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
}
</script>
</body>
</html>
Binary file not shown.

After

Width:  |  Height:  |  Size: 8.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

@@ -0,0 +1,176 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<title></title>
<style>
table {
border: 0
}
.commslog-data {
font-family: Consolas, Courier New, Courier, monospace;
}
.commslog-server {
background-color: red;
color: white
}
.commslog-client {
background-color: green;
color: white
}
</style>
</head>
<body>
<h1>DataStreams via WebSocket</h1>
<p id="stateLabel">Ready to connect...</p>
<div>
<label for="connectionUrl">WebSocket Server URL:</label>
<input id="connectionUrl" />
<button id="connectButton" type="submit">Connect</button>
</div>
<p></p>
<div>
<label for="sendMessage">Query:</label>
<input id="sendMessage" disabled />
<button id="sendButton" type="submit" disabled>Send</button>
<button id="closeButton" disabled>Close Socket</button>
</div>
<h2>Communication Log</h2>
<table style="width: 800px">
<thead>
<tr>
<td style="width: 100px">From</td>
<td style="width: 100px">To</td>
<td>Data</td>
</tr>
</thead>
<tbody id="commsLog">
</tbody>
</table>
<script>
var connectionUrl = document.getElementById("connectionUrl");
var connectButton = document.getElementById("connectButton");
var stateLabel = document.getElementById("stateLabel");
var sendMessage = document.getElementById("sendMessage");
var sendButton = document.getElementById("sendButton");
var commsLog = document.getElementById("commsLog");
var closeButton = document.getElementById("closeButton");
var socket;
var scheme = document.location.protocol === "https:" ? "wss" : "ws";
var port = document.location.port ? (":" + document.location.port) : "";
connectionUrl.value = scheme + "://" + document.location.hostname + port + "/ws" ;
function updateState() {
function disable() {
sendMessage.disabled = true;
sendButton.disabled = true;
closeButton.disabled = true;
}
function enable() {
sendMessage.disabled = false;
sendButton.disabled = false;
closeButton.disabled = false;
}
connectionUrl.disabled = true;
connectButton.disabled = true;
if (!socket) {
disable();
} else {
switch (socket.readyState) {
case WebSocket.CLOSED:
stateLabel.innerHTML = "Closed";
disable();
connectionUrl.disabled = false;
connectButton.disabled = false;
break;
case WebSocket.CLOSING:
stateLabel.innerHTML = "Closing...";
disable();
break;
case WebSocket.CONNECTING:
stateLabel.innerHTML = "Connecting...";
disable();
break;
case WebSocket.OPEN:
stateLabel.innerHTML = "Open";
enable();
break;
default:
stateLabel.innerHTML = "Unknown WebSocket State: " + htmlEscape(socket.readyState);
disable();
break;
}
}
}
closeButton.onclick = function () {
if (!socket || socket.readyState !== WebSocket.OPEN) {
alert("socket not connected");
}
socket.close(1000, "Closing from client");
};
sendButton.onclick = function () {
if (!socket || socket.readyState !== WebSocket.OPEN) {
alert("socket not connected");
}
const rpcReq = {
jsonrpc: "2.0",
method: "subscribe",
params: {
query: sendMessage.value,
},
id: 1,
};
const rpcReqStr = JSON.stringify(rpcReq);
socket.send(rpcReqStr);
commsLog.innerHTML += '<tr>' +
'<td class="commslog-client">Client</td>' +
'<td class="commslog-server">Server</td>' +
'<td class="commslog-data">' + htmlEscape(rpcReqStr) + '</td></tr>';
};
connectButton.onclick = function() {
stateLabel.innerHTML = "Connecting";
socket = new WebSocket(connectionUrl.value);
socket.onopen = function (event) {
updateState();
commsLog.innerHTML += '<tr>' +
'<td colspan="3" class="commslog-data">Connection opened</td>' +
'</tr>';
};
socket.onclose = function (event) {
updateState();
commsLog.innerHTML += '<tr>' +
'<td colspan="3" class="commslog-data">Connection closed. Code: ' + htmlEscape(event.code) + '. Reason: ' + htmlEscape(event.reason) + '</td>' +
'</tr>';
};
socket.onerror = updateState;
socket.onmessage = function (event) {
commsLog.innerHTML += '<tr>' +
'<td class="commslog-server">Server</td>' +
'<td class="commslog-client">Client</td>' +
'<td class="commslog-data">' + htmlEscape(event.data) + '</td></tr>';
};
};
function htmlEscape(str) {
return str.toString()
.replace(/&/g, '&amp;')
.replace(/"/g, '&quot;')
.replace(/'/g, '&#39;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
}
</script>
</body>
</html>
@@ -0,0 +1,13 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net7.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Hosting" Version="7.0.1" />
</ItemGroup>
</Project>
@@ -0,0 +1,59 @@
namespace Bts.Extensions.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;
using System.Diagnostics;
public static class HostBuilderExtensions {
public static IHostBuilder CreateBtsBuilder(string[] args) {
// Obtain calling assembly name here, rather than in the lambda where it is actually needed,
// because want the name of the assembly that called this function, not the assembly that will eventually invoke
// the lambda...that will be the Microsoft.Extensions.Hosting assembly.
// Get the name of the assembly that called this function. That will act as the "appsettings"
// json file prefix
var exeName = GetCallingAssemblyName();
return Host.CreateDefaultBuilder(args)
.ConfigureAppConfiguration((context, configBuilder) =>
ConfigureApp(configBuilder, context.HostingEnvironment.EnvironmentName, exeName));
}
public static void ConfigureApp(IConfigurationBuilder configBuilder, string? environment = null, string? prefix = null) {
var baseFileName = prefix is null ? "appsettings" : $"{prefix}.appsettings";
AddJsonFile($"{baseFileName}.json");
if (environment is not null) {
AddJsonFile($"{baseFileName}.{environment}.json");
}
var cfg = configBuilder.Build().GetSection("config")?.Value;
if (!SetBasePath(cfg)) {
AddJsonFile(cfg);
}
bool SetBasePath(string? path) {
if (!Directory.Exists(path)) {
return false;
}
configBuilder.SetBasePath(path);
return true;
}
bool AddJsonFile(string? path) {
if (!File.Exists(path)) {
return false;
}
configBuilder.AddJsonFile(path);
return true;
}
}
public static string GetCallingAssemblyName() =>
new StackTrace()
.GetFrame(2)
?.GetMethod()
?.Module
?.Assembly
?.GetName()
?.Name
?? throw new InvalidOperationException("Failed to determine name of calling assembly (calling method is null)");
}
Loaded 100 of 2453 files, more files were not shown because too many files have changed in this diff. Show more