Commit Graph
12152 Commits
Author SHA1 Message Date
John Colvin f1ddd8ef6a fix build: Remove unused refs
Sparkd release builds were failing with warnings-as-errors about unused refs
2024-10-30 19:35:05 +00:00
John Colvin 66a1d19dd4 Fix eslint config and use modern flat style
https://eslint.org/docs/latest/use/configure/configuration-files#configuration-file
2024-10-30 19:35:05 +00:00
John Colvin 2ddf14d2b5 cleanup: Move global var into function scope in vite cfg 2024-10-30 19:35:05 +00:00
John Colvin 51aa380c3f Update node pkgs in datastream webapp 2024-10-30 19:35:05 +00:00
John Colvin d718a84f6c Put cme config url in GlbxMarketDataConfig proto
Instead of using magic strings
2024-10-30 19:35:05 +00:00
John Colvin 13552e29fe refactor: Change raw IDGenerator ptr to shared_ptr 2024-10-30 19:35:05 +00:00
John Colvin 0ac95710b1 refactor: Return row id in IStore::retrieve_one
This is a prerequisite to synchronizing security ids stored via IDGenerator
(the "new" way to store ids, added when we implemented Deribit), and
SecurityDB (the original way from spark days). The problem was that these
2 implementations were using separate databases. We need to unify the
interfaces between these storage services to avoid breaking existing code.
2024-10-30 19:35:05 +00:00
John Colvin f23c7af88c Add database dir and file fns to SparkDataDirectory 2024-10-30 19:35:05 +00:00
John Colvin 345b76257b Interpret db strng param as name or path in SqliteStore ctr 2024-10-30 19:35:05 +00:00
John Colvin 4996c27f01 refactor: Normalize sec key indexing for old md sources
Now both old and new (domain) md sources index securities by
the same string format.
2024-10-30 19:35:05 +00:00
John Colvin 9a9bcef348 Add ostream operator for Venue 2024-10-30 19:35:05 +00:00
John Colvin 8502b113fe Add ostream operator for SecurityType 2024-10-30 19:35:05 +00:00
John Colvin 979b2ccb6f refactor: Normalize sec key indexing for new (domain) md sources
This is part of a larger effort to unify the way we index securities in order
to generate unique ids. Older md sources (e.g., MDP3Listener) use
SecurityIDRecorder to generate ids, which uses a different database and
different key-string formatting than the IDGenerator used by the newer md
sources (e.g., GeneralMarketDataListener and various ccapi-based sources).
2024-10-30 19:35:05 +00:00
John Colvin 6e12c5b895 refactor: Replace ConfigDownloader w/ lib_http fns
Essentially all it did was fetch config files via http GET. I ported some http
fns over from the cme-websocket-connector repo, which uses curlpp instead of
curl, and also supports async requests (coroutines) via the 'concurrencpp'

https://github.com/David-Haim/concurrencpp
2024-10-30 19:35:06 +00:00
John Colvin 2624a2c00d cleanup: Get rid of another Security type aliases 2024-10-30 19:35:06 +00:00
John Colvin fb01953296 cleanup: Remove unused code in Security.h/cpp 2024-10-30 19:35:06 +00:00
John Colvin 72a3358f44 Fix typo in 'Quantity' FMT_IMPL
I didn't realize the "value()" function rounded out the decimal,
so I replaced it with a call to "full_value()".
2024-10-30 19:35:06 +00:00
John Colvin 2a505e8ce5 diag: Log when sqlite db is created 2024-10-30 19:35:06 +00:00
John Colvin a4d4a938b0 diag: Log when an MDProducer is created 2024-10-30 19:35:06 +00:00
John Colvin c2922d5e02 diag: Add logs around synthetic security discovery in MDConsumer 2024-10-30 19:35:06 +00:00
John Colvin 8e109c6450 refactor: Break shared mem md code up into smaller files
For easier reading/navigation
2024-10-30 19:35:06 +00:00
John Colvin 85b1909543 refactor: Break md relay/query server code up into smaller files
To make code navigation easier. This was a difficulty which I recently
confronted, while investigating a bug that prevented synthetics from
showing up the ui.
2024-10-30 19:35:06 +00:00
John Colvin ba4e20552e refactor: Get rid of 'TimeStamp' typedef in MDStructure.h
Before this commit, we had 4 different TimeStamp types/identifiers, which is
confusing enough as it is:

* bts::core::TimeStamp
  - in system/time_stamp.h
* bts::utils::TimeStamp
  - in spark/utils/TimeStamp.h
* bts::TimeStamp
  - in system/bts_chrono.h
  - alias for std::chrono::system_clock::time_point
* bts::md::TimeStamp
  - in spark/md/Shm/MDStructure.h
  - alias for bts::utils::TimeStamp
  - removed in this commit

Additional aliases for these TimeStamps only adds to the confusion (esp
when the alias comes from a nested include).
2024-10-30 19:35:06 +00:00
John Colvin b1f1987922 Make security 'formattable'
So it can be used as a param in LOG statements
2024-10-30 19:35:06 +00:00
John Colvin b13725bc09 refactor: Change refs/raw ptrs to smart ptrs in md gateway 2024-10-30 19:35:06 +00:00
John Colvin 85dbee2f00 Fix typo in market_data.conf 2024-10-30 19:35:06 +00:00
John Colvin 14232f5644 Add BTC and ETH to VENUE_SIM in symbols.conf 2024-10-30 19:35:06 +00:00
John Colvin 89d49cd721 fix: Resolve main logging dir in new SparkDataDirectory class 2024-10-30 19:35:05 +00:00
John Colvin 590bc1749a refactor: Replace if/else with switch cases 2024-10-30 19:35:05 +00:00
John Colvin 339767a51f fix: Resolve onixs log & data dirs relative to app data root
When our logging code moved to lib_system, it lost access to the config
system. As a result, the "log_path" function, which was used by ICEMain
and MDP3Main, broke. I added a new class, SparkDataDirectory, to be the
central/common authority on paths to local storage.
2024-10-30 19:35:05 +00:00
John Colvin 47f2f16cd3 cleanup: Remove over-engineered subcommands
We don't need explicit subcommands to 'run onixs' md (just use config).
2024-10-30 19:35:05 +00:00
John Colvin d1b28a86c8 fix: Ensure IDGenerator instance is created before deref 2024-10-30 19:35:05 +00:00
John Colvin 50e7ad925e fix: Expand undefined vars to empty string
Bugs (fixed):
  * Md shared memory file names have unexpanded variable: '${BTS_SHARED_MEMORY_PREFIX}'
2024-10-30 19:35:05 +00:00
John Colvin 132201d2cf fix: Emplace MDConsumerHandle instead of copy assign 2024-10-30 19:35:05 +00:00
John Colvin 9c535616df fix build: Make venue conversion constexpr 2024-10-30 19:35:05 +00:00
John Colvin 1dd2db41f2 build: Use new/robust init_library fn in lib_domain 2024-10-30 19:35:05 +00:00
John Colvin a451c3e880 refactor: Move config services to lib_system
The proto files are still in lib_spark, w/ the exception of
system_config.proto, since that is a dep of ConfigurationState.
2024-10-30 19:35:05 +00:00
John Colvin 5485cc9c4f refactor: Move logging into lib_system
This is a fundamental part of our application suite, so system seems
like a sensible home.

Also moved a few logging deps (to lib_system):
* bts_chrono.h
* bts_no_copy.h
* defer.h
* dispatcher.h/cpp
* file_utils.h/cpp
* type_name.h
2024-10-30 19:35:05 +00:00
John Colvin 5e7e20df5a build: Move http code into its own lib 2024-10-30 19:35:05 +00:00
John Colvin 28f5181585 refactor: Move thread_manager to lib_system 2024-10-30 19:35:05 +00:00
John Colvin 5e7a478bf5 build (linux): Migrate catch2 dep to vcpkg (from externals) 2024-10-30 19:35:05 +00:00
John Colvin 0abc386d71 build (linux): Migrate fmt dep to vcpkg (from externals)
Now both linux and windows use vcpkg's fmt
2024-10-30 19:35:05 +00:00
John Colvin 53b040342a cleanup: Remove unnecessary includes 2024-10-30 19:31:41 +00:00
John Colvin e85f669534 refactor: Move threads.proto to lib_system 2024-10-30 19:31:41 +00:00
John Colvin 935d414b7f refactor: Move warn_ctrl.h into lib_system
This is another logging dep
2024-10-30 19:31:41 +00:00
John Colvin 89f4fe88f9 build: Link boost_filesystem in lib_system 2024-10-30 19:31:41 +00:00
John Colvin 161e0a8c3b refactor: Move os.h/cpp and deps to lib_system
This is a prerequisite for the next objective: move logging into its own lib.
As logging is a core dependency for so much of our system, this will enable
us to move more complex services (like md providers) into smaller libs.
2024-10-30 19:31:41 +00:00
John Colvin f5a6dc7a88 cleanup: Remove unused code 2024-10-30 19:31:41 +00:00
John Colvin 298a2f0617 cleanup: Remove more unnecessary includes 2024-10-30 19:31:41 +00:00
John Colvin c6fc1a0f89 refactor: Move SecurityMarketDataStatistics to lib_domain 2024-10-30 19:31:41 +00:00