Commit Graph
13692 Commits
Author SHA1 Message Date
John Colvin 2bc8cf6db5 refactor: Return row id in IStore::retrieve_one
This is a prerequisite to synchronizing security ids stored via IDGenerator
(the "new" way to store ids, added when we implemented Deribit), and
SecurityDB (the original way from spark days). The problem was that these
2 implementations were using separate databases. We need to unify the
interfaces between these storage services to avoid breaking existing code.
2024-10-17 05:47:47 +00:00
John Colvin 881651e3d6 Add database dir and file fns to SparkDataDirectory 2024-10-17 05:47:47 +00:00
John Colvin 64b3d0d092 Interpret db strng param as name or path in SqliteStore ctr 2024-10-17 05:47:47 +00:00
John Colvin 7b40f56498 refactor: Normalize sec key indexing for old md sources
Now both old and new (domain) md sources index securities by
the same string format.
2024-10-17 05:47:47 +00:00
John Colvin f4b010f3f0 Add ostream operator for Venue 2024-10-17 05:47:47 +00:00
John Colvin edd0c2b2b7 Add ostream operator for SecurityType 2024-10-17 05:47:47 +00:00
John Colvin 74b2860414 refactor: Normalize sec key indexing for new (domain) md sources
This is part of a larger effort to unify the way we index securities in order
to generate unique ids. Older md sources (e.g., MDP3Listener) use
SecurityIDRecorder to generate ids, which uses a different database and
different key-string formatting than the IDGenerator used by the newer md
sources (e.g., GeneralMarketDataListener and various ccapi-based sources).
2024-10-17 05:47:47 +00:00
John Colvin 65ac7a03ad refactor: Replace ConfigDownloader w/ lib_http fns
Essentially all it did was fetch config files via http GET. I ported some http
fns over from the cme-websocket-connector repo, which uses curlpp instead of
curl, and also supports async requests (coroutines) via the 'concurrencpp'

https://github.com/David-Haim/concurrencpp
2024-10-17 05:47:47 +00:00
John Colvin 5a5649e440 cleanup: Get rid of another Security type aliases 2024-10-17 05:47:47 +00:00
John Colvin 98d98cb899 cleanup: Remove unused code in Security.h/cpp 2024-10-17 05:47:47 +00:00
John Colvin 1f2f483e21 Fix typo in 'Quantity' FMT_IMPL
I didn't realize the "value()" function rounded out the decimal,
so I replaced it with a call to "full_value()".
2024-10-17 05:47:47 +00:00
John Colvin 0d93cce697 diag: Log when sqlite db is created 2024-10-17 05:47:47 +00:00
John Colvin fe53fd0f2b diag: Log when an MDProducer is created 2024-10-17 05:47:46 +00:00
John Colvin 0b2a0266c1 diag: Add logs around synthetic security discovery in MDConsumer 2024-10-17 05:47:46 +00:00
John Colvin 670ed49534 refactor: Break shared mem md code up into smaller files
For easier reading/navigation
2024-10-17 05:47:46 +00:00
John Colvin 3d59e2d049 refactor: Break md relay/query server code up into smaller files
To make code navigation easier. This was a difficulty which I recently
confronted, while investigating a bug that prevented synthetics from
showing up the ui.
2024-10-17 05:47:46 +00:00
John Colvin c8a7cd0961 refactor: Get rid of 'TimeStamp' typedef in MDStructure.h
Before this commit, we had 4 different TimeStamp types/identifiers, which is
confusing enough as it is:

* bts::core::TimeStamp
  - in system/time_stamp.h
* bts::utils::TimeStamp
  - in spark/utils/TimeStamp.h
* bts::TimeStamp
  - in system/bts_chrono.h
  - alias for std::chrono::system_clock::time_point
* bts::md::TimeStamp
  - in spark/md/Shm/MDStructure.h
  - alias for bts::utils::TimeStamp
  - removed in this commit

Additional aliases for these TimeStamps only adds to the confusion (esp
when the alias comes from a nested include).
2024-10-17 05:47:46 +00:00
John Colvin 7598d9a237 Make security 'formattable'
So it can be used as a param in LOG statements
2024-10-17 05:47:46 +00:00
John Colvin f4fbb304b8 refactor: Change refs/raw ptrs to smart ptrs in md gateway 2024-10-17 05:47:46 +00:00
John Colvin 7c0d2bf046 Fix typo in market_data.conf 2024-10-17 05:47:46 +00:00
John Colvin 79aa309141 Add BTC and ETH to VENUE_SIM in symbols.conf 2024-10-17 05:47:46 +00:00
John Colvin 27e834dee6 fix: Resolve main logging dir in new SparkDataDirectory class 2024-10-17 05:47:46 +00:00
John Colvin 032d53de5b refactor: Replace if/else with switch cases 2024-10-17 05:47:46 +00:00
John Colvin b0a7f28070 fix: Resolve onixs log & data dirs relative to app data root
When our logging code moved to lib_system, it lost access to the config
system. As a result, the "log_path" function, which was used by ICEMain
and MDP3Main, broke. I added a new class, SparkDataDirectory, to be the
central/common authority on paths to local storage.
2024-10-17 05:47:46 +00:00
John Colvin 0d6359904c cleanup: Remove over-engineered subcommands
We don't need explicit subcommands to 'run onixs' md (just use config).
2024-10-17 05:47:46 +00:00
John Colvin e74fb56431 fix: Ensure IDGenerator instance is created before deref 2024-10-17 05:47:46 +00:00
John Colvin cf7c0830d3 fix: Expand undefined vars to empty string
Bugs (fixed):
  * Md shared memory file names have unexpanded variable: '${BTS_SHARED_MEMORY_PREFIX}'
2024-10-17 05:47:46 +00:00
John Colvin 255c76cf3b fix: Emplace MDConsumerHandle instead of copy assign 2024-10-17 05:47:46 +00:00
John Colvin 873f9568b6 Generate c++ protos w/ relative paths
We shouldn't be generating protos defined outside of the current CMakeLists.txt dir.
2024-10-17 05:47:46 +00:00
John Colvin 17acf3424f fix build: Make venue conversion constexpr 2024-10-17 05:47:46 +00:00
John Colvin 3a36ffcb16 build: Use new/robust init_library fn in lib_domain 2024-10-17 05:47:46 +00:00
John Colvin 0883b4af48 refactor: Move config services to lib_system
The proto files are still in lib_spark, w/ the exception of
system_config.proto, since that is a dep of ConfigurationState.
2024-10-17 05:37:17 +00:00
John Colvin 8c3a12cda6 refactor: Move logging into lib_system
This is a fundamental part of our application suite, so system seems
like a sensible home.

Also moved a few logging deps (to lib_system):
* bts_chrono.h
* bts_no_copy.h
* defer.h
* dispatcher.h/cpp
* file_utils.h/cpp
* type_name.h
2024-10-17 05:37:17 +00:00
John Colvin b8799622d5 build: Move http code into its own lib 2024-10-17 05:37:16 +00:00
John Colvin 8b80c9821f refactor: Move thread_manager to lib_system 2024-10-17 05:34:30 +00:00
John Colvin 23d2c2d6c2 build (linux): Migrate catch2 dep to vcpkg (from externals) 2024-10-17 05:34:30 +00:00
John Colvin 60a34785a0 build (linux): Migrate fmt dep to vcpkg (from externals)
Now both linux and windows use vcpkg's fmt
2024-10-17 05:34:30 +00:00
John Colvin b8c4f772cc cleanup: Remove unnecessary includes 2024-10-17 05:33:12 +00:00
John Colvin d343f35fff refactor: Move threads.proto to lib_system 2024-10-17 05:33:12 +00:00
John Colvin ef7892e14e refactor: Move warn_ctrl.h into lib_system
This is another logging dep
2024-10-17 05:33:11 +00:00
John Colvin 08e8255b5a build: Link boost_filesystem in lib_system 2024-10-17 05:33:11 +00:00
John Colvin 4d1e0008d5 refactor: Move os.h/cpp and deps to lib_system
This is a prerequisite for the next objective: move logging into its own lib.
As logging is a core dependency for so much of our system, this will enable
us to move more complex services (like md providers) into smaller libs.
2024-10-17 05:33:11 +00:00
John Colvin cd2e7a5795 cleanup: Remove unused code 2024-10-17 05:33:11 +00:00
John Colvin e4a099cf6d cleanup: Remove more unnecessary includes 2024-10-17 05:33:11 +00:00
John Colvin d2bbd43ecb refactor: Move SecurityMarketDataStatistics to lib_domain 2024-10-17 05:33:11 +00:00
John Colvin 86eb53cc29 cleanup: Remove unnecessary code 2024-10-17 05:33:11 +00:00
John Colvin 95fda4f268 cleanup: Remove empty header FTXSecurity.h 2024-10-17 05:33:11 +00:00
John Colvin f7fba0b638 cleanup: Remove temporary redirect headers
These were used to make code migration smoother (to the new lib_system and lib_domain)

* spark/algorithm/TryCatch.h -> system/algorithm/try_catch.h
* spark/core/BoolName.h -> system/bool_name.h
* spark/core/EnumMacros.h -> system/enum_macros.h
* spark/core/FMT.h -> system/FMT.h
* spark/core/FMT_IMPL.h -> system/FMT_IMPL.h
* spark/core/TimeSpan.h -> system/time_span.h
* spark/core/TimeStamp.h -> system/time_stamp.h
* spark/domain/Price.h -> domain/price.h
* spark/io/JsonReader.h -> system/json/json_reader.h
* spark/io/JsonDumper.h -> system/json/json_dumper.h
2024-10-17 05:33:11 +00:00
John Colvin 08598d2b78 refactor: Move RFQ to lib_domain 2024-10-17 05:33:11 +00:00
John Colvin 5ecef5f34d refactor: Move AggressorSide proto to lib_domain
Get rid of non-proto AggressorSide enum while we're at it (consolidate to 1 type)
2024-10-17 05:33:11 +00:00