Commit Graph
13695 Commits
Author SHA1 Message Date
John Colvin e87e4b2dcb refactor: Change raw IDGenerator ptr to shared_ptr 2024-10-21 14:48:11 +00:00
John Colvin f793a3af9b refactor: Return row id in IStore::retrieve_one
This is a prerequisite to synchronizing security ids stored via IDGenerator
(the "new" way to store ids, added when we implemented Deribit), and
SecurityDB (the original way from spark days). The problem was that these
2 implementations were using separate databases. We need to unify the
interfaces between these storage services to avoid breaking existing code.
2024-10-21 14:48:11 +00:00
John Colvin bbc7e90c47 Add database dir and file fns to SparkDataDirectory 2024-10-21 14:48:11 +00:00
John Colvin 0f912842d8 Interpret db strng param as name or path in SqliteStore ctr 2024-10-21 14:48:11 +00:00
John Colvin 8f25e18b4b refactor: Normalize sec key indexing for old md sources
Now both old and new (domain) md sources index securities by
the same string format.
2024-10-21 14:48:11 +00:00
John Colvin 987862cb64 Add ostream operator for Venue 2024-10-21 14:48:11 +00:00
John Colvin 1f6b89a84d Add ostream operator for SecurityType 2024-10-21 14:48:11 +00:00
John Colvin cf5df0f5d5 refactor: Normalize sec key indexing for new (domain) md sources
This is part of a larger effort to unify the way we index securities in order
to generate unique ids. Older md sources (e.g., MDP3Listener) use
SecurityIDRecorder to generate ids, which uses a different database and
different key-string formatting than the IDGenerator used by the newer md
sources (e.g., GeneralMarketDataListener and various ccapi-based sources).
2024-10-21 14:48:10 +00:00
John Colvin 75f6c8110a refactor: Replace ConfigDownloader w/ lib_http fns
Essentially all it did was fetch config files via http GET. I ported some http
fns over from the cme-websocket-connector repo, which uses curlpp instead of
curl, and also supports async requests (coroutines) via the 'concurrencpp'

https://github.com/David-Haim/concurrencpp
2024-10-21 14:31:48 +00:00
John Colvin b6a4148149 cleanup: Get rid of another Security type aliases 2024-10-21 14:31:48 +00:00
John Colvin 66ec306205 cleanup: Remove unused code in Security.h/cpp 2024-10-21 14:31:48 +00:00
John Colvin 79abe5a283 Fix typo in 'Quantity' FMT_IMPL
I didn't realize the "value()" function rounded out the decimal,
so I replaced it with a call to "full_value()".
2024-10-21 14:31:48 +00:00
John Colvin 8c12d4a4aa diag: Log when sqlite db is created 2024-10-21 14:31:48 +00:00
John Colvin 0f3407a5b7 diag: Log when an MDProducer is created 2024-10-21 14:31:48 +00:00
John Colvin ae21d215be diag: Add logs around synthetic security discovery in MDConsumer 2024-10-21 14:31:48 +00:00
John Colvin dd90bf183a refactor: Break shared mem md code up into smaller files
For easier reading/navigation
2024-10-21 14:31:48 +00:00
John Colvin e6d41cf01a refactor: Break md relay/query server code up into smaller files
To make code navigation easier. This was a difficulty which I recently
confronted, while investigating a bug that prevented synthetics from
showing up the ui.
2024-10-21 14:31:48 +00:00
John Colvin 51be818dff refactor: Get rid of 'TimeStamp' typedef in MDStructure.h
Before this commit, we had 4 different TimeStamp types/identifiers, which is
confusing enough as it is:

* bts::core::TimeStamp
  - in system/time_stamp.h
* bts::utils::TimeStamp
  - in spark/utils/TimeStamp.h
* bts::TimeStamp
  - in system/bts_chrono.h
  - alias for std::chrono::system_clock::time_point
* bts::md::TimeStamp
  - in spark/md/Shm/MDStructure.h
  - alias for bts::utils::TimeStamp
  - removed in this commit

Additional aliases for these TimeStamps only adds to the confusion (esp
when the alias comes from a nested include).
2024-10-21 14:31:48 +00:00
John Colvin 89de23de7a Make security 'formattable'
So it can be used as a param in LOG statements
2024-10-21 14:31:48 +00:00
John Colvin 7afb4bbcc1 refactor: Change refs/raw ptrs to smart ptrs in md gateway 2024-10-21 14:31:48 +00:00
John Colvin 1f4fcdaabd Fix typo in market_data.conf 2024-10-21 14:31:48 +00:00
John Colvin d3b417c732 Add BTC and ETH to VENUE_SIM in symbols.conf 2024-10-21 14:31:48 +00:00
John Colvin db0c1d56eb fix: Resolve main logging dir in new SparkDataDirectory class 2024-10-21 14:31:48 +00:00
John Colvin 69dffec5cc refactor: Replace if/else with switch cases 2024-10-21 14:31:48 +00:00
John Colvin 1a212cb9c6 fix: Resolve onixs log & data dirs relative to app data root
When our logging code moved to lib_system, it lost access to the config
system. As a result, the "log_path" function, which was used by ICEMain
and MDP3Main, broke. I added a new class, SparkDataDirectory, to be the
central/common authority on paths to local storage.
2024-10-21 14:31:48 +00:00
John Colvin 5c297157f9 cleanup: Remove over-engineered subcommands
We don't need explicit subcommands to 'run onixs' md (just use config).
2024-10-21 14:31:48 +00:00
John Colvin d315848003 fix: Ensure IDGenerator instance is created before deref 2024-10-21 14:31:48 +00:00
John Colvin a55db502b9 fix: Expand undefined vars to empty string
Bugs (fixed):
  * Md shared memory file names have unexpanded variable: '${BTS_SHARED_MEMORY_PREFIX}'
2024-10-21 14:31:48 +00:00
John Colvin 69049c985c fix: Emplace MDConsumerHandle instead of copy assign 2024-10-21 14:31:48 +00:00
John Colvin 859d16eab7 Generate c++ protos w/ relative paths
We shouldn't be generating protos defined outside of the current CMakeLists.txt dir.
2024-10-21 14:31:48 +00:00
John Colvin e83f5336d9 fix build: Make venue conversion constexpr 2024-10-21 14:31:48 +00:00
John Colvin 24faec2bf6 build: Use new/robust init_library fn in lib_domain 2024-10-21 14:31:48 +00:00
John Colvin 14fa797938 refactor: Move config services to lib_system
The proto files are still in lib_spark, w/ the exception of
system_config.proto, since that is a dep of ConfigurationState.
2024-10-21 14:31:47 +00:00
John Colvin a1945354d3 refactor: Move logging into lib_system
This is a fundamental part of our application suite, so system seems
like a sensible home.

Also moved a few logging deps (to lib_system):
* bts_chrono.h
* bts_no_copy.h
* defer.h
* dispatcher.h/cpp
* file_utils.h/cpp
* type_name.h
2024-10-21 14:18:38 +00:00
John Colvin d93581cd08 build: Move http code into its own lib 2024-10-21 14:10:00 +00:00
John Colvin 69a6f99b98 refactor: Move thread_manager to lib_system 2024-10-21 14:10:00 +00:00
John Colvin 32d971762d build (linux): Migrate catch2 dep to vcpkg (from externals) 2024-10-21 14:10:00 +00:00
John Colvin fd8d536baf build (linux): Migrate fmt dep to vcpkg (from externals)
Now both linux and windows use vcpkg's fmt
2024-10-21 14:10:00 +00:00
John Colvin 1e8fe2fc13 cleanup: Remove unnecessary includes 2024-10-21 14:10:00 +00:00
John Colvin fdfc9fbfaa refactor: Move threads.proto to lib_system 2024-10-21 14:10:00 +00:00
John Colvin 245c7deb80 refactor: Move warn_ctrl.h into lib_system
This is another logging dep
2024-10-21 14:10:00 +00:00
John Colvin 14d8e635f9 build: Link boost_filesystem in lib_system 2024-10-21 14:10:00 +00:00
John Colvin 56e504820e refactor: Move os.h/cpp and deps to lib_system
This is a prerequisite for the next objective: move logging into its own lib.
As logging is a core dependency for so much of our system, this will enable
us to move more complex services (like md providers) into smaller libs.
2024-10-21 14:10:00 +00:00
John Colvin fe20bbb531 cleanup: Remove unused code 2024-10-21 14:10:00 +00:00
John Colvin 35a2a2ebc1 cleanup: Remove more unnecessary includes 2024-10-21 14:10:00 +00:00
John Colvin 30df01452b refactor: Move SecurityMarketDataStatistics to lib_domain 2024-10-21 14:10:00 +00:00
John Colvin 96535bdf09 cleanup: Remove unnecessary code 2024-10-21 14:10:00 +00:00
John Colvin 1ae2366538 cleanup: Remove empty header FTXSecurity.h 2024-10-21 14:10:00 +00:00
John Colvin 251c84367a cleanup: Remove temporary redirect headers
These were used to make code migration smoother (to the new lib_system and lib_domain)

* spark/algorithm/TryCatch.h -> system/algorithm/try_catch.h
* spark/core/BoolName.h -> system/bool_name.h
* spark/core/EnumMacros.h -> system/enum_macros.h
* spark/core/FMT.h -> system/FMT.h
* spark/core/FMT_IMPL.h -> system/FMT_IMPL.h
* spark/core/TimeSpan.h -> system/time_span.h
* spark/core/TimeStamp.h -> system/time_stamp.h
* spark/domain/Price.h -> domain/price.h
* spark/io/JsonReader.h -> system/json/json_reader.h
* spark/io/JsonDumper.h -> system/json/json_dumper.h
2024-10-21 14:09:59 +00:00
John Colvin 04495235a8 refactor: Move RFQ to lib_domain 2024-10-21 13:57:15 +00:00