Commit Graph
2784 Commits
Author SHA1 Message Date
John Colvin 9930b65ddc refactor: Split MarketDataGateway up into providers/sources
MarketDataGateway still exists, but is greatly simplified. It creates
several `IMarketDataProvider`s, each of which generates a list of
`IMarketDataSource`s. The market data sources provide market data to an
`IMarketDataListener`.

* EmumbaMarketDataProvider
* CCApiMarketDataProvider
* PyMarketDataProvider
* OnixsMarketDataProvider
* ReplayMarketDataProvider

As a side effect, individual py market data sources are now explicitly
limited to a single venue (it probably should have been this way all
along, since the shared memory writers do not expect updates to involve
multiple venues).
2025-05-20 20:20:07 +00:00
arham bc5ddade7f Add: okx exe small changes to maybe make ping better 2025-05-01 09:42:54 +00:00
John Colvin b4f5e66466 fix: Set okx perp exp to year 2100
Because the year 3000 is too big for signed int64
nanosecond timestamp
2025-04-29 19:35:15 +00:00
John Colvin 69a0bcc9cd fix: Request binance fut md on ccapi dispatcher thread 2025-04-29 19:32:21 +00:00
John Colvin d72ebdf421 perf: Send Binance Usds Futs md rest reqs on background thread 2025-04-29 19:32:21 +00:00
John Colvin 8e93856767 fix: Send exchangeInfo req via our HttpClient instead of ccapi
For binance usds futs (we already did this for spots)

Bugs (fixed):
  * We get a boost beast exception about the read limit being exceeded
    when requesting securities from Binance prod url (testnet works).
2025-04-29 19:32:21 +00:00
John Colvin c2150ae450 refactor: Move BinanceUsdsFuturesFilterTuple fns to cpp file 2025-04-29 19:32:21 +00:00
John Colvin 3756a8a8b7 Add api key type enum to binance exe gateway config
ED25519 keys offer performance benefits over the HMAC-SHA256
keys when using the websocket api, as you can log in on the websocket
connection to avoid signing each request.
2025-04-29 19:32:21 +00:00
John Colvin 61f04c4605 fix: Implement state machine for binance cxl/replace 2025-04-28 17:12:59 +00:00
John Colvin ab217401eb cleanup: Move CreateOrderMsg impl to .cpp, get rid of unused code 2025-04-28 17:12:58 +00:00
John Colvin 40ad1f1b48 Update binance websocket endpoints
The api just changed, and old urls will be retired in May 2025
2025-04-28 17:12:58 +00:00
John Colvin 484e36b330 refactor: Make timestamp internal value to signed (int64)
I am sick of casting uint64 -> int64. The latter is a much more
common argument type than the former.
2025-04-28 17:12:58 +00:00
John Colvin 5476466d55 Fetch secrets for config profiles with new aws sso config 2025-04-28 17:12:58 +00:00
John Colvin 29bf7d8d7e deploy (fix): Update okx urls
Their aws domain has been retired.
2025-04-28 17:12:58 +00:00
John Colvin 98bc78463d build (linux): Migrate catch2 dep to vcpkg (from externals) 2025-04-28 17:12:58 +00:00
John Colvin 2a17d656a5 fix: Dump json to str in LOG statements
The nlohmann::json obj isn't formattable (i.e., doesn't have a
fmt::formatter<T> specialization)
2025-04-28 17:12:58 +00:00
John Colvin 623012b0b0 deploy: Add "sparkd_symbols" image to Dockerfile
Normally, we bake spark config into the image, which is simpler and less
error prone than managing separate configurations per host machine
and/or container. The sparkd_symbols deployment breaks the mold. Config
is mounted to a dir on the host machine so symbols can be added/removed
via config (i.,e., spark.conf) when we bounce sparkd.
2025-04-17 05:53:05 +00:00
arham 960a8af77a Fix: OKX futures working now 2025-04-10 05:06:17 +00:00
arham 14e3eee0bc Add: config changes to make server controller work and set up hk zen server 2025-04-09 09:42:47 -04:00
John Colvin 7c7c689170 fix build: Don't cast to unsigned int when checking bytesRead < 0 2025-04-04 15:36:55 -04:00
John Colvin 116506c8da fix build: Lock databento version to v0.31.0
Previously, we were simply fetching HEAD. This was fine until they
released v0.32.0 (on 04/02/2025), which introduced a new dependency
requirement that broke our build: OpenSSL >= 3.0.
2025-04-04 15:36:55 -04:00
Adeel Khan 3feda49ef1 Fix: Taifex Time Conversion 2025-04-04 15:36:54 -04:00
Adeel Khan 10318be7bc FIX: Taifex Crash Fixed, Added Diag Logs 2025-04-04 15:36:54 -04:00
Adeel Khan f7d525a527 ADD: TBBO & Symbol wise Naming in Recorder 2025-04-04 15:36:54 -04:00
Adeel Khan d2fe9f167a ADD: TWSE Code Complete 2025-04-04 15:36:54 -04:00
Adeel Khan d575ba0ea8 ADD: TWSE throughout Sparkd 2025-04-04 15:36:54 -04:00
John Colvin e888149c41 Add order msg throttle to sparkd config profiles 2025-03-20 17:29:43 +00:00
John Colvin d49efef93d Increase ram to 8gb in alphaworx_prod config
The previous value of 4gb wasn't enough, as the container would
occasionally crash with a SIGKILL interrupt.
2025-03-19 15:26:14 +00:00
John Colvin 3980f2d030 Add order msg throttle to alphaworx_prod config 2025-03-17 13:00:15 +00:00
John Colvin 6d4d8f6ac1 fix: Disable ccapi order book cross check for Binance md
When enabled, the checks fail regularly (usually more than once per
hour). Sometimes many of them happen in rapid succession before sparkd
crashes. I'm disabling them in the hopes that the crosses are benign,
and this will stop the crashes.
2025-03-17 00:51:54 +00:00
John Colvin 379a46afa2 Add order msg throttle to sim_dev config 2025-03-15 20:07:14 +00:00
John Colvin b251e1d47e build: Generate semantic version header from git
Dump it to console and log file when sparkd runs
2025-03-14 16:00:06 +00:00
Mohammad Arham 5ba7901258 Fix: undo special case for btc swap and add ccp for swaps 2025-03-12 20:47:44 +00:00
Mohammad Arham bc6eb59110 Fix: Change cttype from optional to string 2025-03-12 20:45:27 +00:00
Mohammad Arham ff86b10895 Fix: temp fix to move okx btc swap to config for min tick 2025-03-11 20:35:24 +00:00
John Colvin d0abca3fd9 Fix symbols and account alphaworx_prod config 2025-03-11 19:47:34 +00:00
John Colvin 7895fe7357 Remove taifex and glbx from NotSimExecution in conf profiles 2025-03-11 15:24:08 +00:00
John Colvin 2072105c0f cleanup: Add using aliases to make code more concise 2025-03-10 14:53:07 +00:00
John Colvin d663c86aa0 improvement: Only load synth metadata from disk once in md dumper
Instead of every time a synth is dumped (i.e., do it lazily, outside the
loop body)
2025-03-10 14:53:07 +00:00
John Colvin 4a2dfa46a1 improvement: Print delimiter line when dumping secs to console 2025-03-10 14:53:07 +00:00
John Colvin 6e35db6a24 improvement: Apply filter to shared mem secs in md dumper
Previously, filter only applied to db securities (i.e., when --db flag
was used)
2025-03-10 14:52:39 +00:00
John Colvin a1912907b8 Add brief readmes for sparkd config profiles 2025-03-10 05:24:49 +00:00
John Colvin b1b3dcdeb8 Add sim_dev config profile for sparkd 2025-03-10 05:24:49 +00:00
John Colvin 96f28acf91 Add dev config profile for sparkd 2025-03-10 05:24:49 +00:00
John Colvin c2c2658e21 Add alphaworx_prod config profile for sparkd 2025-03-10 05:24:49 +00:00
John Colvin d6b8e359cd Expand environment vars in binance gw config 2025-03-10 03:05:59 +00:00
John Colvin 84a0850d95 Tweak binance_dev config profile
* Factor out repeated symbol list into SymbolLists entry
* Change "binance_dev" account to "BINANCE_DEV" in exe gw json
* Adjust log level
* Remove unused config:
  - ExchangeSubAccounts
  - JetFIX_Connection.max_order_msgs_per_window
2025-03-10 03:05:59 +00:00
John Colvin b9fe39f6e3 refactor: Make MDRecorder join the run thread in destructor
This wasn't broken, but detached threads bother me.
2025-03-10 03:05:59 +00:00
John Colvin b555850015 build: Package & deploy sparkd with docker compose 2025-03-10 03:05:59 +00:00
Mohammad Arham 93b93969f5 Add: config to control databento md better 2025-03-06 02:44:40 +00:00