From a4ff7743da3d815028c7bfc2fb232c608dfedb19 Mon Sep 17 00:00:00 2001 From: arham Date: Fri, 4 Sep 2026 13:28:03 +0000 Subject: [PATCH] Add: simple doc to expplain security handling in sparkd --- doc/sparkd-security-id-lifecycle.md | 942 ++++++++++++++++++++++++++++ 1 file changed, 942 insertions(+) create mode 100644 doc/sparkd-security-id-lifecycle.md diff --git a/doc/sparkd-security-id-lifecycle.md b/doc/sparkd-security-id-lifecycle.md new file mode 100644 index 000000000..2ca0e400c --- /dev/null +++ b/doc/sparkd-security-id-lifecycle.md @@ -0,0 +1,942 @@ +# Security ID handling in `sparkd` + +This document traces how security identifiers are created, translated, persisted, +indexed, and published by the `sparkd` market-data paths. It focuses on the +legacy CME MDP3/OnixS pipeline and the newer generalized market-data pipeline. + +## Executive summary + +`sparkd` uses two identifiers that must not be confused: + +| Identifier | Meaning | Typical field | Lifetime | +|---|---|---|---| +| Exchange/feed ID | The venue's identifier used to route incoming definitions, books, and trades | `FixedSecurity::numeric_id`, `ExchangeKey`, or `domain::Security::security_common.exchange_sec_id` | Feed/session-specific unless the source guarantees stability | +| Spark security key | Spark's internal non-zero identifier used in shared memory, APIs, orders, positions, and downstream services | `FixedSecurity::key`, `api::Security::id()`, `SecurityKey`; historically also called `security_id` | Intended to be stable across restarts | + +`invalid_security_key` is `0`; a security with key `0` is not yet publicly +discoverable through the shared-memory consumer path. + +The two principal paths differ: + +- **MDP3/OnixS:** the definition callback sets only the CME ID in + `numeric_id`. `MDProducer::add_security()` allocates the shared-memory entry + with key `0`. `SecurityIDRecorder`, driven by a 200 ms dispatcher timer, + resolves or inserts a stable key in `key_ids.sqlite`; its callback finally + calls `MDProducerHandle::set_key()`. +- **Generalized/newer pipeline:** `GeneralMarketDataListener::on_securities()` + currently uses the ID already present on the incoming domain security + (`use_given_sec_key == true`). Live adapters normally obtain that ID + synchronously from `IDGenerator` first. `_copy_security_common()` then copies + it to both `FixedSecurity::key` and `numeric_id`, so + `MDProducer::add_security()` publishes it synchronously. Only the listener's + own fallback `IDGenerator` call is bypassed by the current constant. + +## End-to-end diagrams + +### CME MDP3 / OnixS + +```mermaid +sequenceDiagram + participant FE as OnixS feed-engine thread + participant L as MDP3Listener + participant P as MDProducer / shared memory + participant Q as SecurityIDRecorder queues + participant D as sparkd dispatcher thread + participant DB as key_ids.sqlite + + FE->>L: instrument definition (CME SecurityID) + L->>L: set_security_common() - numeric_id = CME SecurityID, key remains 0 + L->>P: add_security(FixedSecurity) + P->>P: allocate hot + cold entries with key 0 + P->>Q: ensure_key(handle) + L->>L: index[CME SecurityID] = handle + Note over FE,L: Books can route by CME ID while the Spark key is pending + D->>Q: every 200 ms - work_spreads(), work_handles(), work_db() + Q->>DB: find (namespace, descriptor), or insert a new row + DB-->>Q: SQLite rowid = Spark security key + Q->>P: handle.set_key(rowid) + P->>P: hot.key first, then cold security.key, then increment security_count + Note over P: The security is now public to shared-memory consumers +``` + +### Generalized/newer pipeline (current behavior) + +```mermaid +sequenceDiagram + participant A as Venue adapter / plugin + participant DB as IDGenerator / key_ids.sqlite + participant G as GeneralMarketDataListener + participant I as GeneralSecurityIndex + participant P as MDProducer / shared memory + + A->>DB: get_or_add(venue, type, adapter key) + DB-->>A: SQLite rowid as domain SecurityID + A->>G: on_securities(domain Security list with ID) + G->>G: _find_sec() - use incoming security_common.id + G->>G: _copy_security_common() - key = id, numeric_id = id + G->>P: add_security(FixedSecurity with non-zero key) + P->>P: set_key() during add_security() + G->>I: index[id] = handle + Note over G,P: No asynchronous ID assignment in the current configuration +``` + +## Data-flow charts + +The sequence diagrams above emphasize timing and thread transitions. These +flowcharts emphasize which identifier is carried through each representation +and where data is retained. + +### MDP3 / legacy data flow + +```mermaid +flowchart TD + CME["CME MDP3 packets
CME SecurityID"] --> ONIXS["OnixS feed engine
decode definition and updates"] + + ONIXS --> DEF["MDP3Listener definition callback"] + DEF --> FIXED["FixedSecurity
numeric_id = CME SecurityID
key = 0"] + + FIXED --> PRODUCER["MDProducer::add_security"] + PRODUCER --> PENDING["Hot and cold shared memory
allocated but not public
key = 0"] + PRODUCER --> QUEUE["SecurityIDRecorder
pending_handle queue"] + + FIXED --> CMEINDEX["CMESecurityIndex
CME SecurityID to handle"] + + QUEUE --> TIMER["Dispatcher timer
every 200 ms"] + TIMER --> LEGS{"Spread legs
all resolved?"} + LEGS -- No --> UNRESOLVED["unresolved_legs
exchange IDs awaiting keys"] + UNRESOLVED --> CMEINDEX + CMEINDEX --> LEGS + LEGS -- Yes or outright --> LOGICAL["Logical identity
venue + type + security_desc"] + + LOGICAL --> SELECT["SQLite SELECT
key_ids.sqlite"] + SELECT -- Existing row --> ROWID["SQLite rowid
Spark security key"] + SELECT -- Missing row --> INSERT["Batched SQLite INSERT"] + INSERT --> ROWID + + ROWID --> SETKEY["MDProducerHandle::set_key"] + SETKEY --> HOT["Write hot-entry key"] + HOT --> COLD["Write cold security key"] + COLD --> COUNT["Increment security_count"] + COUNT --> PUBLIC["Public security
visible to MDConsumer"] + + ONIXS --> UPDATE["Book, trade, status, or RFQ
carrying CME SecurityID"] + UPDATE --> CMEINDEX + CMEINDEX --> HANDLE["MDProducerHandle"] + HANDLE --> LIVE["Update live shared-memory data"] + PUBLIC --> LIVE +``` + +The CME index is usable before the Spark key is assigned, so feed updates can +reach the internal handle while the security is still hidden from consumers. +The persistent rowid crosses the publication boundary only at `set_key()`. + +### Generalized/newer data flow + +```mermaid +flowchart TD + FEED["Venue feed, Databento,
Python plugin, or replay"] --> ADAPTER["Source adapter
build domain::Security"] + + ADAPTER --> IDTYPE{"Live definition or
recorded/pre-keyed input?"} + IDTYPE -- Live --> IDINPUT["Logical identity
venue + type + adapter-chosen name"] + IDINPUT --> MUTEX["IDGenerator global mutex"] + MUTEX --> SELECT["Synchronous SQLite SELECT
key_ids.sqlite"] + SELECT -- Existing row --> ROWID["SQLite rowid"] + SELECT -- Missing row --> INSERT["Synchronous SQLite INSERT
autocommit"] + INSERT --> ROWID + ROWID --> DOMAINID["domain::SecurityID
used as Spark ID"] + IDTYPE -- Recorded or pre-keyed --> DOMAINID + + DOMAINID --> DOMAINSEC["SecurityCommon::id = Spark ID"] + DOMAINSEC --> LISTENER["GeneralMarketDataListener::on_securities"] + LISTENER --> CCMAP["cc_sec_id_map
incoming ID to Spark ID"] + LISTENER --> CONVERT["_copy_security_common
FixedSecurity.key = ID
numeric_id = ID"] + + CONVERT --> PRODUCER["MDProducer::add_security"] + PRODUCER --> SETKEY["Immediate set_key
no recorder timer"] + SETKEY --> SHM["Hot and cold shared memory
immediately public"] + CONVERT --> GENERALINDEX["GeneralSecurityIndex
ID to handle"] + + FEED --> EVENT["Book, trade, or statistics
with exchange/source identity"] + EVENT --> EVENTMAP["Adapter map
source identity to domain::SecurityID"] + EVENTMAP --> EVENTID["Event carries domain::SecurityID"] + EVENT --> PYHOT["Python bridge only
synchronous IDGenerator lookup"] + PYHOT --> EVENTID + PYHOT -. uses same mutex and SQLite path .-> MUTEX + EVENTID --> CCMAP + CCMAP --> GENERALINDEX + GENERALINDEX --> HANDLE["MDProducerHandle"] + HANDLE --> SHMUPDATE["Update book, trade,
statistics, or status"] + SHM --> SHMUPDATE + +``` + +In the normal newer live path, persistence occurs before the definition enters +the generalized listener. The DB call therefore adds synchronous adapter-side +latency, but publication needs no later timer. Most adapters use an in-memory +map for subsequent events; the dotted Python path is the documented exception. + +## The publication boundary + +`MDProducerHandle::set_key()` is the definitive publication point for the +legacy shared-memory representation. Its write ordering is intentional: + +1. Write the key into the hot book entry (`HotEntry` or `HotDepthEntry`). +2. Write the key into the cold security entry. +3. Increment the cold header's public `security_count`. + +The cold security key is written last because consumers treat a non-zero cold +key as visibility. At that moment the corresponding hot entry must already be +addressable. `set_key()` only permits the transition from `0` to one key; it is +not a key-update mechanism. + +## Identifier representations + +The names are historically inconsistent. In particular, many protobuf fields +named `security_id` contain the **Spark key**, not the exchange's ID. + +| Layer | Exchange/feed identifier | Spark identifier | +|---|---|---| +| MDP3 wire/OnixS | `securityId()` | not present | +| Legacy fixed security | `FixedSecurity::numeric_id` (`uint64_t`) | `FixedSecurity::key` / `security_key` (`uint32_t`) | +| Legacy API security | venue-specific `exchange_id` | top-level `Security.id()` | +| New domain model | `SecurityCommon::exchange_sec_id` (string) is descriptive; `SecurityCommon::id` is also used to correlate source events | `domain::SecurityID` (`uint64_t`), currently copied into the Spark key | +| Shared-memory hot book | none | `HotEntry::key` or `HotDepthEntry::key` | +| Shared-memory cold metadata | `FixedSecurity::numeric_id` | `FixedSecurity::key` | +| CME runtime lookup | `CMESecurityIndex` key | `SecurityData::handle.key()` | +| General runtime lookup | incoming/canonical `domain::SecurityID` | `GeneralSecurityIndex` key and `handle.key()` (currently the same value) | + +The generalized model declares `domain::SecurityID` as 64-bit, but the legacy +shared-memory `security_key` remains 32-bit. `_copy_security_common()` assigns a +64-bit value to both `numeric_id` and the 32-bit `key`. This is safe only while +allocated IDs fit in 32 bits; unlike the legacy lookup path, this assignment has +no explicit checked conversion. + +## Which parts use exchange IDs, descriptors, or Spark security IDs? + +The three identifier classes serve different jobs: + +- **Exchange/external ID:** routes data inside a venue or vendor feed adapter. +- **Descriptor/natural key:** recognizes a definition logically at the + persistence, discovery, or human-facing boundary. +- **Spark security ID/key:** joins the instrument to books, orders, fills, + positions, and services after publication. + +The core downstream system does not repeatedly identify a security from its +description. The description is primarily used to obtain a Spark key; after +that boundary, most joins use the numeric Spark key. + +### Component-level usage matrix + +| Component or data path | Exchange/external ID | Descriptor or name | Spark security ID/key | +|---|---|---|---| +| CME MDP3 definition decoder | Reads `securityId()` and stores it in `FixedSecurity::numeric_id` | Copies definition symbol/tag 55 into `security_desc` | Leaves `FixedSecurity::key` at `0` initially | +| `CMESecurityIndex` | Primary map key; CME ID resolves to a handle | Not used for update routing | Returns `handle.key()` when the Spark key has been assigned | +| MDP3 books, trades, status, statistics, and RFQs | Incoming CME ID selects the handle through `CMESecurityIndex` | Not used to locate each update | The selected handle writes keyed shared-memory/trade data after publication | +| MDP3 spread definitions | Each leg initially contains `legSecurityId()` as `exchange_key` | Parent `security_desc` identifies the spread's DB row | Each leg's Spark key and then the parent Spark key must be resolved before publication | +| Legacy ICE OnixS | Uses ICE `marketId` as `numeric_id` and index key | ICE security description becomes the logical lookup name | Uses the same recorder-assigned Spark key pattern as MDP3 | +| `SecurityIDRecorder` | Does not allocate from the external ID | Uses `(venue, security type, securities::descriptor())` to find/create a row | Treats the SQLite rowid as the resulting Spark key | +| Native generalized adapters | Use exchange strings/numeric IDs to parse definitions and correlate updates | Usually pass a description or exchange instrument name to `IDGenerator` | Put the returned ID into `SecurityCommon::id` and later events | +| Databento adapter | Uses `msg.hd.instrument_id` for feed correlation and its in-memory map | Uses the decimal instrument ID string—not the human description—as the `IDGenerator` name | Caches `instrument_id -> domain::SecurityID` and places that ID on books/trades | +| Python bridge | Reads `exchange_security_id` from definitions, books, and trades | Uses that external-ID string as the `IDGenerator` name | Resolves it to `domain::SecurityID`; currently performs this lookup for every event | +| `IDGenerator` | Does not interpret whether `name` came from an exchange ID | Uses exactly `(venue, type, caller-supplied name)` | Returns the SQLite rowid as a 64-bit ID | +| `GeneralMarketDataListener` | Does not route on `exchange_sec_id` directly | Copies `sec.desc()` into fixed metadata; its disabled fallback allocator would use `sec.desc()` | Uses incoming `SecurityCommon::id`, maps events through `cc_sec_id_map`, and copies the value into both `key` and `numeric_id` | +| `GeneralSecurityIndex` | No separate external-ID concept in the current use-given path | Not used | Maps the generalized ID/Spark key to `MDProducerHandle` | +| `MDProducer` / shared memory | Retains `numeric_id`; in legacy feeds this is the exchange ID, but in the generalized path it is currently the Spark ID | Stores `security_desc` in the cold definition | Uses the key in hot/cold entries; non-zero key is the visibility boundary | +| `MDConsumer` | Not normally used to find live books | Can enumerate/filter converted security definitions | Finds handles and book data by `security_key` | +| `SecurityCache` / `securities.sqlite` | Preserves venue-specific exchange data inside the serialized security blob | Maintains descriptor/name lookup indexes and supports venue/type/name queries | Primary persisted definition key is `api::Security.id()` | +| MBO processor and MBO service | May retain exchange **order** IDs, which are separate from security exchange IDs | Not used to find the instrument | Indexes snapshots and requests by `SecurityKey`; MBO protobuf `security_id` means Spark key | +| Order manager and algorithms | Execution gateways eventually use venue-specific fields from the resolved security to encode orders | Descriptions appear in logs and errors, not as the primary join | Order definitions, subscriptions, book events, native orders, spread legs, and fills use `security_key` / `Security.id()` | +| Position, PNL, and risk services | Not used as their position join | May resolve metadata for display/reporting | Positions, fills, overrides, and PNL handles are keyed by Spark security ID | +| Relay and market-data recordings | Serialized security definitions may preserve venue exchange IDs | Preserve full descriptors in recorded definitions | Books, updates, trades, and replay correlation preserve the recorded Spark key | +| Synthetic security service | No venue exchange ID; synthetics use `VENUE_BTS` | Synthetic name is the recorder's logical descriptor | Recorder assigns the SQLite rowid and synthetic metadata is keyed by it | + +### Exchange/external IDs: where they stop + +In MDP3 the separation is explicit: + +```text +CME SecurityID + -> FixedSecurity.numeric_id + -> CMESecurityIndex lookup + -> MDProducerHandle + -> Spark security_key for downstream publication +``` + +The external CME ID continues to route feed messages but does not become the +order/position key. This permits the same downstream Spark key to survive a +restart as long as the logical descriptor mapping remains in +`key_ids.sqlite`, even if the feed must rediscover the CME routing ID. + +The generalized path collapses more of these roles: + +```text +exchange/vendor identity + -> adapter calls IDGenerator + -> domain::SecurityID + -> FixedSecurity.key and FixedSecurity.numeric_id + -> GeneralSecurityIndex and downstream Spark key +``` + +`SecurityCommon::exchange_sec_id` may still contain the original textual +external identifier, but `GeneralMarketDataListener` does not use it to route +events and does not put it in `numeric_id`. The adapter must translate later +events to `domain::SecurityID` before invoking the listener. + +### Descriptor/name users + +Descriptors are used in three main places: + +1. **Key allocation:** `SecurityIDRecorder` uses the legacy venue-specific + descriptor, while `IDGenerator` uses its caller-provided `name`. +2. **Definition lookup and discovery:** `SecurityCache` supports + venue/type/name lookup; symbol filters and administrative tools use security + metadata to select or display instruments. +3. **Diagnostics:** logs include symbols and descriptions so humans can + understand a numeric key. + +Descriptions generally do not drive the high-rate book, trade, order, fill, or +position paths. Those paths use numeric IDs after definition resolution. + +It is important that “descriptor” does not mean one universal canonical field: + +- Legacy CME uses Globex `security_desc` from tag 55. +- Many generalized adapters use `sec.desc()` or an exchange instrument name. +- Databento uses its numeric instrument ID converted to a string. +- Python uses `exchange_security_id`. + +Therefore, two adapters may describe the same economic contract but request +different logical rows and receive different Spark IDs. + +### Spark security-ID users + +Once a security is published, its Spark key is the dominant system identifier: + +```mermaid +flowchart LR + KEY["Spark security key"] --> SHM["Hot/cold shared memory"] + KEY --> MDC["MDConsumer and book subscriptions"] + KEY --> MBO["MBO snapshots and queries"] + KEY --> OMS["Orders, algorithms, and spread legs"] + KEY --> FILLS["Fills and execution records"] + KEY --> POS["Positions, PNL, and risk"] + KEY --> API["Services and client APIs"] + KEY --> REC["Relay and market-data recordings"] + KEY --> SECDB["Serialized security definitions"] +``` + +Fields named `security_id`, `security_key`, or top-level `Security.id()` in +these downstream areas almost always mean the Spark key. They should not be +compared directly with CME `SecurityID`, Databento `instrument_id`, or another +venue/vendor identifier without going through the relevant adapter/index +mapping. + +## Persistence and allocation + +### The authoritative key mapping: `key_ids.sqlite` + +Both allocators use `SqliteStore("key_ids")`. With the normal configuration the +file is: + +```text +/databases/key_ids/key_ids.sqlite +``` + +The database contains one table and two indexes: + +```sql +CREATE TABLE store (ns TEXT, key TEXT, value TEXT); +CREATE UNIQUE INDEX key_index ON store(ns, key); +CREATE INDEX ns_index ON store(ns); +``` + +The SQLite `rowid` is the generated Spark key. `ns` is normally +`.` (for example `GLBX.FUTURE`), and `key` is the security +descriptor chosen by the caller. `value` is normally empty. Therefore ID +stability depends on all three of these remaining stable: + +- the configured `data_path` and preservation of this database; +- the venue/type namespace; +- the descriptor supplied by the feed adapter. + +The two allocator implementations share the file and convention: + +- `SecurityIDRecorder` (legacy): derives the namespace from legacy API enums and + uses `securities::descriptor()`. For CME this is the Globex + `security_desc`, populated from the definition's symbol/tag 55. +- `IDGenerator` (newer): derives the namespace from new domain enums and uses + the caller's `name`. Most adapters pass `sec.desc()` or an exchange instrument + identifier; Databento deliberately passes the decimal instrument ID, and the + Python bridge uses the exchange security ID. + +### Logical-row examples + +The following names are illustrative. Production uses the exact string emitted +by the selected feed/adapter, including its case, spacing, and weekly-series +notation. + +#### Legacy `SecurityIDRecorder` + +For CME, `set_security_common()` copies the MDP3 definition's symbol/tag 55 into +Globex `security_desc`. After converting the fixed security to +`api::Security`, `securities::descriptor()` returns that same +`glbx.security_desc()` string. + +| Instrument | Legacy enums | Example descriptor from `security_desc` | Row searched/inserted in `store` | +|---|---|---|---| +| Future | `VENUE_GLBX`, `SECURITYTYPE_FUTURE` | `ESZ6` | `ns='GLBX.FUTURE', key='ESZ6'` | +| Standard option | `VENUE_GLBX`, `SECURITYTYPE_OPTION` | `ESZ6 C6500` | `ns='GLBX.OPTION', key='ESZ6 C6500'` | +| Weekly option | `VENUE_GLBX`, `SECURITYTYPE_OPTION` | `ESZ6-W1 C6500` | `ns='GLBX.OPTION', key='ESZ6-W1 C6500'` | +| Spot from a legacy-compatible venue path | `VENUE_COINBASE`, `SECURITYTYPE_SPOT` | `BTC-USD` | `ns='COINBASE.SPOT', key='BTC-USD'` | + +A weekly option does not get a special namespace. Standard and weekly options +are both `SECURITYTYPE_OPTION`, so both live in `GLBX.OPTION`; the exact +`security_desc` distinguishes the weekly series. For example, if the database +assigned rowids `42001` through `42004` to the rows above, those integers—not +the descriptor strings—would be the Spark security keys. + +Conceptually, legacy allocation is: + +```text +future: lookup_or_insert("GLBX.FUTURE", "ESZ6") +option: lookup_or_insert("GLBX.OPTION", "ESZ6 C6500") +weekly option: lookup_or_insert("GLBX.OPTION", "ESZ6-W1 C6500") +spot: lookup_or_insert("COINBASE.SPOT", "BTC-USD") +``` + +#### Newer `IDGenerator` + +The namespace result is similar, but the third argument is chosen by the +adapter. Examples using a description-oriented adapter are: + +```cpp +get_or_add(VENUE_GLBX, SECURITY_TYPE_FUTURE, "ESZ6"); +// store: ns="GLBX.FUTURE", key="ESZ6" + +get_or_add(VENUE_GLBX, SECURITY_TYPE_OPTION, "ESZ6 C6500"); +// store: ns="GLBX.OPTION", key="ESZ6 C6500" + +get_or_add(VENUE_GLBX, SECURITY_TYPE_OPTION, "ESZ6-W1 C6500"); +// store: ns="GLBX.OPTION", key="ESZ6-W1 C6500" + +get_or_add(VENUE_COINBASE, SECURITY_TYPE_SPOT, "BTC-USD"); +// store: ns="COINBASE.SPOT", key="BTC-USD" +``` + +Databento intentionally produces a different logical row even when the human +description is `ESZ6`. If its instrument ID were `12345678`, it would call the +equivalent of: + +```cpp +get_or_add(VENUE_GLBX, SECURITY_TYPE_FUTURE, "12345678"); +// store: ns="GLBX.FUTURE", key="12345678" +``` + +Likewise, the Python bridge uses `exchange_security_id`. A spot definition with +`exchange_security_id="BTC-USD"` produces the `COINBASE.SPOT / BTC-USD` row, +while a provider that sends `exchange_security_id="BTCUSD"` produces a +different row and therefore a different Spark key. + +These examples show why the tuple is only logical when the adapter's naming +contract is also known. `GLBX.FUTURE / ESZ6` and `GLBX.FUTURE / 12345678` may +describe the same contract to a human but are distinct rows to `key_ids.sqlite`. + +Both enum-prefix removals currently produce the same namespace spelling (for +example, `GLBX.OPTION`), so both mechanisms allocate from one global SQLite +rowid space. Live rows cannot share a rowid. The table does not declare +`AUTOINCREMENT`, however, so deleting rows—especially the highest row—can allow +SQLite to reuse a number later. Security-key rows should therefore be treated +as append-only operational data. + +`SqliteStore` uses WAL mode and `PRAGMA synchronous = 0`. This favors startup +and feed latency over maximum durability against a host/power failure. The +unique `(ns, key)` index prevents two different rows for the same logical +security. + +### Other places the key is stored + +| Location | Purpose | Authoritative for allocation? | +|---|---|---| +| Hot and cold shared-memory segments | Live publication and fast consumer lookup | No; rebuilt at process startup | +| `CMESecurityIndex` / `GeneralSecurityIndex` and adapter maps | In-process routing from an incoming ID to a producer handle | No; in-memory only | +| `/databases/md/securities.sqlite` | Serialized legacy `api::Security` definitions keyed by Spark key; used by security cache and MDP3 Sunday preload | No; it stores metadata after a key exists | +| Relay/market-data recordings | Preserve the key embedded in recorded securities, books, and trades for deterministic replay | No; replay normally reuses the recorded value | +| Orders, fills, positions, APIs | Reference a security by the published Spark key | No | + +The `SecurityIDRecorder::work_db()` comment says a published security is also +recorded in the security database, but that function does **not** write +`securities.sqlite` in the current code. Full security definitions are persisted +separately by `SecurityCache::flush_to_disk()` through +`db::insert_or_replace()`. + +## CME MDP3 / OnixS pipeline in detail + +### 1. Construction and thread setup + +`MDP3Main::PImpl` owns one `CMESecurityIndex`, one `SecurityIDRecorder`, and one +`MDProducer`. It constructs the recorder with: + +- a lookup callback that maps `ExchangeKey` through the CME index to the + handle's current Spark key; and +- `Dispatcher::current()`, which drives a 200 ms recorder timer. + +In the current `MarketDataGateway` route, `OnixsGlbxSubscription` creates a +dedicated `dispatcher_thread`, constructs and starts `MDP3Main` on that thread, +then calls that thread's dispatcher `run()`. The older `md run onixs` command +uses its command/dispatcher thread similarly. + +The OnixS feed engine has its own configured thread pool. Instrument and market +data callbacks therefore originate on OnixS feed-engine threads, while recorder +database work and final key assignment execute on the MDP3 dispatcher thread. + +| MDP3 stage | Executing thread | DB call? | +|---|---|---| +| Decode definition and set CME `numeric_id` | OnixS feed-engine callback thread | No | +| Allocate hot/cold entry and queue handle | Same OnixS callback thread | No | +| Add CME-ID index entry | Same OnixS callback thread | No | +| Resolve spread legs | MDP3 dispatcher thread, on 200 ms timer | No | +| Select/insert Spark key | MDP3 dispatcher thread, on 200 ms timer | Yes | +| `handle.set_key()` and public visibility | MDP3 dispatcher thread, in DB callback | No additional call | + +### 2. Definition callback: exchange ID is set, Spark key is not + +For futures, options, and spreads, `MDP3Listener::set_security_common()` does: + +```cpp +fsec.numeric_id = def.securityId(); +``` + +It also copies symbol/descriptor/venue and pricing fields. A newly constructed +`FixedSecurity` already has `key == invalid_security_key` (`0`), and this helper +does not change it. + +For a spread, every leg starts with: + +```cpp +leg.exchange_key = ExchangeKey(definition_leg.legSecurityId()); +leg.key = invalid_security_key; +``` + +Thus both the parent and legs initially carry exchange-routing identity, but no +Spark identity. + +### 3. Shared-memory allocation and exchange indexing + +`MDP3Listener::add_security()` performs these operations in order: + +1. `_producer.add_security(security)` allocates a hot entry and a variable-sized + cold entry and copies the security into the cold segment. +2. Because the key is `0`, `MDProducer` queues the handle through + `SecurityIDRecorder::ensure_key()`. +3. `_index.add_security(security.numeric_id, handle)` immediately creates the + CME-ID-to-handle mapping. +4. The initial exchange status is written through the handle. + +This ordering means incoming books can already be routed by CME ID while the +Spark key is pending. Shared-memory consumers cannot discover the security yet. +Also, `MDProducerHandle::add_trade()` only appends to the public trade queue +when `key() != 0`, so a trade received in this short window updates the cold +last-trade state but is not added to the keyed trade log. + +### 4. Recorder tick: spreads, handles, then database + +Every 200 ms the recorder calls, in this exact order: + +1. `work_spreads()` tries to fill previously unresolved spread leg keys by + asking `CMESecurityIndex` for each leg exchange ID. +2. `work_handles()` drains new handles. Outrights are converted to an API + security and queued for DB lookup immediately. A spread without all leg keys + is retained in `pending_handle` and its leg pointers are registered in + `unresolved_legs`. +3. `work_db()` resolves queued `(venue, type, descriptor)` requests. + +Because spread work precedes new-handle work, a newly seen unresolved spread +normally waits until at least the next timer tick to resolve legs. It waits +longer if any outright leg has not yet received its Spark key. A spread's own +key is not requested until every leg key is non-zero. + +### 5. Lookup or generation + +`work_db()` first queries `(ns, key)`: + +- Existing row: use its `rowid` as the key and invoke the callback. +- Missing row: batch it into `insert_many_resolve_ids()`. SQLite assigns a new + rowid inside a transaction, the row object receives that ID, and the callback + runs after the transaction completes. + +New inserts are throttled to roughly 200 per timer pass (the current `> 200` +check can admit 201) to avoid delaying heartbeats during startup. The queues are +TBB concurrent queues, but all SQLite access is serialized onto the recorder's +dispatcher thread. `check_access()` asserts this single-thread rule after the +first work pass records the dispatcher thread ID. + +### 6. Final assignment and visibility + +The queued callback captures a copy of `MDProducerHandle` and calls +`handle.set_key(id)` on the dispatcher thread. `set_key()` writes hot, then cold, +then increments `security_count`. From this point: + +- shared-memory scans report the security; +- `CMESecurityIndex::get_key(CME_ID)` returns the Spark key; +- trades and RFQs can be emitted with a non-zero key; +- resolved spreads can copy this key into their leg structures. + +### 7. Sunday preload is a fast path + +For configured symbols on Sunday, `MDP3Main::prepare_listener()` loads stored +definitions from `securities.sqlite`. It converts them with +`set_security_key=true`, so both the security key and spread leg keys come from +the serialized API security. `MDProducer::add_security()` calls `set_key()` +immediately, and no recorder lookup is required for that preloaded handle. The +CME exchange ID is still restored into `numeric_id`, allowing the normal CME +index to be built. + +## Generalized/newer pipeline in detail + +### 1. Topology and ownership + +`Run::run()` starts a detached thread named `MarketDataGateway`. On that thread, +`run_md_gwy()` obtains the process-wide `IDGenerator`, constructs +`MarketDataGateway`, and starts the configured providers. For each `(venue, +source)` returned by a provider, the gateway creates an `MdSink` containing: + +- a venue-local `GeneralSecurityIndex`; +- a default `MDProducer` (not wired to a `SecurityIDRecorder`); and +- a `GeneralMarketDataListener` connecting the source to that producer/index. + +The source decides its callback thread. Some sources call the listener while +starting; others own network or replay threads. The generalized listener calls +the producer inline on that source thread. The process-wide `IDGenerator` +serializes its own calls with a mutex, but the listener's transient +`cc_sec_id_map` is an ordinary `unordered_map`, so each source/listener pair is +expected to serialize security/event callbacks. + +| Generalized stage | Executing thread | DB call? | +|---|---|---| +| Build/parse domain definition | Source/adaptor thread | No | +| `IDGenerator::get_or_add()` | Same source/adaptor thread | Yes, synchronously | +| `GeneralMarketDataListener::on_securities()` | Same source callback thread | No in current use-given mode | +| Copy ID to `FixedSecurity::key` and `numeric_id` | Same source callback thread | No | +| `MDProducer::add_security()` / `set_key()` | Same source callback thread | No additional call | +| Add `GeneralSecurityIndex` entry | Same source callback thread | No | + +There are two important compatibility exceptions inside the provider framework: + +- `OnixsGlbxSubscription::start()` ignores the supplied generalized listener + and starts `MDP3Main`; CME/OnixS therefore still follows the asynchronous + legacy flow documented above. +- `EmumbaDeribitSubscription::start()` also ignores that listener and uses + `EmMDListener`, which allocates via `IDGenerator` and writes through its own + producer/index. Its assignment is still synchronous and pre-populated, so it + resembles the newer ID flow even though it does not pass through + `GeneralMarketDataListener`. + +### 2. How a new ID is constructed + +For a normal live generalized adapter, ID creation happens **in the adapter, +before** `GeneralMarketDataListener::on_securities()`: + +```text +logical identity = (domain venue, domain security type, adapter-chosen name) +Spark ID = SQLite rowid for that identity in key_ids.sqlite +``` + +`IDGenerator::get_or_add(venue, type, name)` formats a namespace by stripping +the enum prefixes: + +```text +VENUE_TAIFEX + SECURITY_TYPE_FUTURE -> TAIFEX.FUTURE +``` + +It then calls `retrieve_one(ns, name)`. If the row exists, it returns that +row's `rowid`. If absent, it inserts `(ns, name, "")`, and SQLite assigns the +next rowid. That integer is wrapped in `domain::SecurityID` and placed in +`SecurityCommon::id`. + +The third identity component is adapter-specific, and this matters when trying +to reproduce a mapping: + +| Source/adaptor family | Typical `name` sent to `IDGenerator` | +|---|---| +| Binance, Coinbase, Deribit, OKX and similar native/CCAPI paths | Usually `sec.desc()` / exchange instrument name | +| Taifex | product/instrument ID string | +| Databento | `std::to_string(msg.hd.instrument_id)`, not the human-readable description | +| Python bridge | `exchange_security_id` (and it asserts Python supplied ID is zero for definitions) | +| Emumba Deribit | extracted Emumba security descriptor | +| Replay | no new allocation for a recorded definition; it reuses the recorded `SecurityCommon::id` | + +So “security name” alone is not a universal logical key. Reproduction requires +the exact venue enum, exact security-type enum, and the exact adapter-specific +name used at insertion time. + +### 3. What `GeneralMarketDataListener` does with that ID + +`on_securities()` first calls `_find_sec(sec, true)`. Its current, compile-time +behavior is controlled by: + +```cpp +const bool use_given_sec_key = true; +``` + +Consequently, for a newly seen incoming `domain::SecurityID` it does **not** +call the DB. It copies `security_common.id.value()` into its transient +`cc_sec_id_map`. `_add_future()`, `_add_option()`, `_add_spread()`, or +`_add_spot()` then calls `_copy_security_common()`, which sets: + +```cpp +to.key = incoming_id; +to.numeric_id = incoming_id; +``` + +This collapses the generalized source/correlation ID and Spark shared-memory +key into the same number. The source's text `exchange_sec_id` remains on the +domain security but is not copied into `FixedSecurity::numeric_id` by this +path. + +`_commit_security()` then: + +1. calls `MDProducer::add_security()` with an already non-zero key; +2. causes `MDProducerHandle::set_key()` to run immediately during that call; +3. inserts the handle in `GeneralSecurityIndex` under `numeric_id` (the same + number); and +4. returns with the security already visible in shared memory. + +Because the sink's producer has no recorder, supplying `id == 0` would allocate +the shared-memory entry but never assign or publish its key. Generalized live +sources must therefore provide a valid non-zero ID before `on_securities()`. + +The listener contains an alternative branch for `use_given_sec_key == false`. +That branch would call `IDGenerator::get_or_add(me->venue, sec.type(), +sec.desc())` itself. It is currently disabled, and `_generate_security_key()` is +also effectively unused. Turning the flag off is not a mechanical change: an +adapter such as Databento allocates using numeric instrument ID while the +listener branch would use `sec.desc()`, potentially creating a second key for +the same instrument. + +There is also a spread failure edge case. `_find_sec(sec, true)` records the +parent ID in `cc_sec_id_map` before `_add_spread()` resolves its legs. If any leg +ID is absent, `_add_spread()` logs and returns without committing the spread. A +later repeat of the parent definition is then classified as a duplicate and +ignored, so definition ordering is a hard requirement in the current code. + +### 4. How later books, trades, and statistics find the security + +Every later domain event carries a `domain::SecurityID`: + +- `Book::security_id()`; +- `TradeRecord::sec_id`; or +- `SecurityMarketDataStatistics::security_id`. + +The listener first uses `cc_sec_id_map` to translate that event ID to the value +stored in `GeneralSecurityIndex`, then obtains the `MDProducerHandle`. With +`use_given_sec_key == true`, this mapping is numerically identity-to-identity, +but it still serves as the admission check that the definition was seen. + +Most high-throughput adapters avoid SQLite on market-data events: + +- Databento allocates on the definition and caches + `instrument_id -> domain::SecurityID` in `sec_id_map` for books/trades. +- Taifex keeps the resolved ID in its security-details mapper. +- Other native subscriptions maintain comparable descriptor/ID maps. + +The Python bridge is a notable exception: `to_bts_security_()`, +`to_bts_trade_()`, and `to_book_()` all call `get_sec_id_()`, which calls +`IDGenerator::get_id(..., add_if_new=true)`. As written, that performs at least +one SQLite `SELECT` for each converted Python definition, trade, or book. It +returns the existing ID after the first insert, but it is still a DB call and +takes the global `IDGenerator` mutex on every event. + +### 5. Persistence timing and latency + +The newer allocator is synchronous: + +- Existing security: mutex acquisition plus one SQLite `SELECT` on the caller's + thread. +- New security: mutex acquisition, one `SELECT`, then one SQLite `INSERT` + (SQLite autocommit), also on the caller's thread. +- Listener publication: no further DB call under the current + `use_given_sec_key` setting. + +The ID therefore exists in `key_ids.sqlite` before the adapter sends the +definition to the listener. There is no intentional 200 ms publication delay, +but the source callback/startup path directly pays DB and mutex latency. Bulk +definition discovery performs one lookup and possibly one insert per security; +`IDGenerator` does not batch those operations. + +WAL mode helps concurrency, while `synchronous=0` reduces fsync cost. It does +not eliminate SQL parsing/stepping, mutex contention, page-cache misses, or +occasional file-system work. The impact is usually a startup/definition cost +when adapters cache the result, but it becomes hot-path latency for code such as +the current Python bridge that resolves every event through the DB. + +`GeneralMarketDataListener` itself does not persist the full contract definition +to `securities.sqlite`; that remains the separate security-cache/metadata +persistence path described earlier. + +## Comparison: MDP3 recorder vs generalized allocator + +| Property | MDP3 / `SecurityIDRecorder` | Generalized / `IDGenerator` | +|---|---|---| +| Logical input | venue + type + legacy `security_desc` | venue + type + adapter-chosen name | +| Numeric output | SQLite rowid, checked/read as 32-bit `security_key` | SQLite rowid returned as 64-bit `domain::SecurityID`, then narrowed into 32-bit shared-memory key | +| Allocation point | After definition is copied to shared memory | Normally inside adapter before listener sees definition | +| DB execution | Asynchronous on dispatcher timer | Synchronous on caller/source thread | +| Existing ID cost | One deferred `SELECT` | One immediate `SELECT` per call | +| New ID cost | Deferred, batched transaction (about 200/pass) | Immediate `SELECT` + individual autocommit `INSERT` | +| Time until public | Usually 0–200+ ms; spreads may take extra ticks | Immediate after synchronous lookup/insert completes | +| Feed callback blocking | Queue push only for ID work | Yes, if the adapter invokes `IDGenerator` there | +| Spread legs | Parent waits asynchronously until all leg keys resolve by exchange ID | Definition processing requires every leg's incoming ID already exist in `cc_sec_id_map`; otherwise the spread is aborted | +| Runtime ID relationship | Exchange ID and Spark key remain distinct | Incoming/canonical ID, `numeric_id`, index ID, and Spark key are currently the same number | +| Persistence database | `key_ids.sqlite` | The same `key_ids.sqlite` | +| Batching | Yes for new rows | No | +| Failure behavior | Security has an internal handle with key 0 while normal queued work is pending; DB work does not block the feed callback | DB errors/zero results occur in the synchronous source path; a zero ID can leave an unpublishable entry, with no pending-key state machine | + +Neither algorithm generates a UUID or a content hash. Both ultimately use an +opaque, database-assigned SQLite rowid. It is normally increasing and looks like +a sequence, but this schema does not request SQLite `AUTOINCREMENT`. The tuple +used to *look up* the row is logical; the resulting number is not derivable from +that tuple. + +## Can another system reconstruct the same keys? + +Not from `(venue, type, name)` alone. Another system can reconstruct the +**logical lookup tuple**, but it cannot calculate the numeric Spark key because +the key depends on the historical insertion order of every row in +`key_ids.sqlite`. + +For example, two empty systems can insert the exact same `GLBX.FUTURE / ESZ6` +row and receive different IDs if other securities were inserted before it on +one system. They are sequential opaque IDs, not deterministic hashes and not +UUIDs. + +Ways to preserve cross-system identity are: + +1. Copy/replicate `key_ids.sqlite` (including rowids) and keep descriptor rules + identical. This preserves existing numeric keys but requires coordinated + allocation for new ones. +2. Export an explicit mapping of `(namespace, adapter key) -> Spark key`, then + import rows with their original rowids before either system allocates new + securities. +3. Treat a stable external identity (venue + type + exchange instrument ID, or + another canonical instrument identifier) as the cross-system key and carry a + translation table to each system's local Spark key. +4. Introduce a deterministic ID scheme only as a versioned migration. A hash can + be computed independently, but collision handling, descriptor + canonicalization, aliases/renames, the existing 32-bit ABI, and compatibility + with orders/recordings/databases all need explicit design. + +Copying only `securities.sqlite` is insufficient as an allocator strategy. It +contains security blobs keyed by already-assigned IDs, but `key_ids.sqlite` is +the mapping used to find or allocate those IDs. Conversely, copying only +`key_ids.sqlite` preserves the numbers but not all contract metadata. + +## Other assignment paths in `sparkd` + +The two main pipelines reduce to four ways a producer handle can acquire a key: + +1. **Already keyed input.** `MDProducer::add_security()` sees a non-zero + `FixedSecurity::key` and calls `set_key()` immediately. Generalized feeds, + Sunday preload, relay playback, and other replay paths use this mechanism. +2. **Recorder-backed live input.** The input key is zero and the producer owns a + `SecurityIDRecorder`; `ensure_key()` queues asynchronous assignment. CME + MDP3 and legacy OnixS ICE use this mechanism. ICE uses `marketId` as + `numeric_id` and otherwise follows the same recorder/index design. +3. **Service assignment.** `SyntheticSecurityService` calls + `SecurityIDRecorder::assign_key()` directly on a background dispatcher. The + callback receives a copied `api::Security` with its top-level `id` set, then + publishes it to synthetic subscribers. It uses the same `key_ids.sqlite` + rowid allocator. +4. **Unkeyed input with no recorder.** The entry is allocated with key zero and + never becomes public. This is not a valid production assignment path, but it + is the failure mode if a generalized source omits its ID. + +Relay and replay deserve care when reasoning about persistence: + +- The generalized `MDReplayer` forwards the recorded domain security ID and + emits later books/statistics with the recorded `sec_key`, so the listener's + current use-given mode preserves the recording's numeric identity. +- The legacy relay producer converts a recorded `api::Security` with + `set_security_key=true`, so its shared-memory publication also preserves the + embedded key. +- The legacy relay recorder additionally calls `bulk_insert(..., "REPLAY: ")` + for logical mappings. Those rows use a `REPLAY: ` namespace prefix; this is a + separate mapping population step, not a deterministic derivation of the key. + +## Operational implications + +- Back up `key_ids.sqlite` and `securities.sqlite` together if numeric ID and + definition continuity matter. +- Do not delete/rebuild `key_ids.sqlite` on one host and assume IDs will match + another host, even when all instruments are identical. +- Before moving a feed between adapters, compare the exact `name` each adapter + supplies to `IDGenerator`. Matching venue/type is insufficient. +- Cache generated IDs in the source after definition processing. Avoid calling + `IDGenerator` from book/trade hot paths; the Python bridge currently violates + this guideline. +- Keep generated rowids below the 32-bit shared-memory ceiling until the + `security_key` ABI is widened or explicit bounds checking is added to the + generalized path. +- Ensure spread legs are announced before generalized spread parents. +- Treat `synchronous=0` as a durability decision. A copied DB should be taken + with SQLite-aware backup/checkpoint handling rather than by casually copying + active WAL files. + +Useful inspection queries (run against the configured DB path) are: + +```sql +SELECT rowid AS spark_key, ns, key +FROM store +ORDER BY rowid; + +SELECT rowid AS spark_key, ns, key +FROM store +WHERE ns = 'GLBX.FUTURE' AND key = ''; +``` + +## Source map + +- [`SecurityIDRecorder.cpp`](../src/lib_spark/spark/md/SecurityIDRecorder.cpp): + legacy queues, 200 ms work ordering, logical row construction, DB callbacks, + and spread-leg resolution. +- [`IDGenerator.cpp`](../src/lib_spark/spark/persistence/IDGenerator.cpp): + synchronous generalized lookup/allocation and process-wide mutex. +- [`SqliteStore.cpp`](../src/lib_spark/spark/persistence/SqliteStore.cpp): DB + location, schema, WAL/synchronous pragmas, individual inserts, and batch + inserts. +- [`MDProducer.cpp`](../src/lib_spark/spark/md/MDProducer.cpp) and + [`MDProducerHandle.cpp`](../src/lib_spark/spark/md/MDProducerHandle.cpp): + shared-memory allocation, recorder handoff, and ordered publication. +- [`FixedSecurity.h`](../src/lib_spark/spark/md/Shm/FixedSecurity.h) and + [`MDStructure.h`](../src/lib_spark/spark/md/Shm/MDStructure.h): identifier + widths and hot/cold storage fields. +- [`MDP3Main.cpp`](../src/app_sparkd/sparkd/md/platforms/mdp3_onixs/MDP3Main.cpp) + and + [`MDP3Listener.h`](../src/app_sparkd/sparkd/md/platforms/mdp3_onixs/MDP3Listener.h): + MDP3 construction, definition callbacks, CME ID extraction, and spread legs. +- [`MDP3Listener.cpp`](../src/app_sparkd/sparkd/md/platforms/mdp3_onixs/MDP3Listener.cpp): + producer/index insertion order. +- [`OnixsGlbxSubscription.cpp`](../src/app_sparkd/sparkd/md/glbx/OnixsGlbxSubscription.cpp): + current dedicated MDP3 dispatcher-thread ownership. +- [`MarketDataGateway.cpp`](../src/app_sparkd/sparkd/md/MarketDataGateway.cpp) + and + [`GeneralMarketDataListener.cpp`](../src/lib_spark/spark/md/GeneralMarketDataListener.cpp): + generalized source/sink topology, incoming-ID mapping, fixed-security + conversion, and publication. +- [`DatabentoDataHandler.cpp`](../src/app_sparkd/sparkd/md/databento/DatabentoDataHandler.cpp), + [`TaifexDataProcessors.h`](../src/app_sparkd/sparkd/md/taifex/TaifexDataProcessors.h), + and + [`PyMarketDataSubscription.cpp`](../src/app_sparkd/sparkd/md/PyMarketDataSubscription.cpp): + representative adapter-side allocation and caching behavior. +- [`SecurityDB.cpp`](../src/lib_spark/spark/md/SecurityDB.cpp) and + [`SecurityCache.cpp`](../src/lib_spark/spark/md/SecurityCache.cpp): separate + persistence of full security definitions. +- [`SecurityIndex.h`](../src/lib_spark/spark/md/SecurityIndex.h), + [`MBOServiceServer.cpp`](../src/lib_spark/spark/md/MBOServiceServer.cpp), and + [`MDConsumer.cpp`](../src/lib_spark/spark/md/MDConsumer.cpp): exchange-index + translation followed by Spark-keyed market-data consumption and MBO access. +- [`OrderFactory.cpp`](../src/lib_spark/spark/orderman/order_factory/OrderFactory.cpp), + [`FillStore.cpp`](../src/lib_spark/spark/orderman/actors/FillStore.cpp), and + [`PositionMap.cpp`](../src/app_sparkd/sparkd/position/PositionMap.cpp): + representative downstream use of Spark security keys. +- [`MDReplayer.cpp`](../src/app_sparkd/sparkd/md/recording/MDReplayer.cpp) and + [`RelayClient.cpp`](../src/lib_spark/spark/md/relay/RelayClient.cpp): recorded + ID reuse.