Add: simple doc to expplain security handling in sparkd

This commit is contained in:
arham committed 2026-09-04 13:28:03 +00:00
1 parent 2853af9e0f
commit a4ff7743da
1 file changed
+942
+942
View File
@@ -0,0 +1,942 @@
# Security ID handling in `sparkd`
This document traces how security identifiers are created, translated, persisted,
indexed, and published by the `sparkd` market-data paths. It focuses on the
legacy CME MDP3/OnixS pipeline and the newer generalized market-data pipeline.
## Executive summary
`sparkd` uses two identifiers that must not be confused:
| Identifier | Meaning | Typical field | Lifetime |
|---|---|---|---|
| Exchange/feed ID | The venue's identifier used to route incoming definitions, books, and trades | `FixedSecurity::numeric_id`, `ExchangeKey`, or `domain::Security::security_common.exchange_sec_id` | Feed/session-specific unless the source guarantees stability |
| Spark security key | Spark's internal non-zero identifier used in shared memory, APIs, orders, positions, and downstream services | `FixedSecurity::key`, `api::Security::id()`, `SecurityKey`; historically also called `security_id` | Intended to be stable across restarts |
`invalid_security_key` is `0`; a security with key `0` is not yet publicly
discoverable through the shared-memory consumer path.
The two principal paths differ:
- **MDP3/OnixS:** the definition callback sets only the CME ID in
`numeric_id`. `MDProducer::add_security()` allocates the shared-memory entry
with key `0`. `SecurityIDRecorder`, driven by a 200 ms dispatcher timer,
resolves or inserts a stable key in `key_ids.sqlite`; its callback finally
calls `MDProducerHandle::set_key()`.
- **Generalized/newer pipeline:** `GeneralMarketDataListener::on_securities()`
currently uses the ID already present on the incoming domain security
(`use_given_sec_key == true`). Live adapters normally obtain that ID
synchronously from `IDGenerator` first. `_copy_security_common()` then copies
it to both `FixedSecurity::key` and `numeric_id`, so
`MDProducer::add_security()` publishes it synchronously. Only the listener's
own fallback `IDGenerator` call is bypassed by the current constant.
## End-to-end diagrams
### CME MDP3 / OnixS
```mermaid
sequenceDiagram
participant FE as OnixS feed-engine thread
participant L as MDP3Listener
participant P as MDProducer / shared memory
participant Q as SecurityIDRecorder queues
participant D as sparkd dispatcher thread
participant DB as key_ids.sqlite
FE->>L: instrument definition (CME SecurityID)
L->>L: set_security_common() - numeric_id = CME SecurityID, key remains 0
L->>P: add_security(FixedSecurity)
P->>P: allocate hot + cold entries with key 0
P->>Q: ensure_key(handle)
L->>L: index[CME SecurityID] = handle
Note over FE,L: Books can route by CME ID while the Spark key is pending
D->>Q: every 200 ms - work_spreads(), work_handles(), work_db()
Q->>DB: find (namespace, descriptor), or insert a new row
DB-->>Q: SQLite rowid = Spark security key
Q->>P: handle.set_key(rowid)
P->>P: hot.key first, then cold security.key, then increment security_count
Note over P: The security is now public to shared-memory consumers
```
### Generalized/newer pipeline (current behavior)
```mermaid
sequenceDiagram
participant A as Venue adapter / plugin
participant DB as IDGenerator / key_ids.sqlite
participant G as GeneralMarketDataListener
participant I as GeneralSecurityIndex
participant P as MDProducer / shared memory
A->>DB: get_or_add(venue, type, adapter key)
DB-->>A: SQLite rowid as domain SecurityID
A->>G: on_securities(domain Security list with ID)
G->>G: _find_sec() - use incoming security_common.id
G->>G: _copy_security_common() - key = id, numeric_id = id
G->>P: add_security(FixedSecurity with non-zero key)
P->>P: set_key() during add_security()
G->>I: index[id] = handle
Note over G,P: No asynchronous ID assignment in the current configuration
```
## Data-flow charts
The sequence diagrams above emphasize timing and thread transitions. These
flowcharts emphasize which identifier is carried through each representation
and where data is retained.
### MDP3 / legacy data flow
```mermaid
flowchart TD
CME["CME MDP3 packets<br/>CME SecurityID"] --> ONIXS["OnixS feed engine<br/>decode definition and updates"]
ONIXS --> DEF["MDP3Listener definition callback"]
DEF --> FIXED["FixedSecurity<br/>numeric_id = CME SecurityID<br/>key = 0"]
FIXED --> PRODUCER["MDProducer::add_security"]
PRODUCER --> PENDING["Hot and cold shared memory<br/>allocated but not public<br/>key = 0"]
PRODUCER --> QUEUE["SecurityIDRecorder<br/>pending_handle queue"]
FIXED --> CMEINDEX["CMESecurityIndex<br/>CME SecurityID to handle"]
QUEUE --> TIMER["Dispatcher timer<br/>every 200 ms"]
TIMER --> LEGS{"Spread legs<br/>all resolved?"}
LEGS -- No --> UNRESOLVED["unresolved_legs<br/>exchange IDs awaiting keys"]
UNRESOLVED --> CMEINDEX
CMEINDEX --> LEGS
LEGS -- Yes or outright --> LOGICAL["Logical identity<br/>venue + type + security_desc"]
LOGICAL --> SELECT["SQLite SELECT<br/>key_ids.sqlite"]
SELECT -- Existing row --> ROWID["SQLite rowid<br/>Spark security key"]
SELECT -- Missing row --> INSERT["Batched SQLite INSERT"]
INSERT --> ROWID
ROWID --> SETKEY["MDProducerHandle::set_key"]
SETKEY --> HOT["Write hot-entry key"]
HOT --> COLD["Write cold security key"]
COLD --> COUNT["Increment security_count"]
COUNT --> PUBLIC["Public security<br/>visible to MDConsumer"]
ONIXS --> UPDATE["Book, trade, status, or RFQ<br/>carrying CME SecurityID"]
UPDATE --> CMEINDEX
CMEINDEX --> HANDLE["MDProducerHandle"]
HANDLE --> LIVE["Update live shared-memory data"]
PUBLIC --> LIVE
```
The CME index is usable before the Spark key is assigned, so feed updates can
reach the internal handle while the security is still hidden from consumers.
The persistent rowid crosses the publication boundary only at `set_key()`.
### Generalized/newer data flow
```mermaid
flowchart TD
FEED["Venue feed, Databento,<br/>Python plugin, or replay"] --> ADAPTER["Source adapter<br/>build domain::Security"]
ADAPTER --> IDTYPE{"Live definition or<br/>recorded/pre-keyed input?"}
IDTYPE -- Live --> IDINPUT["Logical identity<br/>venue + type + adapter-chosen name"]
IDINPUT --> MUTEX["IDGenerator global mutex"]
MUTEX --> SELECT["Synchronous SQLite SELECT<br/>key_ids.sqlite"]
SELECT -- Existing row --> ROWID["SQLite rowid"]
SELECT -- Missing row --> INSERT["Synchronous SQLite INSERT<br/>autocommit"]
INSERT --> ROWID
ROWID --> DOMAINID["domain::SecurityID<br/>used as Spark ID"]
IDTYPE -- Recorded or pre-keyed --> DOMAINID
DOMAINID --> DOMAINSEC["SecurityCommon::id = Spark ID"]
DOMAINSEC --> LISTENER["GeneralMarketDataListener::on_securities"]
LISTENER --> CCMAP["cc_sec_id_map<br/>incoming ID to Spark ID"]
LISTENER --> CONVERT["_copy_security_common<br/>FixedSecurity.key = ID<br/>numeric_id = ID"]
CONVERT --> PRODUCER["MDProducer::add_security"]
PRODUCER --> SETKEY["Immediate set_key<br/>no recorder timer"]
SETKEY --> SHM["Hot and cold shared memory<br/>immediately public"]
CONVERT --> GENERALINDEX["GeneralSecurityIndex<br/>ID to handle"]
FEED --> EVENT["Book, trade, or statistics<br/>with exchange/source identity"]
EVENT --> EVENTMAP["Adapter map<br/>source identity to domain::SecurityID"]
EVENTMAP --> EVENTID["Event carries domain::SecurityID"]
EVENT --> PYHOT["Python bridge only<br/>synchronous IDGenerator lookup"]
PYHOT --> EVENTID
PYHOT -. uses same mutex and SQLite path .-> MUTEX
EVENTID --> CCMAP
CCMAP --> GENERALINDEX
GENERALINDEX --> HANDLE["MDProducerHandle"]
HANDLE --> SHMUPDATE["Update book, trade,<br/>statistics, or status"]
SHM --> SHMUPDATE
```
In the normal newer live path, persistence occurs before the definition enters
the generalized listener. The DB call therefore adds synchronous adapter-side
latency, but publication needs no later timer. Most adapters use an in-memory
map for subsequent events; the dotted Python path is the documented exception.
## The publication boundary
`MDProducerHandle::set_key()` is the definitive publication point for the
legacy shared-memory representation. Its write ordering is intentional:
1. Write the key into the hot book entry (`HotEntry` or `HotDepthEntry`).
2. Write the key into the cold security entry.
3. Increment the cold header's public `security_count`.
The cold security key is written last because consumers treat a non-zero cold
key as visibility. At that moment the corresponding hot entry must already be
addressable. `set_key()` only permits the transition from `0` to one key; it is
not a key-update mechanism.
## Identifier representations
The names are historically inconsistent. In particular, many protobuf fields
named `security_id` contain the **Spark key**, not the exchange's ID.
| Layer | Exchange/feed identifier | Spark identifier |
|---|---|---|
| MDP3 wire/OnixS | `securityId()` | not present |
| Legacy fixed security | `FixedSecurity::numeric_id` (`uint64_t`) | `FixedSecurity::key` / `security_key` (`uint32_t`) |
| Legacy API security | venue-specific `exchange_id` | top-level `Security.id()` |
| New domain model | `SecurityCommon::exchange_sec_id` (string) is descriptive; `SecurityCommon::id` is also used to correlate source events | `domain::SecurityID` (`uint64_t`), currently copied into the Spark key |
| Shared-memory hot book | none | `HotEntry::key` or `HotDepthEntry::key` |
| Shared-memory cold metadata | `FixedSecurity::numeric_id` | `FixedSecurity::key` |
| CME runtime lookup | `CMESecurityIndex` key | `SecurityData::handle.key()` |
| General runtime lookup | incoming/canonical `domain::SecurityID` | `GeneralSecurityIndex` key and `handle.key()` (currently the same value) |
The generalized model declares `domain::SecurityID` as 64-bit, but the legacy
shared-memory `security_key` remains 32-bit. `_copy_security_common()` assigns a
64-bit value to both `numeric_id` and the 32-bit `key`. This is safe only while
allocated IDs fit in 32 bits; unlike the legacy lookup path, this assignment has
no explicit checked conversion.
## Which parts use exchange IDs, descriptors, or Spark security IDs?
The three identifier classes serve different jobs:
- **Exchange/external ID:** routes data inside a venue or vendor feed adapter.
- **Descriptor/natural key:** recognizes a definition logically at the
persistence, discovery, or human-facing boundary.
- **Spark security ID/key:** joins the instrument to books, orders, fills,
positions, and services after publication.
The core downstream system does not repeatedly identify a security from its
description. The description is primarily used to obtain a Spark key; after
that boundary, most joins use the numeric Spark key.
### Component-level usage matrix
| Component or data path | Exchange/external ID | Descriptor or name | Spark security ID/key |
|---|---|---|---|
| CME MDP3 definition decoder | Reads `securityId()` and stores it in `FixedSecurity::numeric_id` | Copies definition symbol/tag 55 into `security_desc` | Leaves `FixedSecurity::key` at `0` initially |
| `CMESecurityIndex` | Primary map key; CME ID resolves to a handle | Not used for update routing | Returns `handle.key()` when the Spark key has been assigned |
| MDP3 books, trades, status, statistics, and RFQs | Incoming CME ID selects the handle through `CMESecurityIndex` | Not used to locate each update | The selected handle writes keyed shared-memory/trade data after publication |
| MDP3 spread definitions | Each leg initially contains `legSecurityId()` as `exchange_key` | Parent `security_desc` identifies the spread's DB row | Each leg's Spark key and then the parent Spark key must be resolved before publication |
| Legacy ICE OnixS | Uses ICE `marketId` as `numeric_id` and index key | ICE security description becomes the logical lookup name | Uses the same recorder-assigned Spark key pattern as MDP3 |
| `SecurityIDRecorder` | Does not allocate from the external ID | Uses `(venue, security type, securities::descriptor())` to find/create a row | Treats the SQLite rowid as the resulting Spark key |
| Native generalized adapters | Use exchange strings/numeric IDs to parse definitions and correlate updates | Usually pass a description or exchange instrument name to `IDGenerator` | Put the returned ID into `SecurityCommon::id` and later events |
| Databento adapter | Uses `msg.hd.instrument_id` for feed correlation and its in-memory map | Uses the decimal instrument ID string—not the human description—as the `IDGenerator` name | Caches `instrument_id -> domain::SecurityID` and places that ID on books/trades |
| Python bridge | Reads `exchange_security_id` from definitions, books, and trades | Uses that external-ID string as the `IDGenerator` name | Resolves it to `domain::SecurityID`; currently performs this lookup for every event |
| `IDGenerator` | Does not interpret whether `name` came from an exchange ID | Uses exactly `(venue, type, caller-supplied name)` | Returns the SQLite rowid as a 64-bit ID |
| `GeneralMarketDataListener` | Does not route on `exchange_sec_id` directly | Copies `sec.desc()` into fixed metadata; its disabled fallback allocator would use `sec.desc()` | Uses incoming `SecurityCommon::id`, maps events through `cc_sec_id_map`, and copies the value into both `key` and `numeric_id` |
| `GeneralSecurityIndex` | No separate external-ID concept in the current use-given path | Not used | Maps the generalized ID/Spark key to `MDProducerHandle` |
| `MDProducer` / shared memory | Retains `numeric_id`; in legacy feeds this is the exchange ID, but in the generalized path it is currently the Spark ID | Stores `security_desc` in the cold definition | Uses the key in hot/cold entries; non-zero key is the visibility boundary |
| `MDConsumer` | Not normally used to find live books | Can enumerate/filter converted security definitions | Finds handles and book data by `security_key` |
| `SecurityCache` / `securities.sqlite` | Preserves venue-specific exchange data inside the serialized security blob | Maintains descriptor/name lookup indexes and supports venue/type/name queries | Primary persisted definition key is `api::Security.id()` |
| MBO processor and MBO service | May retain exchange **order** IDs, which are separate from security exchange IDs | Not used to find the instrument | Indexes snapshots and requests by `SecurityKey`; MBO protobuf `security_id` means Spark key |
| Order manager and algorithms | Execution gateways eventually use venue-specific fields from the resolved security to encode orders | Descriptions appear in logs and errors, not as the primary join | Order definitions, subscriptions, book events, native orders, spread legs, and fills use `security_key` / `Security.id()` |
| Position, PNL, and risk services | Not used as their position join | May resolve metadata for display/reporting | Positions, fills, overrides, and PNL handles are keyed by Spark security ID |
| Relay and market-data recordings | Serialized security definitions may preserve venue exchange IDs | Preserve full descriptors in recorded definitions | Books, updates, trades, and replay correlation preserve the recorded Spark key |
| Synthetic security service | No venue exchange ID; synthetics use `VENUE_BTS` | Synthetic name is the recorder's logical descriptor | Recorder assigns the SQLite rowid and synthetic metadata is keyed by it |
### Exchange/external IDs: where they stop
In MDP3 the separation is explicit:
```text
CME SecurityID
-> FixedSecurity.numeric_id
-> CMESecurityIndex lookup
-> MDProducerHandle
-> Spark security_key for downstream publication
```
The external CME ID continues to route feed messages but does not become the
order/position key. This permits the same downstream Spark key to survive a
restart as long as the logical descriptor mapping remains in
`key_ids.sqlite`, even if the feed must rediscover the CME routing ID.
The generalized path collapses more of these roles:
```text
exchange/vendor identity
-> adapter calls IDGenerator
-> domain::SecurityID
-> FixedSecurity.key and FixedSecurity.numeric_id
-> GeneralSecurityIndex and downstream Spark key
```
`SecurityCommon::exchange_sec_id` may still contain the original textual
external identifier, but `GeneralMarketDataListener` does not use it to route
events and does not put it in `numeric_id`. The adapter must translate later
events to `domain::SecurityID` before invoking the listener.
### Descriptor/name users
Descriptors are used in three main places:
1. **Key allocation:** `SecurityIDRecorder` uses the legacy venue-specific
descriptor, while `IDGenerator` uses its caller-provided `name`.
2. **Definition lookup and discovery:** `SecurityCache` supports
venue/type/name lookup; symbol filters and administrative tools use security
metadata to select or display instruments.
3. **Diagnostics:** logs include symbols and descriptions so humans can
understand a numeric key.
Descriptions generally do not drive the high-rate book, trade, order, fill, or
position paths. Those paths use numeric IDs after definition resolution.
It is important that “descriptor” does not mean one universal canonical field:
- Legacy CME uses Globex `security_desc` from tag 55.
- Many generalized adapters use `sec.desc()` or an exchange instrument name.
- Databento uses its numeric instrument ID converted to a string.
- Python uses `exchange_security_id`.
Therefore, two adapters may describe the same economic contract but request
different logical rows and receive different Spark IDs.
### Spark security-ID users
Once a security is published, its Spark key is the dominant system identifier:
```mermaid
flowchart LR
KEY["Spark security key"] --> SHM["Hot/cold shared memory"]
KEY --> MDC["MDConsumer and book subscriptions"]
KEY --> MBO["MBO snapshots and queries"]
KEY --> OMS["Orders, algorithms, and spread legs"]
KEY --> FILLS["Fills and execution records"]
KEY --> POS["Positions, PNL, and risk"]
KEY --> API["Services and client APIs"]
KEY --> REC["Relay and market-data recordings"]
KEY --> SECDB["Serialized security definitions"]
```
Fields named `security_id`, `security_key`, or top-level `Security.id()` in
these downstream areas almost always mean the Spark key. They should not be
compared directly with CME `SecurityID`, Databento `instrument_id`, or another
venue/vendor identifier without going through the relevant adapter/index
mapping.
## Persistence and allocation
### The authoritative key mapping: `key_ids.sqlite`
Both allocators use `SqliteStore("key_ids")`. With the normal configuration the
file is:
```text
<SystemConfig.data_path>/databases/key_ids/key_ids.sqlite
```
The database contains one table and two indexes:
```sql
CREATE TABLE store (ns TEXT, key TEXT, value TEXT);
CREATE UNIQUE INDEX key_index ON store(ns, key);
CREATE INDEX ns_index ON store(ns);
```
The SQLite `rowid` is the generated Spark key. `ns` is normally
`<VENUE>.<SECURITY_TYPE>` (for example `GLBX.FUTURE`), and `key` is the security
descriptor chosen by the caller. `value` is normally empty. Therefore ID
stability depends on all three of these remaining stable:
- the configured `data_path` and preservation of this database;
- the venue/type namespace;
- the descriptor supplied by the feed adapter.
The two allocator implementations share the file and convention:
- `SecurityIDRecorder` (legacy): derives the namespace from legacy API enums and
uses `securities::descriptor()`. For CME this is the Globex
`security_desc`, populated from the definition's symbol/tag 55.
- `IDGenerator` (newer): derives the namespace from new domain enums and uses
the caller's `name`. Most adapters pass `sec.desc()` or an exchange instrument
identifier; Databento deliberately passes the decimal instrument ID, and the
Python bridge uses the exchange security ID.
### Logical-row examples
The following names are illustrative. Production uses the exact string emitted
by the selected feed/adapter, including its case, spacing, and weekly-series
notation.
#### Legacy `SecurityIDRecorder`
For CME, `set_security_common()` copies the MDP3 definition's symbol/tag 55 into
Globex `security_desc`. After converting the fixed security to
`api::Security`, `securities::descriptor()` returns that same
`glbx.security_desc()` string.
| Instrument | Legacy enums | Example descriptor from `security_desc` | Row searched/inserted in `store` |
|---|---|---|---|
| Future | `VENUE_GLBX`, `SECURITYTYPE_FUTURE` | `ESZ6` | `ns='GLBX.FUTURE', key='ESZ6'` |
| Standard option | `VENUE_GLBX`, `SECURITYTYPE_OPTION` | `ESZ6 C6500` | `ns='GLBX.OPTION', key='ESZ6 C6500'` |
| Weekly option | `VENUE_GLBX`, `SECURITYTYPE_OPTION` | `ESZ6-W1 C6500` | `ns='GLBX.OPTION', key='ESZ6-W1 C6500'` |
| Spot from a legacy-compatible venue path | `VENUE_COINBASE`, `SECURITYTYPE_SPOT` | `BTC-USD` | `ns='COINBASE.SPOT', key='BTC-USD'` |
A weekly option does not get a special namespace. Standard and weekly options
are both `SECURITYTYPE_OPTION`, so both live in `GLBX.OPTION`; the exact
`security_desc` distinguishes the weekly series. For example, if the database
assigned rowids `42001` through `42004` to the rows above, those integers—not
the descriptor strings—would be the Spark security keys.
Conceptually, legacy allocation is:
```text
future: lookup_or_insert("GLBX.FUTURE", "ESZ6")
option: lookup_or_insert("GLBX.OPTION", "ESZ6 C6500")
weekly option: lookup_or_insert("GLBX.OPTION", "ESZ6-W1 C6500")
spot: lookup_or_insert("COINBASE.SPOT", "BTC-USD")
```
#### Newer `IDGenerator`
The namespace result is similar, but the third argument is chosen by the
adapter. Examples using a description-oriented adapter are:
```cpp
get_or_add(VENUE_GLBX, SECURITY_TYPE_FUTURE, "ESZ6");
// store: ns="GLBX.FUTURE", key="ESZ6"
get_or_add(VENUE_GLBX, SECURITY_TYPE_OPTION, "ESZ6 C6500");
// store: ns="GLBX.OPTION", key="ESZ6 C6500"
get_or_add(VENUE_GLBX, SECURITY_TYPE_OPTION, "ESZ6-W1 C6500");
// store: ns="GLBX.OPTION", key="ESZ6-W1 C6500"
get_or_add(VENUE_COINBASE, SECURITY_TYPE_SPOT, "BTC-USD");
// store: ns="COINBASE.SPOT", key="BTC-USD"
```
Databento intentionally produces a different logical row even when the human
description is `ESZ6`. If its instrument ID were `12345678`, it would call the
equivalent of:
```cpp
get_or_add(VENUE_GLBX, SECURITY_TYPE_FUTURE, "12345678");
// store: ns="GLBX.FUTURE", key="12345678"
```
Likewise, the Python bridge uses `exchange_security_id`. A spot definition with
`exchange_security_id="BTC-USD"` produces the `COINBASE.SPOT / BTC-USD` row,
while a provider that sends `exchange_security_id="BTCUSD"` produces a
different row and therefore a different Spark key.
These examples show why the tuple is only logical when the adapter's naming
contract is also known. `GLBX.FUTURE / ESZ6` and `GLBX.FUTURE / 12345678` may
describe the same contract to a human but are distinct rows to `key_ids.sqlite`.
Both enum-prefix removals currently produce the same namespace spelling (for
example, `GLBX.OPTION`), so both mechanisms allocate from one global SQLite
rowid space. Live rows cannot share a rowid. The table does not declare
`AUTOINCREMENT`, however, so deleting rows—especially the highest row—can allow
SQLite to reuse a number later. Security-key rows should therefore be treated
as append-only operational data.
`SqliteStore` uses WAL mode and `PRAGMA synchronous = 0`. This favors startup
and feed latency over maximum durability against a host/power failure. The
unique `(ns, key)` index prevents two different rows for the same logical
security.
### Other places the key is stored
| Location | Purpose | Authoritative for allocation? |
|---|---|---|
| Hot and cold shared-memory segments | Live publication and fast consumer lookup | No; rebuilt at process startup |
| `CMESecurityIndex` / `GeneralSecurityIndex` and adapter maps | In-process routing from an incoming ID to a producer handle | No; in-memory only |
| `<data_path>/databases/md/securities.sqlite` | Serialized legacy `api::Security` definitions keyed by Spark key; used by security cache and MDP3 Sunday preload | No; it stores metadata after a key exists |
| Relay/market-data recordings | Preserve the key embedded in recorded securities, books, and trades for deterministic replay | No; replay normally reuses the recorded value |
| Orders, fills, positions, APIs | Reference a security by the published Spark key | No |
The `SecurityIDRecorder::work_db()` comment says a published security is also
recorded in the security database, but that function does **not** write
`securities.sqlite` in the current code. Full security definitions are persisted
separately by `SecurityCache::flush_to_disk()` through
`db::insert_or_replace()`.
## CME MDP3 / OnixS pipeline in detail
### 1. Construction and thread setup
`MDP3Main::PImpl` owns one `CMESecurityIndex`, one `SecurityIDRecorder`, and one
`MDProducer`. It constructs the recorder with:
- a lookup callback that maps `ExchangeKey` through the CME index to the
handle's current Spark key; and
- `Dispatcher::current()`, which drives a 200 ms recorder timer.
In the current `MarketDataGateway` route, `OnixsGlbxSubscription` creates a
dedicated `dispatcher_thread`, constructs and starts `MDP3Main` on that thread,
then calls that thread's dispatcher `run()`. The older `md run onixs` command
uses its command/dispatcher thread similarly.
The OnixS feed engine has its own configured thread pool. Instrument and market
data callbacks therefore originate on OnixS feed-engine threads, while recorder
database work and final key assignment execute on the MDP3 dispatcher thread.
| MDP3 stage | Executing thread | DB call? |
|---|---|---|
| Decode definition and set CME `numeric_id` | OnixS feed-engine callback thread | No |
| Allocate hot/cold entry and queue handle | Same OnixS callback thread | No |
| Add CME-ID index entry | Same OnixS callback thread | No |
| Resolve spread legs | MDP3 dispatcher thread, on 200 ms timer | No |
| Select/insert Spark key | MDP3 dispatcher thread, on 200 ms timer | Yes |
| `handle.set_key()` and public visibility | MDP3 dispatcher thread, in DB callback | No additional call |
### 2. Definition callback: exchange ID is set, Spark key is not
For futures, options, and spreads, `MDP3Listener::set_security_common()` does:
```cpp
fsec.numeric_id = def.securityId();
```
It also copies symbol/descriptor/venue and pricing fields. A newly constructed
`FixedSecurity` already has `key == invalid_security_key` (`0`), and this helper
does not change it.
For a spread, every leg starts with:
```cpp
leg.exchange_key = ExchangeKey(definition_leg.legSecurityId());
leg.key = invalid_security_key;
```
Thus both the parent and legs initially carry exchange-routing identity, but no
Spark identity.
### 3. Shared-memory allocation and exchange indexing
`MDP3Listener::add_security()` performs these operations in order:
1. `_producer.add_security(security)` allocates a hot entry and a variable-sized
cold entry and copies the security into the cold segment.
2. Because the key is `0`, `MDProducer` queues the handle through
`SecurityIDRecorder::ensure_key()`.
3. `_index.add_security(security.numeric_id, handle)` immediately creates the
CME-ID-to-handle mapping.
4. The initial exchange status is written through the handle.
This ordering means incoming books can already be routed by CME ID while the
Spark key is pending. Shared-memory consumers cannot discover the security yet.
Also, `MDProducerHandle::add_trade()` only appends to the public trade queue
when `key() != 0`, so a trade received in this short window updates the cold
last-trade state but is not added to the keyed trade log.
### 4. Recorder tick: spreads, handles, then database
Every 200 ms the recorder calls, in this exact order:
1. `work_spreads()` tries to fill previously unresolved spread leg keys by
asking `CMESecurityIndex` for each leg exchange ID.
2. `work_handles()` drains new handles. Outrights are converted to an API
security and queued for DB lookup immediately. A spread without all leg keys
is retained in `pending_handle` and its leg pointers are registered in
`unresolved_legs`.
3. `work_db()` resolves queued `(venue, type, descriptor)` requests.
Because spread work precedes new-handle work, a newly seen unresolved spread
normally waits until at least the next timer tick to resolve legs. It waits
longer if any outright leg has not yet received its Spark key. A spread's own
key is not requested until every leg key is non-zero.
### 5. Lookup or generation
`work_db()` first queries `(ns, key)`:
- Existing row: use its `rowid` as the key and invoke the callback.
- Missing row: batch it into `insert_many_resolve_ids()`. SQLite assigns a new
rowid inside a transaction, the row object receives that ID, and the callback
runs after the transaction completes.
New inserts are throttled to roughly 200 per timer pass (the current `> 200`
check can admit 201) to avoid delaying heartbeats during startup. The queues are
TBB concurrent queues, but all SQLite access is serialized onto the recorder's
dispatcher thread. `check_access()` asserts this single-thread rule after the
first work pass records the dispatcher thread ID.
### 6. Final assignment and visibility
The queued callback captures a copy of `MDProducerHandle` and calls
`handle.set_key(id)` on the dispatcher thread. `set_key()` writes hot, then cold,
then increments `security_count`. From this point:
- shared-memory scans report the security;
- `CMESecurityIndex::get_key(CME_ID)` returns the Spark key;
- trades and RFQs can be emitted with a non-zero key;
- resolved spreads can copy this key into their leg structures.
### 7. Sunday preload is a fast path
For configured symbols on Sunday, `MDP3Main::prepare_listener()` loads stored
definitions from `securities.sqlite`. It converts them with
`set_security_key=true`, so both the security key and spread leg keys come from
the serialized API security. `MDProducer::add_security()` calls `set_key()`
immediately, and no recorder lookup is required for that preloaded handle. The
CME exchange ID is still restored into `numeric_id`, allowing the normal CME
index to be built.
## Generalized/newer pipeline in detail
### 1. Topology and ownership
`Run::run()` starts a detached thread named `MarketDataGateway`. On that thread,
`run_md_gwy()` obtains the process-wide `IDGenerator`, constructs
`MarketDataGateway`, and starts the configured providers. For each `(venue,
source)` returned by a provider, the gateway creates an `MdSink` containing:
- a venue-local `GeneralSecurityIndex`;
- a default `MDProducer` (not wired to a `SecurityIDRecorder`); and
- a `GeneralMarketDataListener` connecting the source to that producer/index.
The source decides its callback thread. Some sources call the listener while
starting; others own network or replay threads. The generalized listener calls
the producer inline on that source thread. The process-wide `IDGenerator`
serializes its own calls with a mutex, but the listener's transient
`cc_sec_id_map` is an ordinary `unordered_map`, so each source/listener pair is
expected to serialize security/event callbacks.
| Generalized stage | Executing thread | DB call? |
|---|---|---|
| Build/parse domain definition | Source/adaptor thread | No |
| `IDGenerator::get_or_add()` | Same source/adaptor thread | Yes, synchronously |
| `GeneralMarketDataListener::on_securities()` | Same source callback thread | No in current use-given mode |
| Copy ID to `FixedSecurity::key` and `numeric_id` | Same source callback thread | No |
| `MDProducer::add_security()` / `set_key()` | Same source callback thread | No additional call |
| Add `GeneralSecurityIndex` entry | Same source callback thread | No |
There are two important compatibility exceptions inside the provider framework:
- `OnixsGlbxSubscription::start()` ignores the supplied generalized listener
and starts `MDP3Main`; CME/OnixS therefore still follows the asynchronous
legacy flow documented above.
- `EmumbaDeribitSubscription::start()` also ignores that listener and uses
`EmMDListener`, which allocates via `IDGenerator` and writes through its own
producer/index. Its assignment is still synchronous and pre-populated, so it
resembles the newer ID flow even though it does not pass through
`GeneralMarketDataListener`.
### 2. How a new ID is constructed
For a normal live generalized adapter, ID creation happens **in the adapter,
before** `GeneralMarketDataListener::on_securities()`:
```text
logical identity = (domain venue, domain security type, adapter-chosen name)
Spark ID = SQLite rowid for that identity in key_ids.sqlite
```
`IDGenerator::get_or_add(venue, type, name)` formats a namespace by stripping
the enum prefixes:
```text
VENUE_TAIFEX + SECURITY_TYPE_FUTURE -> TAIFEX.FUTURE
```
It then calls `retrieve_one(ns, name)`. If the row exists, it returns that
row's `rowid`. If absent, it inserts `(ns, name, "")`, and SQLite assigns the
next rowid. That integer is wrapped in `domain::SecurityID` and placed in
`SecurityCommon::id`.
The third identity component is adapter-specific, and this matters when trying
to reproduce a mapping:
| Source/adaptor family | Typical `name` sent to `IDGenerator` |
|---|---|
| Binance, Coinbase, Deribit, OKX and similar native/CCAPI paths | Usually `sec.desc()` / exchange instrument name |
| Taifex | product/instrument ID string |
| Databento | `std::to_string(msg.hd.instrument_id)`, not the human-readable description |
| Python bridge | `exchange_security_id` (and it asserts Python supplied ID is zero for definitions) |
| Emumba Deribit | extracted Emumba security descriptor |
| Replay | no new allocation for a recorded definition; it reuses the recorded `SecurityCommon::id` |
So “security name” alone is not a universal logical key. Reproduction requires
the exact venue enum, exact security-type enum, and the exact adapter-specific
name used at insertion time.
### 3. What `GeneralMarketDataListener` does with that ID
`on_securities()` first calls `_find_sec(sec, true)`. Its current, compile-time
behavior is controlled by:
```cpp
const bool use_given_sec_key = true;
```
Consequently, for a newly seen incoming `domain::SecurityID` it does **not**
call the DB. It copies `security_common.id.value()` into its transient
`cc_sec_id_map`. `_add_future()`, `_add_option()`, `_add_spread()`, or
`_add_spot()` then calls `_copy_security_common()`, which sets:
```cpp
to.key = incoming_id;
to.numeric_id = incoming_id;
```
This collapses the generalized source/correlation ID and Spark shared-memory
key into the same number. The source's text `exchange_sec_id` remains on the
domain security but is not copied into `FixedSecurity::numeric_id` by this
path.
`_commit_security()` then:
1. calls `MDProducer::add_security()` with an already non-zero key;
2. causes `MDProducerHandle::set_key()` to run immediately during that call;
3. inserts the handle in `GeneralSecurityIndex` under `numeric_id` (the same
number); and
4. returns with the security already visible in shared memory.
Because the sink's producer has no recorder, supplying `id == 0` would allocate
the shared-memory entry but never assign or publish its key. Generalized live
sources must therefore provide a valid non-zero ID before `on_securities()`.
The listener contains an alternative branch for `use_given_sec_key == false`.
That branch would call `IDGenerator::get_or_add(me->venue, sec.type(),
sec.desc())` itself. It is currently disabled, and `_generate_security_key()` is
also effectively unused. Turning the flag off is not a mechanical change: an
adapter such as Databento allocates using numeric instrument ID while the
listener branch would use `sec.desc()`, potentially creating a second key for
the same instrument.
There is also a spread failure edge case. `_find_sec(sec, true)` records the
parent ID in `cc_sec_id_map` before `_add_spread()` resolves its legs. If any leg
ID is absent, `_add_spread()` logs and returns without committing the spread. A
later repeat of the parent definition is then classified as a duplicate and
ignored, so definition ordering is a hard requirement in the current code.
### 4. How later books, trades, and statistics find the security
Every later domain event carries a `domain::SecurityID`:
- `Book::security_id()`;
- `TradeRecord::sec_id`; or
- `SecurityMarketDataStatistics::security_id`.
The listener first uses `cc_sec_id_map` to translate that event ID to the value
stored in `GeneralSecurityIndex`, then obtains the `MDProducerHandle`. With
`use_given_sec_key == true`, this mapping is numerically identity-to-identity,
but it still serves as the admission check that the definition was seen.
Most high-throughput adapters avoid SQLite on market-data events:
- Databento allocates on the definition and caches
`instrument_id -> domain::SecurityID` in `sec_id_map` for books/trades.
- Taifex keeps the resolved ID in its security-details mapper.
- Other native subscriptions maintain comparable descriptor/ID maps.
The Python bridge is a notable exception: `to_bts_security_()`,
`to_bts_trade_()`, and `to_book_()` all call `get_sec_id_()`, which calls
`IDGenerator::get_id(..., add_if_new=true)`. As written, that performs at least
one SQLite `SELECT` for each converted Python definition, trade, or book. It
returns the existing ID after the first insert, but it is still a DB call and
takes the global `IDGenerator` mutex on every event.
### 5. Persistence timing and latency
The newer allocator is synchronous:
- Existing security: mutex acquisition plus one SQLite `SELECT` on the caller's
thread.
- New security: mutex acquisition, one `SELECT`, then one SQLite `INSERT`
(SQLite autocommit), also on the caller's thread.
- Listener publication: no further DB call under the current
`use_given_sec_key` setting.
The ID therefore exists in `key_ids.sqlite` before the adapter sends the
definition to the listener. There is no intentional 200 ms publication delay,
but the source callback/startup path directly pays DB and mutex latency. Bulk
definition discovery performs one lookup and possibly one insert per security;
`IDGenerator` does not batch those operations.
WAL mode helps concurrency, while `synchronous=0` reduces fsync cost. It does
not eliminate SQL parsing/stepping, mutex contention, page-cache misses, or
occasional file-system work. The impact is usually a startup/definition cost
when adapters cache the result, but it becomes hot-path latency for code such as
the current Python bridge that resolves every event through the DB.
`GeneralMarketDataListener` itself does not persist the full contract definition
to `securities.sqlite`; that remains the separate security-cache/metadata
persistence path described earlier.
## Comparison: MDP3 recorder vs generalized allocator
| Property | MDP3 / `SecurityIDRecorder` | Generalized / `IDGenerator` |
|---|---|---|
| Logical input | venue + type + legacy `security_desc` | venue + type + adapter-chosen name |
| Numeric output | SQLite rowid, checked/read as 32-bit `security_key` | SQLite rowid returned as 64-bit `domain::SecurityID`, then narrowed into 32-bit shared-memory key |
| Allocation point | After definition is copied to shared memory | Normally inside adapter before listener sees definition |
| DB execution | Asynchronous on dispatcher timer | Synchronous on caller/source thread |
| Existing ID cost | One deferred `SELECT` | One immediate `SELECT` per call |
| New ID cost | Deferred, batched transaction (about 200/pass) | Immediate `SELECT` + individual autocommit `INSERT` |
| Time until public | Usually 0–200+ ms; spreads may take extra ticks | Immediate after synchronous lookup/insert completes |
| Feed callback blocking | Queue push only for ID work | Yes, if the adapter invokes `IDGenerator` there |
| Spread legs | Parent waits asynchronously until all leg keys resolve by exchange ID | Definition processing requires every leg's incoming ID already exist in `cc_sec_id_map`; otherwise the spread is aborted |
| Runtime ID relationship | Exchange ID and Spark key remain distinct | Incoming/canonical ID, `numeric_id`, index ID, and Spark key are currently the same number |
| Persistence database | `key_ids.sqlite` | The same `key_ids.sqlite` |
| Batching | Yes for new rows | No |
| Failure behavior | Security has an internal handle with key 0 while normal queued work is pending; DB work does not block the feed callback | DB errors/zero results occur in the synchronous source path; a zero ID can leave an unpublishable entry, with no pending-key state machine |
Neither algorithm generates a UUID or a content hash. Both ultimately use an
opaque, database-assigned SQLite rowid. It is normally increasing and looks like
a sequence, but this schema does not request SQLite `AUTOINCREMENT`. The tuple
used to *look up* the row is logical; the resulting number is not derivable from
that tuple.
## Can another system reconstruct the same keys?
Not from `(venue, type, name)` alone. Another system can reconstruct the
**logical lookup tuple**, but it cannot calculate the numeric Spark key because
the key depends on the historical insertion order of every row in
`key_ids.sqlite`.
For example, two empty systems can insert the exact same `GLBX.FUTURE / ESZ6`
row and receive different IDs if other securities were inserted before it on
one system. They are sequential opaque IDs, not deterministic hashes and not
UUIDs.
Ways to preserve cross-system identity are:
1. Copy/replicate `key_ids.sqlite` (including rowids) and keep descriptor rules
identical. This preserves existing numeric keys but requires coordinated
allocation for new ones.
2. Export an explicit mapping of `(namespace, adapter key) -> Spark key`, then
import rows with their original rowids before either system allocates new
securities.
3. Treat a stable external identity (venue + type + exchange instrument ID, or
another canonical instrument identifier) as the cross-system key and carry a
translation table to each system's local Spark key.
4. Introduce a deterministic ID scheme only as a versioned migration. A hash can
be computed independently, but collision handling, descriptor
canonicalization, aliases/renames, the existing 32-bit ABI, and compatibility
with orders/recordings/databases all need explicit design.
Copying only `securities.sqlite` is insufficient as an allocator strategy. It
contains security blobs keyed by already-assigned IDs, but `key_ids.sqlite` is
the mapping used to find or allocate those IDs. Conversely, copying only
`key_ids.sqlite` preserves the numbers but not all contract metadata.
## Other assignment paths in `sparkd`
The two main pipelines reduce to four ways a producer handle can acquire a key:
1. **Already keyed input.** `MDProducer::add_security()` sees a non-zero
`FixedSecurity::key` and calls `set_key()` immediately. Generalized feeds,
Sunday preload, relay playback, and other replay paths use this mechanism.
2. **Recorder-backed live input.** The input key is zero and the producer owns a
`SecurityIDRecorder`; `ensure_key()` queues asynchronous assignment. CME
MDP3 and legacy OnixS ICE use this mechanism. ICE uses `marketId` as
`numeric_id` and otherwise follows the same recorder/index design.
3. **Service assignment.** `SyntheticSecurityService` calls
`SecurityIDRecorder::assign_key()` directly on a background dispatcher. The
callback receives a copied `api::Security` with its top-level `id` set, then
publishes it to synthetic subscribers. It uses the same `key_ids.sqlite`
rowid allocator.
4. **Unkeyed input with no recorder.** The entry is allocated with key zero and
never becomes public. This is not a valid production assignment path, but it
is the failure mode if a generalized source omits its ID.
Relay and replay deserve care when reasoning about persistence:
- The generalized `MDReplayer` forwards the recorded domain security ID and
emits later books/statistics with the recorded `sec_key`, so the listener's
current use-given mode preserves the recording's numeric identity.
- The legacy relay producer converts a recorded `api::Security` with
`set_security_key=true`, so its shared-memory publication also preserves the
embedded key.
- The legacy relay recorder additionally calls `bulk_insert(..., "REPLAY: ")`
for logical mappings. Those rows use a `REPLAY: ` namespace prefix; this is a
separate mapping population step, not a deterministic derivation of the key.
## Operational implications
- Back up `key_ids.sqlite` and `securities.sqlite` together if numeric ID and
definition continuity matter.
- Do not delete/rebuild `key_ids.sqlite` on one host and assume IDs will match
another host, even when all instruments are identical.
- Before moving a feed between adapters, compare the exact `name` each adapter
supplies to `IDGenerator`. Matching venue/type is insufficient.
- Cache generated IDs in the source after definition processing. Avoid calling
`IDGenerator` from book/trade hot paths; the Python bridge currently violates
this guideline.
- Keep generated rowids below the 32-bit shared-memory ceiling until the
`security_key` ABI is widened or explicit bounds checking is added to the
generalized path.
- Ensure spread legs are announced before generalized spread parents.
- Treat `synchronous=0` as a durability decision. A copied DB should be taken
with SQLite-aware backup/checkpoint handling rather than by casually copying
active WAL files.
Useful inspection queries (run against the configured DB path) are:
```sql
SELECT rowid AS spark_key, ns, key
FROM store
ORDER BY rowid;
SELECT rowid AS spark_key, ns, key
FROM store
WHERE ns = 'GLBX.FUTURE' AND key = '<exact-adapter-key>';
```
## Source map
- [`SecurityIDRecorder.cpp`](../src/lib_spark/spark/md/SecurityIDRecorder.cpp):
legacy queues, 200 ms work ordering, logical row construction, DB callbacks,
and spread-leg resolution.
- [`IDGenerator.cpp`](../src/lib_spark/spark/persistence/IDGenerator.cpp):
synchronous generalized lookup/allocation and process-wide mutex.
- [`SqliteStore.cpp`](../src/lib_spark/spark/persistence/SqliteStore.cpp): DB
location, schema, WAL/synchronous pragmas, individual inserts, and batch
inserts.
- [`MDProducer.cpp`](../src/lib_spark/spark/md/MDProducer.cpp) and
[`MDProducerHandle.cpp`](../src/lib_spark/spark/md/MDProducerHandle.cpp):
shared-memory allocation, recorder handoff, and ordered publication.
- [`FixedSecurity.h`](../src/lib_spark/spark/md/Shm/FixedSecurity.h) and
[`MDStructure.h`](../src/lib_spark/spark/md/Shm/MDStructure.h): identifier
widths and hot/cold storage fields.
- [`MDP3Main.cpp`](../src/app_sparkd/sparkd/md/platforms/mdp3_onixs/MDP3Main.cpp)
and
[`MDP3Listener.h`](../src/app_sparkd/sparkd/md/platforms/mdp3_onixs/MDP3Listener.h):
MDP3 construction, definition callbacks, CME ID extraction, and spread legs.
- [`MDP3Listener.cpp`](../src/app_sparkd/sparkd/md/platforms/mdp3_onixs/MDP3Listener.cpp):
producer/index insertion order.
- [`OnixsGlbxSubscription.cpp`](../src/app_sparkd/sparkd/md/glbx/OnixsGlbxSubscription.cpp):
current dedicated MDP3 dispatcher-thread ownership.
- [`MarketDataGateway.cpp`](../src/app_sparkd/sparkd/md/MarketDataGateway.cpp)
and
[`GeneralMarketDataListener.cpp`](../src/lib_spark/spark/md/GeneralMarketDataListener.cpp):
generalized source/sink topology, incoming-ID mapping, fixed-security
conversion, and publication.
- [`DatabentoDataHandler.cpp`](../src/app_sparkd/sparkd/md/databento/DatabentoDataHandler.cpp),
[`TaifexDataProcessors.h`](../src/app_sparkd/sparkd/md/taifex/TaifexDataProcessors.h),
and
[`PyMarketDataSubscription.cpp`](../src/app_sparkd/sparkd/md/PyMarketDataSubscription.cpp):
representative adapter-side allocation and caching behavior.
- [`SecurityDB.cpp`](../src/lib_spark/spark/md/SecurityDB.cpp) and
[`SecurityCache.cpp`](../src/lib_spark/spark/md/SecurityCache.cpp): separate
persistence of full security definitions.
- [`SecurityIndex.h`](../src/lib_spark/spark/md/SecurityIndex.h),
[`MBOServiceServer.cpp`](../src/lib_spark/spark/md/MBOServiceServer.cpp), and
[`MDConsumer.cpp`](../src/lib_spark/spark/md/MDConsumer.cpp): exchange-index
translation followed by Spark-keyed market-data consumption and MBO access.
- [`OrderFactory.cpp`](../src/lib_spark/spark/orderman/order_factory/OrderFactory.cpp),
[`FillStore.cpp`](../src/lib_spark/spark/orderman/actors/FillStore.cpp), and
[`PositionMap.cpp`](../src/app_sparkd/sparkd/position/PositionMap.cpp):
representative downstream use of Spark security keys.
- [`MDReplayer.cpp`](../src/app_sparkd/sparkd/md/recording/MDReplayer.cpp) and
[`RelayClient.cpp`](../src/lib_spark/spark/md/relay/RelayClient.cpp): recorded
ID reuse.