48 KiB
Security ID handling in sparkd
This document traces how security identifiers are created, translated, persisted,
indexed, and published by the sparkd market-data paths. It focuses on the
legacy CME MDP3/OnixS pipeline and the newer generalized market-data pipeline.
Executive summary
sparkd uses two identifiers that must not be confused:
| Identifier | Meaning | Typical field | Lifetime |
|---|---|---|---|
| Exchange/feed ID | The venue's identifier used to route incoming definitions, books, and trades | FixedSecurity::numeric_id, ExchangeKey, or domain::Security::security_common.exchange_sec_id |
Feed/session-specific unless the source guarantees stability |
| Spark security key | Spark's internal non-zero identifier used in shared memory, APIs, orders, positions, and downstream services | FixedSecurity::key, api::Security::id(), SecurityKey; historically also called security_id |
Intended to be stable across restarts |
invalid_security_key is 0; a security with key 0 is not yet publicly
discoverable through the shared-memory consumer path.
The two principal paths differ:
- MDP3/OnixS: the definition callback sets only the CME ID in
numeric_id.MDProducer::add_security()allocates the shared-memory entry with key0.SecurityIDRecorder, driven by a 200 ms dispatcher timer, resolves or inserts a stable key inkey_ids.sqlite; its callback finally callsMDProducerHandle::set_key(). - Generalized/newer pipeline:
GeneralMarketDataListener::on_securities()currently uses the ID already present on the incoming domain security (use_given_sec_key == true). Live adapters normally obtain that ID synchronously fromIDGeneratorfirst._copy_security_common()then copies it to bothFixedSecurity::keyandnumeric_id, soMDProducer::add_security()publishes it synchronously. Only the listener's own fallbackIDGeneratorcall is bypassed by the current constant.
End-to-end diagrams
CME MDP3 / OnixS
sequenceDiagram
participant FE as OnixS feed-engine thread
participant L as MDP3Listener
participant P as MDProducer / shared memory
participant Q as SecurityIDRecorder queues
participant D as sparkd dispatcher thread
participant DB as key_ids.sqlite
FE->>L: instrument definition (CME SecurityID)
L->>L: set_security_common() - numeric_id = CME SecurityID, key remains 0
L->>P: add_security(FixedSecurity)
P->>P: allocate hot + cold entries with key 0
P->>Q: ensure_key(handle)
L->>L: index[CME SecurityID] = handle
Note over FE,L: Books can route by CME ID while the Spark key is pending
D->>Q: every 200 ms - work_spreads(), work_handles(), work_db()
Q->>DB: find (namespace, descriptor), or insert a new row
DB-->>Q: SQLite rowid = Spark security key
Q->>P: handle.set_key(rowid)
P->>P: hot.key first, then cold security.key, then increment security_count
Note over P: The security is now public to shared-memory consumers
Generalized/newer pipeline (current behavior)
sequenceDiagram
participant A as Venue adapter / plugin
participant DB as IDGenerator / key_ids.sqlite
participant G as GeneralMarketDataListener
participant I as GeneralSecurityIndex
participant P as MDProducer / shared memory
A->>DB: get_or_add(venue, type, adapter key)
DB-->>A: SQLite rowid as domain SecurityID
A->>G: on_securities(domain Security list with ID)
G->>G: _find_sec() - use incoming security_common.id
G->>G: _copy_security_common() - key = id, numeric_id = id
G->>P: add_security(FixedSecurity with non-zero key)
P->>P: set_key() during add_security()
G->>I: index[id] = handle
Note over G,P: No asynchronous ID assignment in the current configuration
Data-flow charts
The sequence diagrams above emphasize timing and thread transitions. These flowcharts emphasize which identifier is carried through each representation and where data is retained.
MDP3 / legacy data flow
flowchart TD
CME["CME MDP3 packets<br/>CME SecurityID"] --> ONIXS["OnixS feed engine<br/>decode definition and updates"]
ONIXS --> DEF["MDP3Listener definition callback"]
DEF --> FIXED["FixedSecurity<br/>numeric_id = CME SecurityID<br/>key = 0"]
FIXED --> PRODUCER["MDProducer::add_security"]
PRODUCER --> PENDING["Hot and cold shared memory<br/>allocated but not public<br/>key = 0"]
PRODUCER --> QUEUE["SecurityIDRecorder<br/>pending_handle queue"]
FIXED --> CMEINDEX["CMESecurityIndex<br/>CME SecurityID to handle"]
QUEUE --> TIMER["Dispatcher timer<br/>every 200 ms"]
TIMER --> LEGS{"Spread legs<br/>all resolved?"}
LEGS -- No --> UNRESOLVED["unresolved_legs<br/>exchange IDs awaiting keys"]
UNRESOLVED --> CMEINDEX
CMEINDEX --> LEGS
LEGS -- Yes or outright --> LOGICAL["Logical identity<br/>venue + type + security_desc"]
LOGICAL --> SELECT["SQLite SELECT<br/>key_ids.sqlite"]
SELECT -- Existing row --> ROWID["SQLite rowid<br/>Spark security key"]
SELECT -- Missing row --> INSERT["Batched SQLite INSERT"]
INSERT --> ROWID
ROWID --> SETKEY["MDProducerHandle::set_key"]
SETKEY --> HOT["Write hot-entry key"]
HOT --> COLD["Write cold security key"]
COLD --> COUNT["Increment security_count"]
COUNT --> PUBLIC["Public security<br/>visible to MDConsumer"]
ONIXS --> UPDATE["Book, trade, status, or RFQ<br/>carrying CME SecurityID"]
UPDATE --> CMEINDEX
CMEINDEX --> HANDLE["MDProducerHandle"]
HANDLE --> LIVE["Update live shared-memory data"]
PUBLIC --> LIVE
The CME index is usable before the Spark key is assigned, so feed updates can
reach the internal handle while the security is still hidden from consumers.
The persistent rowid crosses the publication boundary only at set_key().
Generalized/newer data flow
flowchart TD
FEED["Venue feed, Databento,<br/>Python plugin, or replay"] --> ADAPTER["Source adapter<br/>build domain::Security"]
ADAPTER --> IDTYPE{"Live definition or<br/>recorded/pre-keyed input?"}
IDTYPE -- Live --> IDINPUT["Logical identity<br/>venue + type + adapter-chosen name"]
IDINPUT --> MUTEX["IDGenerator global mutex"]
MUTEX --> SELECT["Synchronous SQLite SELECT<br/>key_ids.sqlite"]
SELECT -- Existing row --> ROWID["SQLite rowid"]
SELECT -- Missing row --> INSERT["Synchronous SQLite INSERT<br/>autocommit"]
INSERT --> ROWID
ROWID --> DOMAINID["domain::SecurityID<br/>used as Spark ID"]
IDTYPE -- Recorded or pre-keyed --> DOMAINID
DOMAINID --> DOMAINSEC["SecurityCommon::id = Spark ID"]
DOMAINSEC --> LISTENER["GeneralMarketDataListener::on_securities"]
LISTENER --> CCMAP["cc_sec_id_map<br/>incoming ID to Spark ID"]
LISTENER --> CONVERT["_copy_security_common<br/>FixedSecurity.key = ID<br/>numeric_id = ID"]
CONVERT --> PRODUCER["MDProducer::add_security"]
PRODUCER --> SETKEY["Immediate set_key<br/>no recorder timer"]
SETKEY --> SHM["Hot and cold shared memory<br/>immediately public"]
CONVERT --> GENERALINDEX["GeneralSecurityIndex<br/>ID to handle"]
FEED --> EVENT["Book, trade, or statistics<br/>with exchange/source identity"]
EVENT --> EVENTMAP["Adapter map<br/>source identity to domain::SecurityID"]
EVENTMAP --> EVENTID["Event carries domain::SecurityID"]
EVENT --> PYHOT["Python bridge only<br/>synchronous IDGenerator lookup"]
PYHOT --> EVENTID
PYHOT -. uses same mutex and SQLite path .-> MUTEX
EVENTID --> CCMAP
CCMAP --> GENERALINDEX
GENERALINDEX --> HANDLE["MDProducerHandle"]
HANDLE --> SHMUPDATE["Update book, trade,<br/>statistics, or status"]
SHM --> SHMUPDATE
In the normal newer live path, persistence occurs before the definition enters the generalized listener. The DB call therefore adds synchronous adapter-side latency, but publication needs no later timer. Most adapters use an in-memory map for subsequent events; the dotted Python path is the documented exception.
The publication boundary
MDProducerHandle::set_key() is the definitive publication point for the
legacy shared-memory representation. Its write ordering is intentional:
- Write the key into the hot book entry (
HotEntryorHotDepthEntry). - Write the key into the cold security entry.
- Increment the cold header's public
security_count.
The cold security key is written last because consumers treat a non-zero cold
key as visibility. At that moment the corresponding hot entry must already be
addressable. set_key() only permits the transition from 0 to one key; it is
not a key-update mechanism.
Identifier representations
The names are historically inconsistent. In particular, many protobuf fields
named security_id contain the Spark key, not the exchange's ID.
| Layer | Exchange/feed identifier | Spark identifier |
|---|---|---|
| MDP3 wire/OnixS | securityId() |
not present |
| Legacy fixed security | FixedSecurity::numeric_id (uint64_t) |
FixedSecurity::key / security_key (uint32_t) |
| Legacy API security | venue-specific exchange_id |
top-level Security.id() |
| New domain model | SecurityCommon::exchange_sec_id (string) is descriptive; SecurityCommon::id is also used to correlate source events |
domain::SecurityID (uint64_t), currently copied into the Spark key |
| Shared-memory hot book | none | HotEntry::key or HotDepthEntry::key |
| Shared-memory cold metadata | FixedSecurity::numeric_id |
FixedSecurity::key |
| CME runtime lookup | CMESecurityIndex key |
SecurityData::handle.key() |
| General runtime lookup | incoming/canonical domain::SecurityID |
GeneralSecurityIndex key and handle.key() (currently the same value) |
The generalized model declares domain::SecurityID as 64-bit, but the legacy
shared-memory security_key remains 32-bit. _copy_security_common() assigns a
64-bit value to both numeric_id and the 32-bit key. This is safe only while
allocated IDs fit in 32 bits; unlike the legacy lookup path, this assignment has
no explicit checked conversion.
Which parts use exchange IDs, descriptors, or Spark security IDs?
The three identifier classes serve different jobs:
- Exchange/external ID: routes data inside a venue or vendor feed adapter.
- Descriptor/natural key: recognizes a definition logically at the persistence, discovery, or human-facing boundary.
- Spark security ID/key: joins the instrument to books, orders, fills, positions, and services after publication.
The core downstream system does not repeatedly identify a security from its description. The description is primarily used to obtain a Spark key; after that boundary, most joins use the numeric Spark key.
Component-level usage matrix
| Component or data path | Exchange/external ID | Descriptor or name | Spark security ID/key |
|---|---|---|---|
| CME MDP3 definition decoder | Reads securityId() and stores it in FixedSecurity::numeric_id |
Copies definition symbol/tag 55 into security_desc |
Leaves FixedSecurity::key at 0 initially |
CMESecurityIndex |
Primary map key; CME ID resolves to a handle | Not used for update routing | Returns handle.key() when the Spark key has been assigned |
| MDP3 books, trades, status, statistics, and RFQs | Incoming CME ID selects the handle through CMESecurityIndex |
Not used to locate each update | The selected handle writes keyed shared-memory/trade data after publication |
| MDP3 spread definitions | Each leg initially contains legSecurityId() as exchange_key |
Parent security_desc identifies the spread's DB row |
Each leg's Spark key and then the parent Spark key must be resolved before publication |
| Legacy ICE OnixS | Uses ICE marketId as numeric_id and index key |
ICE security description becomes the logical lookup name | Uses the same recorder-assigned Spark key pattern as MDP3 |
SecurityIDRecorder |
Does not allocate from the external ID | Uses (venue, security type, securities::descriptor()) to find/create a row |
Treats the SQLite rowid as the resulting Spark key |
| Native generalized adapters | Use exchange strings/numeric IDs to parse definitions and correlate updates | Usually pass a description or exchange instrument name to IDGenerator |
Put the returned ID into SecurityCommon::id and later events |
| Databento adapter | Uses msg.hd.instrument_id for feed correlation and its in-memory map |
Uses the decimal instrument ID string—not the human description—as the IDGenerator name |
Caches instrument_id -> domain::SecurityID and places that ID on books/trades |
| Python bridge | Reads exchange_security_id from definitions, books, and trades |
Uses that external-ID string as the IDGenerator name |
Resolves it to domain::SecurityID; currently performs this lookup for every event |
IDGenerator |
Does not interpret whether name came from an exchange ID |
Uses exactly (venue, type, caller-supplied name) |
Returns the SQLite rowid as a 64-bit ID |
GeneralMarketDataListener |
Does not route on exchange_sec_id directly |
Copies sec.desc() into fixed metadata; its disabled fallback allocator would use sec.desc() |
Uses incoming SecurityCommon::id, maps events through cc_sec_id_map, and copies the value into both key and numeric_id |
GeneralSecurityIndex |
No separate external-ID concept in the current use-given path | Not used | Maps the generalized ID/Spark key to MDProducerHandle |
MDProducer / shared memory |
Retains numeric_id; in legacy feeds this is the exchange ID, but in the generalized path it is currently the Spark ID |
Stores security_desc in the cold definition |
Uses the key in hot/cold entries; non-zero key is the visibility boundary |
MDConsumer |
Not normally used to find live books | Can enumerate/filter converted security definitions | Finds handles and book data by security_key |
SecurityCache / securities.sqlite |
Preserves venue-specific exchange data inside the serialized security blob | Maintains descriptor/name lookup indexes and supports venue/type/name queries | Primary persisted definition key is api::Security.id() |
| MBO processor and MBO service | May retain exchange order IDs, which are separate from security exchange IDs | Not used to find the instrument | Indexes snapshots and requests by SecurityKey; MBO protobuf security_id means Spark key |
| Order manager and algorithms | Execution gateways eventually use venue-specific fields from the resolved security to encode orders | Descriptions appear in logs and errors, not as the primary join | Order definitions, subscriptions, book events, native orders, spread legs, and fills use security_key / Security.id() |
| Position, PNL, and risk services | Not used as their position join | May resolve metadata for display/reporting | Positions, fills, overrides, and PNL handles are keyed by Spark security ID |
| Relay and market-data recordings | Serialized security definitions may preserve venue exchange IDs | Preserve full descriptors in recorded definitions | Books, updates, trades, and replay correlation preserve the recorded Spark key |
| Synthetic security service | No venue exchange ID; synthetics use VENUE_BTS |
Synthetic name is the recorder's logical descriptor | Recorder assigns the SQLite rowid and synthetic metadata is keyed by it |
Exchange/external IDs: where they stop
In MDP3 the separation is explicit:
CME SecurityID
-> FixedSecurity.numeric_id
-> CMESecurityIndex lookup
-> MDProducerHandle
-> Spark security_key for downstream publication
The external CME ID continues to route feed messages but does not become the
order/position key. This permits the same downstream Spark key to survive a
restart as long as the logical descriptor mapping remains in
key_ids.sqlite, even if the feed must rediscover the CME routing ID.
The generalized path collapses more of these roles:
exchange/vendor identity
-> adapter calls IDGenerator
-> domain::SecurityID
-> FixedSecurity.key and FixedSecurity.numeric_id
-> GeneralSecurityIndex and downstream Spark key
SecurityCommon::exchange_sec_id may still contain the original textual
external identifier, but GeneralMarketDataListener does not use it to route
events and does not put it in numeric_id. The adapter must translate later
events to domain::SecurityID before invoking the listener.
Descriptor/name users
Descriptors are used in three main places:
- Key allocation:
SecurityIDRecorderuses the legacy venue-specific descriptor, whileIDGeneratoruses its caller-providedname. - Definition lookup and discovery:
SecurityCachesupports venue/type/name lookup; symbol filters and administrative tools use security metadata to select or display instruments. - Diagnostics: logs include symbols and descriptions so humans can understand a numeric key.
Descriptions generally do not drive the high-rate book, trade, order, fill, or position paths. Those paths use numeric IDs after definition resolution.
It is important that “descriptor” does not mean one universal canonical field:
- Legacy CME uses Globex
security_descfrom tag 55. - Many generalized adapters use
sec.desc()or an exchange instrument name. - Databento uses its numeric instrument ID converted to a string.
- Python uses
exchange_security_id.
Therefore, two adapters may describe the same economic contract but request different logical rows and receive different Spark IDs.
Spark security-ID users
Once a security is published, its Spark key is the dominant system identifier:
flowchart LR
KEY["Spark security key"] --> SHM["Hot/cold shared memory"]
KEY --> MDC["MDConsumer and book subscriptions"]
KEY --> MBO["MBO snapshots and queries"]
KEY --> OMS["Orders, algorithms, and spread legs"]
KEY --> FILLS["Fills and execution records"]
KEY --> POS["Positions, PNL, and risk"]
KEY --> API["Services and client APIs"]
KEY --> REC["Relay and market-data recordings"]
KEY --> SECDB["Serialized security definitions"]
Fields named security_id, security_key, or top-level Security.id() in
these downstream areas almost always mean the Spark key. They should not be
compared directly with CME SecurityID, Databento instrument_id, or another
venue/vendor identifier without going through the relevant adapter/index
mapping.
Persistence and allocation
The authoritative key mapping: key_ids.sqlite
Both allocators use SqliteStore("key_ids"). With the normal configuration the
file is:
<SystemConfig.data_path>/databases/key_ids/key_ids.sqlite
The database contains one table and two indexes:
CREATE TABLE store (ns TEXT, key TEXT, value TEXT);
CREATE UNIQUE INDEX key_index ON store(ns, key);
CREATE INDEX ns_index ON store(ns);
The SQLite rowid is the generated Spark key. ns is normally
<VENUE>.<SECURITY_TYPE> (for example GLBX.FUTURE), and key is the security
descriptor chosen by the caller. value is normally empty. Therefore ID
stability depends on all three of these remaining stable:
- the configured
data_pathand preservation of this database; - the venue/type namespace;
- the descriptor supplied by the feed adapter.
The two allocator implementations share the file and convention:
SecurityIDRecorder(legacy): derives the namespace from legacy API enums and usessecurities::descriptor(). For CME this is the Globexsecurity_desc, populated from the definition's symbol/tag 55.IDGenerator(newer): derives the namespace from new domain enums and uses the caller'sname. Most adapters passsec.desc()or an exchange instrument identifier; Databento deliberately passes the decimal instrument ID, and the Python bridge uses the exchange security ID.
Logical-row examples
The following names are illustrative. Production uses the exact string emitted by the selected feed/adapter, including its case, spacing, and weekly-series notation.
Legacy SecurityIDRecorder
For CME, set_security_common() copies the MDP3 definition's symbol/tag 55 into
Globex security_desc. After converting the fixed security to
api::Security, securities::descriptor() returns that same
glbx.security_desc() string.
| Instrument | Legacy enums | Example descriptor from security_desc |
Row searched/inserted in store |
|---|---|---|---|
| Future | VENUE_GLBX, SECURITYTYPE_FUTURE |
ESZ6 |
ns='GLBX.FUTURE', key='ESZ6' |
| Standard option | VENUE_GLBX, SECURITYTYPE_OPTION |
ESZ6 C6500 |
ns='GLBX.OPTION', key='ESZ6 C6500' |
| Weekly option | VENUE_GLBX, SECURITYTYPE_OPTION |
ESZ6-W1 C6500 |
ns='GLBX.OPTION', key='ESZ6-W1 C6500' |
| Spot from a legacy-compatible venue path | VENUE_COINBASE, SECURITYTYPE_SPOT |
BTC-USD |
ns='COINBASE.SPOT', key='BTC-USD' |
A weekly option does not get a special namespace. Standard and weekly options
are both SECURITYTYPE_OPTION, so both live in GLBX.OPTION; the exact
security_desc distinguishes the weekly series. For example, if the database
assigned rowids 42001 through 42004 to the rows above, those integers—not
the descriptor strings—would be the Spark security keys.
Conceptually, legacy allocation is:
future: lookup_or_insert("GLBX.FUTURE", "ESZ6")
option: lookup_or_insert("GLBX.OPTION", "ESZ6 C6500")
weekly option: lookup_or_insert("GLBX.OPTION", "ESZ6-W1 C6500")
spot: lookup_or_insert("COINBASE.SPOT", "BTC-USD")
Newer IDGenerator
The namespace result is similar, but the third argument is chosen by the adapter. Examples using a description-oriented adapter are:
get_or_add(VENUE_GLBX, SECURITY_TYPE_FUTURE, "ESZ6");
// store: ns="GLBX.FUTURE", key="ESZ6"
get_or_add(VENUE_GLBX, SECURITY_TYPE_OPTION, "ESZ6 C6500");
// store: ns="GLBX.OPTION", key="ESZ6 C6500"
get_or_add(VENUE_GLBX, SECURITY_TYPE_OPTION, "ESZ6-W1 C6500");
// store: ns="GLBX.OPTION", key="ESZ6-W1 C6500"
get_or_add(VENUE_COINBASE, SECURITY_TYPE_SPOT, "BTC-USD");
// store: ns="COINBASE.SPOT", key="BTC-USD"
Databento intentionally produces a different logical row even when the human
description is ESZ6. If its instrument ID were 12345678, it would call the
equivalent of:
get_or_add(VENUE_GLBX, SECURITY_TYPE_FUTURE, "12345678");
// store: ns="GLBX.FUTURE", key="12345678"
Likewise, the Python bridge uses exchange_security_id. A spot definition with
exchange_security_id="BTC-USD" produces the COINBASE.SPOT / BTC-USD row,
while a provider that sends exchange_security_id="BTCUSD" produces a
different row and therefore a different Spark key.
These examples show why the tuple is only logical when the adapter's naming
contract is also known. GLBX.FUTURE / ESZ6 and GLBX.FUTURE / 12345678 may
describe the same contract to a human but are distinct rows to key_ids.sqlite.
Both enum-prefix removals currently produce the same namespace spelling (for
example, GLBX.OPTION), so both mechanisms allocate from one global SQLite
rowid space. Live rows cannot share a rowid. The table does not declare
AUTOINCREMENT, however, so deleting rows—especially the highest row—can allow
SQLite to reuse a number later. Security-key rows should therefore be treated
as append-only operational data.
SqliteStore uses WAL mode and PRAGMA synchronous = 0. This favors startup
and feed latency over maximum durability against a host/power failure. The
unique (ns, key) index prevents two different rows for the same logical
security.
Other places the key is stored
| Location | Purpose | Authoritative for allocation? |
|---|---|---|
| Hot and cold shared-memory segments | Live publication and fast consumer lookup | No; rebuilt at process startup |
CMESecurityIndex / GeneralSecurityIndex and adapter maps |
In-process routing from an incoming ID to a producer handle | No; in-memory only |
<data_path>/databases/md/securities.sqlite |
Serialized legacy api::Security definitions keyed by Spark key; used by security cache and MDP3 Sunday preload |
No; it stores metadata after a key exists |
| Relay/market-data recordings | Preserve the key embedded in recorded securities, books, and trades for deterministic replay | No; replay normally reuses the recorded value |
| Orders, fills, positions, APIs | Reference a security by the published Spark key | No |
The SecurityIDRecorder::work_db() comment says a published security is also
recorded in the security database, but that function does not write
securities.sqlite in the current code. Full security definitions are persisted
separately by SecurityCache::flush_to_disk() through
db::insert_or_replace().
CME MDP3 / OnixS pipeline in detail
1. Construction and thread setup
MDP3Main::PImpl owns one CMESecurityIndex, one SecurityIDRecorder, and one
MDProducer. It constructs the recorder with:
- a lookup callback that maps
ExchangeKeythrough the CME index to the handle's current Spark key; and Dispatcher::current(), which drives a 200 ms recorder timer.
In the current MarketDataGateway route, OnixsGlbxSubscription creates a
dedicated dispatcher_thread, constructs and starts MDP3Main on that thread,
then calls that thread's dispatcher run(). The older md run onixs command
uses its command/dispatcher thread similarly.
The OnixS feed engine has its own configured thread pool. Instrument and market data callbacks therefore originate on OnixS feed-engine threads, while recorder database work and final key assignment execute on the MDP3 dispatcher thread.
| MDP3 stage | Executing thread | DB call? |
|---|---|---|
Decode definition and set CME numeric_id |
OnixS feed-engine callback thread | No |
| Allocate hot/cold entry and queue handle | Same OnixS callback thread | No |
| Add CME-ID index entry | Same OnixS callback thread | No |
| Resolve spread legs | MDP3 dispatcher thread, on 200 ms timer | No |
| Select/insert Spark key | MDP3 dispatcher thread, on 200 ms timer | Yes |
handle.set_key() and public visibility |
MDP3 dispatcher thread, in DB callback | No additional call |
2. Definition callback: exchange ID is set, Spark key is not
For futures, options, and spreads, MDP3Listener::set_security_common() does:
fsec.numeric_id = def.securityId();
It also copies symbol/descriptor/venue and pricing fields. A newly constructed
FixedSecurity already has key == invalid_security_key (0), and this helper
does not change it.
For a spread, every leg starts with:
leg.exchange_key = ExchangeKey(definition_leg.legSecurityId());
leg.key = invalid_security_key;
Thus both the parent and legs initially carry exchange-routing identity, but no Spark identity.
3. Shared-memory allocation and exchange indexing
MDP3Listener::add_security() performs these operations in order:
_producer.add_security(security)allocates a hot entry and a variable-sized cold entry and copies the security into the cold segment.- Because the key is
0,MDProducerqueues the handle throughSecurityIDRecorder::ensure_key(). _index.add_security(security.numeric_id, handle)immediately creates the CME-ID-to-handle mapping.- The initial exchange status is written through the handle.
This ordering means incoming books can already be routed by CME ID while the
Spark key is pending. Shared-memory consumers cannot discover the security yet.
Also, MDProducerHandle::add_trade() only appends to the public trade queue
when key() != 0, so a trade received in this short window updates the cold
last-trade state but is not added to the keyed trade log.
4. Recorder tick: spreads, handles, then database
Every 200 ms the recorder calls, in this exact order:
work_spreads()tries to fill previously unresolved spread leg keys by askingCMESecurityIndexfor each leg exchange ID.work_handles()drains new handles. Outrights are converted to an API security and queued for DB lookup immediately. A spread without all leg keys is retained inpending_handleand its leg pointers are registered inunresolved_legs.work_db()resolves queued(venue, type, descriptor)requests.
Because spread work precedes new-handle work, a newly seen unresolved spread normally waits until at least the next timer tick to resolve legs. It waits longer if any outright leg has not yet received its Spark key. A spread's own key is not requested until every leg key is non-zero.
5. Lookup or generation
work_db() first queries (ns, key):
- Existing row: use its
rowidas the key and invoke the callback. - Missing row: batch it into
insert_many_resolve_ids(). SQLite assigns a new rowid inside a transaction, the row object receives that ID, and the callback runs after the transaction completes.
New inserts are throttled to roughly 200 per timer pass (the current > 200
check can admit 201) to avoid delaying heartbeats during startup. The queues are
TBB concurrent queues, but all SQLite access is serialized onto the recorder's
dispatcher thread. check_access() asserts this single-thread rule after the
first work pass records the dispatcher thread ID.
6. Final assignment and visibility
The queued callback captures a copy of MDProducerHandle and calls
handle.set_key(id) on the dispatcher thread. set_key() writes hot, then cold,
then increments security_count. From this point:
- shared-memory scans report the security;
CMESecurityIndex::get_key(CME_ID)returns the Spark key;- trades and RFQs can be emitted with a non-zero key;
- resolved spreads can copy this key into their leg structures.
7. Sunday preload is a fast path
For configured symbols on Sunday, MDP3Main::prepare_listener() loads stored
definitions from securities.sqlite. It converts them with
set_security_key=true, so both the security key and spread leg keys come from
the serialized API security. MDProducer::add_security() calls set_key()
immediately, and no recorder lookup is required for that preloaded handle. The
CME exchange ID is still restored into numeric_id, allowing the normal CME
index to be built.
Generalized/newer pipeline in detail
1. Topology and ownership
Run::run() starts a detached thread named MarketDataGateway. On that thread,
run_md_gwy() obtains the process-wide IDGenerator, constructs
MarketDataGateway, and starts the configured providers. For each (venue, source) returned by a provider, the gateway creates an MdSink containing:
- a venue-local
GeneralSecurityIndex; - a default
MDProducer(not wired to aSecurityIDRecorder); and - a
GeneralMarketDataListenerconnecting the source to that producer/index.
The source decides its callback thread. Some sources call the listener while
starting; others own network or replay threads. The generalized listener calls
the producer inline on that source thread. The process-wide IDGenerator
serializes its own calls with a mutex, but the listener's transient
cc_sec_id_map is an ordinary unordered_map, so each source/listener pair is
expected to serialize security/event callbacks.
| Generalized stage | Executing thread | DB call? |
|---|---|---|
| Build/parse domain definition | Source/adaptor thread | No |
IDGenerator::get_or_add() |
Same source/adaptor thread | Yes, synchronously |
GeneralMarketDataListener::on_securities() |
Same source callback thread | No in current use-given mode |
Copy ID to FixedSecurity::key and numeric_id |
Same source callback thread | No |
MDProducer::add_security() / set_key() |
Same source callback thread | No additional call |
Add GeneralSecurityIndex entry |
Same source callback thread | No |
There are two important compatibility exceptions inside the provider framework:
OnixsGlbxSubscription::start()ignores the supplied generalized listener and startsMDP3Main; CME/OnixS therefore still follows the asynchronous legacy flow documented above.EmumbaDeribitSubscription::start()also ignores that listener and usesEmMDListener, which allocates viaIDGeneratorand writes through its own producer/index. Its assignment is still synchronous and pre-populated, so it resembles the newer ID flow even though it does not pass throughGeneralMarketDataListener.
2. How a new ID is constructed
For a normal live generalized adapter, ID creation happens in the adapter,
before GeneralMarketDataListener::on_securities():
logical identity = (domain venue, domain security type, adapter-chosen name)
Spark ID = SQLite rowid for that identity in key_ids.sqlite
IDGenerator::get_or_add(venue, type, name) formats a namespace by stripping
the enum prefixes:
VENUE_TAIFEX + SECURITY_TYPE_FUTURE -> TAIFEX.FUTURE
It then calls retrieve_one(ns, name). If the row exists, it returns that
row's rowid. If absent, it inserts (ns, name, ""), and SQLite assigns the
next rowid. That integer is wrapped in domain::SecurityID and placed in
SecurityCommon::id.
The third identity component is adapter-specific, and this matters when trying to reproduce a mapping:
| Source/adaptor family | Typical name sent to IDGenerator |
|---|---|
| Binance, Coinbase, Deribit, OKX and similar native/CCAPI paths | Usually sec.desc() / exchange instrument name |
| Taifex | product/instrument ID string |
| Databento | std::to_string(msg.hd.instrument_id), not the human-readable description |
| Python bridge | exchange_security_id (and it asserts Python supplied ID is zero for definitions) |
| Emumba Deribit | extracted Emumba security descriptor |
| Replay | no new allocation for a recorded definition; it reuses the recorded SecurityCommon::id |
So “security name” alone is not a universal logical key. Reproduction requires the exact venue enum, exact security-type enum, and the exact adapter-specific name used at insertion time.
3. What GeneralMarketDataListener does with that ID
on_securities() first calls _find_sec(sec, true). Its current, compile-time
behavior is controlled by:
const bool use_given_sec_key = true;
Consequently, for a newly seen incoming domain::SecurityID it does not
call the DB. It copies security_common.id.value() into its transient
cc_sec_id_map. _add_future(), _add_option(), _add_spread(), or
_add_spot() then calls _copy_security_common(), which sets:
to.key = incoming_id;
to.numeric_id = incoming_id;
This collapses the generalized source/correlation ID and Spark shared-memory
key into the same number. The source's text exchange_sec_id remains on the
domain security but is not copied into FixedSecurity::numeric_id by this
path.
_commit_security() then:
- calls
MDProducer::add_security()with an already non-zero key; - causes
MDProducerHandle::set_key()to run immediately during that call; - inserts the handle in
GeneralSecurityIndexundernumeric_id(the same number); and - returns with the security already visible in shared memory.
Because the sink's producer has no recorder, supplying id == 0 would allocate
the shared-memory entry but never assign or publish its key. Generalized live
sources must therefore provide a valid non-zero ID before on_securities().
The listener contains an alternative branch for use_given_sec_key == false.
That branch would call IDGenerator::get_or_add(me->venue, sec.type(), sec.desc()) itself. It is currently disabled, and _generate_security_key() is
also effectively unused. Turning the flag off is not a mechanical change: an
adapter such as Databento allocates using numeric instrument ID while the
listener branch would use sec.desc(), potentially creating a second key for
the same instrument.
There is also a spread failure edge case. _find_sec(sec, true) records the
parent ID in cc_sec_id_map before _add_spread() resolves its legs. If any leg
ID is absent, _add_spread() logs and returns without committing the spread. A
later repeat of the parent definition is then classified as a duplicate and
ignored, so definition ordering is a hard requirement in the current code.
4. How later books, trades, and statistics find the security
Every later domain event carries a domain::SecurityID:
Book::security_id();TradeRecord::sec_id; orSecurityMarketDataStatistics::security_id.
The listener first uses cc_sec_id_map to translate that event ID to the value
stored in GeneralSecurityIndex, then obtains the MDProducerHandle. With
use_given_sec_key == true, this mapping is numerically identity-to-identity,
but it still serves as the admission check that the definition was seen.
Most high-throughput adapters avoid SQLite on market-data events:
- Databento allocates on the definition and caches
instrument_id -> domain::SecurityIDinsec_id_mapfor books/trades. - Taifex keeps the resolved ID in its security-details mapper.
- Other native subscriptions maintain comparable descriptor/ID maps.
The Python bridge is a notable exception: to_bts_security_(),
to_bts_trade_(), and to_book_() all call get_sec_id_(), which calls
IDGenerator::get_id(..., add_if_new=true). As written, that performs at least
one SQLite SELECT for each converted Python definition, trade, or book. It
returns the existing ID after the first insert, but it is still a DB call and
takes the global IDGenerator mutex on every event.
5. Persistence timing and latency
The newer allocator is synchronous:
- Existing security: mutex acquisition plus one SQLite
SELECTon the caller's thread. - New security: mutex acquisition, one
SELECT, then one SQLiteINSERT(SQLite autocommit), also on the caller's thread. - Listener publication: no further DB call under the current
use_given_sec_keysetting.
The ID therefore exists in key_ids.sqlite before the adapter sends the
definition to the listener. There is no intentional 200 ms publication delay,
but the source callback/startup path directly pays DB and mutex latency. Bulk
definition discovery performs one lookup and possibly one insert per security;
IDGenerator does not batch those operations.
WAL mode helps concurrency, while synchronous=0 reduces fsync cost. It does
not eliminate SQL parsing/stepping, mutex contention, page-cache misses, or
occasional file-system work. The impact is usually a startup/definition cost
when adapters cache the result, but it becomes hot-path latency for code such as
the current Python bridge that resolves every event through the DB.
GeneralMarketDataListener itself does not persist the full contract definition
to securities.sqlite; that remains the separate security-cache/metadata
persistence path described earlier.
Comparison: MDP3 recorder vs generalized allocator
| Property | MDP3 / SecurityIDRecorder |
Generalized / IDGenerator |
|---|---|---|
| Logical input | venue + type + legacy security_desc |
venue + type + adapter-chosen name |
| Numeric output | SQLite rowid, checked/read as 32-bit security_key |
SQLite rowid returned as 64-bit domain::SecurityID, then narrowed into 32-bit shared-memory key |
| Allocation point | After definition is copied to shared memory | Normally inside adapter before listener sees definition |
| DB execution | Asynchronous on dispatcher timer | Synchronous on caller/source thread |
| Existing ID cost | One deferred SELECT |
One immediate SELECT per call |
| New ID cost | Deferred, batched transaction (about 200/pass) | Immediate SELECT + individual autocommit INSERT |
| Time until public | Usually 0–200+ ms; spreads may take extra ticks | Immediate after synchronous lookup/insert completes |
| Feed callback blocking | Queue push only for ID work | Yes, if the adapter invokes IDGenerator there |
| Spread legs | Parent waits asynchronously until all leg keys resolve by exchange ID | Definition processing requires every leg's incoming ID already exist in cc_sec_id_map; otherwise the spread is aborted |
| Runtime ID relationship | Exchange ID and Spark key remain distinct | Incoming/canonical ID, numeric_id, index ID, and Spark key are currently the same number |
| Persistence database | key_ids.sqlite |
The same key_ids.sqlite |
| Batching | Yes for new rows | No |
| Failure behavior | Security has an internal handle with key 0 while normal queued work is pending; DB work does not block the feed callback | DB errors/zero results occur in the synchronous source path; a zero ID can leave an unpublishable entry, with no pending-key state machine |
Neither algorithm generates a UUID or a content hash. Both ultimately use an
opaque, database-assigned SQLite rowid. It is normally increasing and looks like
a sequence, but this schema does not request SQLite AUTOINCREMENT. The tuple
used to look up the row is logical; the resulting number is not derivable from
that tuple.
Can another system reconstruct the same keys?
Not from (venue, type, name) alone. Another system can reconstruct the
logical lookup tuple, but it cannot calculate the numeric Spark key because
the key depends on the historical insertion order of every row in
key_ids.sqlite.
For example, two empty systems can insert the exact same GLBX.FUTURE / ESZ6
row and receive different IDs if other securities were inserted before it on
one system. They are sequential opaque IDs, not deterministic hashes and not
UUIDs.
Ways to preserve cross-system identity are:
- Copy/replicate
key_ids.sqlite(including rowids) and keep descriptor rules identical. This preserves existing numeric keys but requires coordinated allocation for new ones. - Export an explicit mapping of
(namespace, adapter key) -> Spark key, then import rows with their original rowids before either system allocates new securities. - Treat a stable external identity (venue + type + exchange instrument ID, or another canonical instrument identifier) as the cross-system key and carry a translation table to each system's local Spark key.
- Introduce a deterministic ID scheme only as a versioned migration. A hash can be computed independently, but collision handling, descriptor canonicalization, aliases/renames, the existing 32-bit ABI, and compatibility with orders/recordings/databases all need explicit design.
Copying only securities.sqlite is insufficient as an allocator strategy. It
contains security blobs keyed by already-assigned IDs, but key_ids.sqlite is
the mapping used to find or allocate those IDs. Conversely, copying only
key_ids.sqlite preserves the numbers but not all contract metadata.
Other assignment paths in sparkd
The two main pipelines reduce to four ways a producer handle can acquire a key:
- Already keyed input.
MDProducer::add_security()sees a non-zeroFixedSecurity::keyand callsset_key()immediately. Generalized feeds, Sunday preload, relay playback, and other replay paths use this mechanism. - Recorder-backed live input. The input key is zero and the producer owns a
SecurityIDRecorder;ensure_key()queues asynchronous assignment. CME MDP3 and legacy OnixS ICE use this mechanism. ICE usesmarketIdasnumeric_idand otherwise follows the same recorder/index design. - Service assignment.
SyntheticSecurityServicecallsSecurityIDRecorder::assign_key()directly on a background dispatcher. The callback receives a copiedapi::Securitywith its top-levelidset, then publishes it to synthetic subscribers. It uses the samekey_ids.sqliterowid allocator. - Unkeyed input with no recorder. The entry is allocated with key zero and never becomes public. This is not a valid production assignment path, but it is the failure mode if a generalized source omits its ID.
Relay and replay deserve care when reasoning about persistence:
- The generalized
MDReplayerforwards the recorded domain security ID and emits later books/statistics with the recordedsec_key, so the listener's current use-given mode preserves the recording's numeric identity. - The legacy relay producer converts a recorded
api::Securitywithset_security_key=true, so its shared-memory publication also preserves the embedded key. - The legacy relay recorder additionally calls
bulk_insert(..., "REPLAY: ")for logical mappings. Those rows use aREPLAY:namespace prefix; this is a separate mapping population step, not a deterministic derivation of the key.
Operational implications
- Back up
key_ids.sqliteandsecurities.sqlitetogether if numeric ID and definition continuity matter. - Do not delete/rebuild
key_ids.sqliteon one host and assume IDs will match another host, even when all instruments are identical. - Before moving a feed between adapters, compare the exact
nameeach adapter supplies toIDGenerator. Matching venue/type is insufficient. - Cache generated IDs in the source after definition processing. Avoid calling
IDGeneratorfrom book/trade hot paths; the Python bridge currently violates this guideline. - Keep generated rowids below the 32-bit shared-memory ceiling until the
security_keyABI is widened or explicit bounds checking is added to the generalized path. - Ensure spread legs are announced before generalized spread parents.
- Treat
synchronous=0as a durability decision. A copied DB should be taken with SQLite-aware backup/checkpoint handling rather than by casually copying active WAL files.
Useful inspection queries (run against the configured DB path) are:
SELECT rowid AS spark_key, ns, key
FROM store
ORDER BY rowid;
SELECT rowid AS spark_key, ns, key
FROM store
WHERE ns = 'GLBX.FUTURE' AND key = '<exact-adapter-key>';
Source map
SecurityIDRecorder.cpp: legacy queues, 200 ms work ordering, logical row construction, DB callbacks, and spread-leg resolution.IDGenerator.cpp: synchronous generalized lookup/allocation and process-wide mutex.SqliteStore.cpp: DB location, schema, WAL/synchronous pragmas, individual inserts, and batch inserts.MDProducer.cppandMDProducerHandle.cpp: shared-memory allocation, recorder handoff, and ordered publication.FixedSecurity.handMDStructure.h: identifier widths and hot/cold storage fields.MDP3Main.cppandMDP3Listener.h: MDP3 construction, definition callbacks, CME ID extraction, and spread legs.MDP3Listener.cpp: producer/index insertion order.OnixsGlbxSubscription.cpp: current dedicated MDP3 dispatcher-thread ownership.MarketDataGateway.cppandGeneralMarketDataListener.cpp: generalized source/sink topology, incoming-ID mapping, fixed-security conversion, and publication.DatabentoDataHandler.cpp,TaifexDataProcessors.h, andPyMarketDataSubscription.cpp: representative adapter-side allocation and caching behavior.SecurityDB.cppandSecurityCache.cpp: separate persistence of full security definitions.SecurityIndex.h,MBOServiceServer.cpp, andMDConsumer.cpp: exchange-index translation followed by Spark-keyed market-data consumption and MBO access.OrderFactory.cpp,FillStore.cpp, andPositionMap.cpp: representative downstream use of Spark security keys.MDReplayer.cppandRelayClient.cpp: recorded ID reuse.