Files
spark/doc/sparkd-security-id-lifecycle.md

48 KiB
Raw Permalink Blame History

Security ID handling in sparkd

This document traces how security identifiers are created, translated, persisted, indexed, and published by the sparkd market-data paths. It focuses on the legacy CME MDP3/OnixS pipeline and the newer generalized market-data pipeline.

Executive summary

sparkd uses two identifiers that must not be confused:

Identifier Meaning Typical field Lifetime
Exchange/feed ID The venue's identifier used to route incoming definitions, books, and trades FixedSecurity::numeric_id, ExchangeKey, or domain::Security::security_common.exchange_sec_id Feed/session-specific unless the source guarantees stability
Spark security key Spark's internal non-zero identifier used in shared memory, APIs, orders, positions, and downstream services FixedSecurity::key, api::Security::id(), SecurityKey; historically also called security_id Intended to be stable across restarts

invalid_security_key is 0; a security with key 0 is not yet publicly discoverable through the shared-memory consumer path.

The two principal paths differ:

  • MDP3/OnixS: the definition callback sets only the CME ID in numeric_id. MDProducer::add_security() allocates the shared-memory entry with key 0. SecurityIDRecorder, driven by a 200 ms dispatcher timer, resolves or inserts a stable key in key_ids.sqlite; its callback finally calls MDProducerHandle::set_key().
  • Generalized/newer pipeline: GeneralMarketDataListener::on_securities() currently uses the ID already present on the incoming domain security (use_given_sec_key == true). Live adapters normally obtain that ID synchronously from IDGenerator first. _copy_security_common() then copies it to both FixedSecurity::key and numeric_id, so MDProducer::add_security() publishes it synchronously. Only the listener's own fallback IDGenerator call is bypassed by the current constant.

End-to-end diagrams

CME MDP3 / OnixS

sequenceDiagram
    participant FE as OnixS feed-engine thread
    participant L as MDP3Listener
    participant P as MDProducer / shared memory
    participant Q as SecurityIDRecorder queues
    participant D as sparkd dispatcher thread
    participant DB as key_ids.sqlite

    FE->>L: instrument definition (CME SecurityID)
    L->>L: set_security_common() - numeric_id = CME SecurityID, key remains 0
    L->>P: add_security(FixedSecurity)
    P->>P: allocate hot + cold entries with key 0
    P->>Q: ensure_key(handle)
    L->>L: index[CME SecurityID] = handle
    Note over FE,L: Books can route by CME ID while the Spark key is pending
    D->>Q: every 200 ms - work_spreads(), work_handles(), work_db()
    Q->>DB: find (namespace, descriptor), or insert a new row
    DB-->>Q: SQLite rowid = Spark security key
    Q->>P: handle.set_key(rowid)
    P->>P: hot.key first, then cold security.key, then increment security_count
    Note over P: The security is now public to shared-memory consumers

Generalized/newer pipeline (current behavior)

sequenceDiagram
    participant A as Venue adapter / plugin
    participant DB as IDGenerator / key_ids.sqlite
    participant G as GeneralMarketDataListener
    participant I as GeneralSecurityIndex
    participant P as MDProducer / shared memory

    A->>DB: get_or_add(venue, type, adapter key)
    DB-->>A: SQLite rowid as domain SecurityID
    A->>G: on_securities(domain Security list with ID)
    G->>G: _find_sec() - use incoming security_common.id
    G->>G: _copy_security_common() - key = id, numeric_id = id
    G->>P: add_security(FixedSecurity with non-zero key)
    P->>P: set_key() during add_security()
    G->>I: index[id] = handle
    Note over G,P: No asynchronous ID assignment in the current configuration

Data-flow charts

The sequence diagrams above emphasize timing and thread transitions. These flowcharts emphasize which identifier is carried through each representation and where data is retained.

MDP3 / legacy data flow

flowchart TD
    CME["CME MDP3 packets<br/>CME SecurityID"] --> ONIXS["OnixS feed engine<br/>decode definition and updates"]

    ONIXS --> DEF["MDP3Listener definition callback"]
    DEF --> FIXED["FixedSecurity<br/>numeric_id = CME SecurityID<br/>key = 0"]

    FIXED --> PRODUCER["MDProducer::add_security"]
    PRODUCER --> PENDING["Hot and cold shared memory<br/>allocated but not public<br/>key = 0"]
    PRODUCER --> QUEUE["SecurityIDRecorder<br/>pending_handle queue"]

    FIXED --> CMEINDEX["CMESecurityIndex<br/>CME SecurityID to handle"]

    QUEUE --> TIMER["Dispatcher timer<br/>every 200 ms"]
    TIMER --> LEGS{"Spread legs<br/>all resolved?"}
    LEGS -- No --> UNRESOLVED["unresolved_legs<br/>exchange IDs awaiting keys"]
    UNRESOLVED --> CMEINDEX
    CMEINDEX --> LEGS
    LEGS -- Yes or outright --> LOGICAL["Logical identity<br/>venue + type + security_desc"]

    LOGICAL --> SELECT["SQLite SELECT<br/>key_ids.sqlite"]
    SELECT -- Existing row --> ROWID["SQLite rowid<br/>Spark security key"]
    SELECT -- Missing row --> INSERT["Batched SQLite INSERT"]
    INSERT --> ROWID

    ROWID --> SETKEY["MDProducerHandle::set_key"]
    SETKEY --> HOT["Write hot-entry key"]
    HOT --> COLD["Write cold security key"]
    COLD --> COUNT["Increment security_count"]
    COUNT --> PUBLIC["Public security<br/>visible to MDConsumer"]

    ONIXS --> UPDATE["Book, trade, status, or RFQ<br/>carrying CME SecurityID"]
    UPDATE --> CMEINDEX
    CMEINDEX --> HANDLE["MDProducerHandle"]
    HANDLE --> LIVE["Update live shared-memory data"]
    PUBLIC --> LIVE

The CME index is usable before the Spark key is assigned, so feed updates can reach the internal handle while the security is still hidden from consumers. The persistent rowid crosses the publication boundary only at set_key().

Generalized/newer data flow

flowchart TD
    FEED["Venue feed, Databento,<br/>Python plugin, or replay"] --> ADAPTER["Source adapter<br/>build domain::Security"]

    ADAPTER --> IDTYPE{"Live definition or<br/>recorded/pre-keyed input?"}
    IDTYPE -- Live --> IDINPUT["Logical identity<br/>venue + type + adapter-chosen name"]
    IDINPUT --> MUTEX["IDGenerator global mutex"]
    MUTEX --> SELECT["Synchronous SQLite SELECT<br/>key_ids.sqlite"]
    SELECT -- Existing row --> ROWID["SQLite rowid"]
    SELECT -- Missing row --> INSERT["Synchronous SQLite INSERT<br/>autocommit"]
    INSERT --> ROWID
    ROWID --> DOMAINID["domain::SecurityID<br/>used as Spark ID"]
    IDTYPE -- Recorded or pre-keyed --> DOMAINID

    DOMAINID --> DOMAINSEC["SecurityCommon::id = Spark ID"]
    DOMAINSEC --> LISTENER["GeneralMarketDataListener::on_securities"]
    LISTENER --> CCMAP["cc_sec_id_map<br/>incoming ID to Spark ID"]
    LISTENER --> CONVERT["_copy_security_common<br/>FixedSecurity.key = ID<br/>numeric_id = ID"]

    CONVERT --> PRODUCER["MDProducer::add_security"]
    PRODUCER --> SETKEY["Immediate set_key<br/>no recorder timer"]
    SETKEY --> SHM["Hot and cold shared memory<br/>immediately public"]
    CONVERT --> GENERALINDEX["GeneralSecurityIndex<br/>ID to handle"]

    FEED --> EVENT["Book, trade, or statistics<br/>with exchange/source identity"]
    EVENT --> EVENTMAP["Adapter map<br/>source identity to domain::SecurityID"]
    EVENTMAP --> EVENTID["Event carries domain::SecurityID"]
    EVENT --> PYHOT["Python bridge only<br/>synchronous IDGenerator lookup"]
    PYHOT --> EVENTID
    PYHOT -. uses same mutex and SQLite path .-> MUTEX
    EVENTID --> CCMAP
    CCMAP --> GENERALINDEX
    GENERALINDEX --> HANDLE["MDProducerHandle"]
    HANDLE --> SHMUPDATE["Update book, trade,<br/>statistics, or status"]
    SHM --> SHMUPDATE

In the normal newer live path, persistence occurs before the definition enters the generalized listener. The DB call therefore adds synchronous adapter-side latency, but publication needs no later timer. Most adapters use an in-memory map for subsequent events; the dotted Python path is the documented exception.

The publication boundary

MDProducerHandle::set_key() is the definitive publication point for the legacy shared-memory representation. Its write ordering is intentional:

  1. Write the key into the hot book entry (HotEntry or HotDepthEntry).
  2. Write the key into the cold security entry.
  3. Increment the cold header's public security_count.

The cold security key is written last because consumers treat a non-zero cold key as visibility. At that moment the corresponding hot entry must already be addressable. set_key() only permits the transition from 0 to one key; it is not a key-update mechanism.

Identifier representations

The names are historically inconsistent. In particular, many protobuf fields named security_id contain the Spark key, not the exchange's ID.

Layer Exchange/feed identifier Spark identifier
MDP3 wire/OnixS securityId() not present
Legacy fixed security FixedSecurity::numeric_id (uint64_t) FixedSecurity::key / security_key (uint32_t)
Legacy API security venue-specific exchange_id top-level Security.id()
New domain model SecurityCommon::exchange_sec_id (string) is descriptive; SecurityCommon::id is also used to correlate source events domain::SecurityID (uint64_t), currently copied into the Spark key
Shared-memory hot book none HotEntry::key or HotDepthEntry::key
Shared-memory cold metadata FixedSecurity::numeric_id FixedSecurity::key
CME runtime lookup CMESecurityIndex key SecurityData::handle.key()
General runtime lookup incoming/canonical domain::SecurityID GeneralSecurityIndex key and handle.key() (currently the same value)

The generalized model declares domain::SecurityID as 64-bit, but the legacy shared-memory security_key remains 32-bit. _copy_security_common() assigns a 64-bit value to both numeric_id and the 32-bit key. This is safe only while allocated IDs fit in 32 bits; unlike the legacy lookup path, this assignment has no explicit checked conversion.

Which parts use exchange IDs, descriptors, or Spark security IDs?

The three identifier classes serve different jobs:

  • Exchange/external ID: routes data inside a venue or vendor feed adapter.
  • Descriptor/natural key: recognizes a definition logically at the persistence, discovery, or human-facing boundary.
  • Spark security ID/key: joins the instrument to books, orders, fills, positions, and services after publication.

The core downstream system does not repeatedly identify a security from its description. The description is primarily used to obtain a Spark key; after that boundary, most joins use the numeric Spark key.

Component-level usage matrix

Component or data path Exchange/external ID Descriptor or name Spark security ID/key
CME MDP3 definition decoder Reads securityId() and stores it in FixedSecurity::numeric_id Copies definition symbol/tag 55 into security_desc Leaves FixedSecurity::key at 0 initially
CMESecurityIndex Primary map key; CME ID resolves to a handle Not used for update routing Returns handle.key() when the Spark key has been assigned
MDP3 books, trades, status, statistics, and RFQs Incoming CME ID selects the handle through CMESecurityIndex Not used to locate each update The selected handle writes keyed shared-memory/trade data after publication
MDP3 spread definitions Each leg initially contains legSecurityId() as exchange_key Parent security_desc identifies the spread's DB row Each leg's Spark key and then the parent Spark key must be resolved before publication
Legacy ICE OnixS Uses ICE marketId as numeric_id and index key ICE security description becomes the logical lookup name Uses the same recorder-assigned Spark key pattern as MDP3
SecurityIDRecorder Does not allocate from the external ID Uses (venue, security type, securities::descriptor()) to find/create a row Treats the SQLite rowid as the resulting Spark key
Native generalized adapters Use exchange strings/numeric IDs to parse definitions and correlate updates Usually pass a description or exchange instrument name to IDGenerator Put the returned ID into SecurityCommon::id and later events
Databento adapter Uses msg.hd.instrument_id for feed correlation and its in-memory map Uses the decimal instrument ID string—not the human description—as the IDGenerator name Caches instrument_id -> domain::SecurityID and places that ID on books/trades
Python bridge Reads exchange_security_id from definitions, books, and trades Uses that external-ID string as the IDGenerator name Resolves it to domain::SecurityID; currently performs this lookup for every event
IDGenerator Does not interpret whether name came from an exchange ID Uses exactly (venue, type, caller-supplied name) Returns the SQLite rowid as a 64-bit ID
GeneralMarketDataListener Does not route on exchange_sec_id directly Copies sec.desc() into fixed metadata; its disabled fallback allocator would use sec.desc() Uses incoming SecurityCommon::id, maps events through cc_sec_id_map, and copies the value into both key and numeric_id
GeneralSecurityIndex No separate external-ID concept in the current use-given path Not used Maps the generalized ID/Spark key to MDProducerHandle
MDProducer / shared memory Retains numeric_id; in legacy feeds this is the exchange ID, but in the generalized path it is currently the Spark ID Stores security_desc in the cold definition Uses the key in hot/cold entries; non-zero key is the visibility boundary
MDConsumer Not normally used to find live books Can enumerate/filter converted security definitions Finds handles and book data by security_key
SecurityCache / securities.sqlite Preserves venue-specific exchange data inside the serialized security blob Maintains descriptor/name lookup indexes and supports venue/type/name queries Primary persisted definition key is api::Security.id()
MBO processor and MBO service May retain exchange order IDs, which are separate from security exchange IDs Not used to find the instrument Indexes snapshots and requests by SecurityKey; MBO protobuf security_id means Spark key
Order manager and algorithms Execution gateways eventually use venue-specific fields from the resolved security to encode orders Descriptions appear in logs and errors, not as the primary join Order definitions, subscriptions, book events, native orders, spread legs, and fills use security_key / Security.id()
Position, PNL, and risk services Not used as their position join May resolve metadata for display/reporting Positions, fills, overrides, and PNL handles are keyed by Spark security ID
Relay and market-data recordings Serialized security definitions may preserve venue exchange IDs Preserve full descriptors in recorded definitions Books, updates, trades, and replay correlation preserve the recorded Spark key
Synthetic security service No venue exchange ID; synthetics use VENUE_BTS Synthetic name is the recorder's logical descriptor Recorder assigns the SQLite rowid and synthetic metadata is keyed by it

Exchange/external IDs: where they stop

In MDP3 the separation is explicit:

CME SecurityID
    -> FixedSecurity.numeric_id
    -> CMESecurityIndex lookup
    -> MDProducerHandle
    -> Spark security_key for downstream publication

The external CME ID continues to route feed messages but does not become the order/position key. This permits the same downstream Spark key to survive a restart as long as the logical descriptor mapping remains in key_ids.sqlite, even if the feed must rediscover the CME routing ID.

The generalized path collapses more of these roles:

exchange/vendor identity
    -> adapter calls IDGenerator
    -> domain::SecurityID
    -> FixedSecurity.key and FixedSecurity.numeric_id
    -> GeneralSecurityIndex and downstream Spark key

SecurityCommon::exchange_sec_id may still contain the original textual external identifier, but GeneralMarketDataListener does not use it to route events and does not put it in numeric_id. The adapter must translate later events to domain::SecurityID before invoking the listener.

Descriptor/name users

Descriptors are used in three main places:

  1. Key allocation: SecurityIDRecorder uses the legacy venue-specific descriptor, while IDGenerator uses its caller-provided name.
  2. Definition lookup and discovery: SecurityCache supports venue/type/name lookup; symbol filters and administrative tools use security metadata to select or display instruments.
  3. Diagnostics: logs include symbols and descriptions so humans can understand a numeric key.

Descriptions generally do not drive the high-rate book, trade, order, fill, or position paths. Those paths use numeric IDs after definition resolution.

It is important that “descriptor” does not mean one universal canonical field:

  • Legacy CME uses Globex security_desc from tag 55.
  • Many generalized adapters use sec.desc() or an exchange instrument name.
  • Databento uses its numeric instrument ID converted to a string.
  • Python uses exchange_security_id.

Therefore, two adapters may describe the same economic contract but request different logical rows and receive different Spark IDs.

Spark security-ID users

Once a security is published, its Spark key is the dominant system identifier:

flowchart LR
    KEY["Spark security key"] --> SHM["Hot/cold shared memory"]
    KEY --> MDC["MDConsumer and book subscriptions"]
    KEY --> MBO["MBO snapshots and queries"]
    KEY --> OMS["Orders, algorithms, and spread legs"]
    KEY --> FILLS["Fills and execution records"]
    KEY --> POS["Positions, PNL, and risk"]
    KEY --> API["Services and client APIs"]
    KEY --> REC["Relay and market-data recordings"]
    KEY --> SECDB["Serialized security definitions"]

Fields named security_id, security_key, or top-level Security.id() in these downstream areas almost always mean the Spark key. They should not be compared directly with CME SecurityID, Databento instrument_id, or another venue/vendor identifier without going through the relevant adapter/index mapping.

Persistence and allocation

The authoritative key mapping: key_ids.sqlite

Both allocators use SqliteStore("key_ids"). With the normal configuration the file is:

<SystemConfig.data_path>/databases/key_ids/key_ids.sqlite

The database contains one table and two indexes:

CREATE TABLE store (ns TEXT, key TEXT, value TEXT);
CREATE UNIQUE INDEX key_index ON store(ns, key);
CREATE INDEX ns_index ON store(ns);

The SQLite rowid is the generated Spark key. ns is normally <VENUE>.<SECURITY_TYPE> (for example GLBX.FUTURE), and key is the security descriptor chosen by the caller. value is normally empty. Therefore ID stability depends on all three of these remaining stable:

  • the configured data_path and preservation of this database;
  • the venue/type namespace;
  • the descriptor supplied by the feed adapter.

The two allocator implementations share the file and convention:

  • SecurityIDRecorder (legacy): derives the namespace from legacy API enums and uses securities::descriptor(). For CME this is the Globex security_desc, populated from the definition's symbol/tag 55.
  • IDGenerator (newer): derives the namespace from new domain enums and uses the caller's name. Most adapters pass sec.desc() or an exchange instrument identifier; Databento deliberately passes the decimal instrument ID, and the Python bridge uses the exchange security ID.

Logical-row examples

The following names are illustrative. Production uses the exact string emitted by the selected feed/adapter, including its case, spacing, and weekly-series notation.

Legacy SecurityIDRecorder

For CME, set_security_common() copies the MDP3 definition's symbol/tag 55 into Globex security_desc. After converting the fixed security to api::Security, securities::descriptor() returns that same glbx.security_desc() string.

Instrument Legacy enums Example descriptor from security_desc Row searched/inserted in store
Future VENUE_GLBX, SECURITYTYPE_FUTURE ESZ6 ns='GLBX.FUTURE', key='ESZ6'
Standard option VENUE_GLBX, SECURITYTYPE_OPTION ESZ6 C6500 ns='GLBX.OPTION', key='ESZ6 C6500'
Weekly option VENUE_GLBX, SECURITYTYPE_OPTION ESZ6-W1 C6500 ns='GLBX.OPTION', key='ESZ6-W1 C6500'
Spot from a legacy-compatible venue path VENUE_COINBASE, SECURITYTYPE_SPOT BTC-USD ns='COINBASE.SPOT', key='BTC-USD'

A weekly option does not get a special namespace. Standard and weekly options are both SECURITYTYPE_OPTION, so both live in GLBX.OPTION; the exact security_desc distinguishes the weekly series. For example, if the database assigned rowids 42001 through 42004 to the rows above, those integers—not the descriptor strings—would be the Spark security keys.

Conceptually, legacy allocation is:

future:        lookup_or_insert("GLBX.FUTURE",  "ESZ6")
option:        lookup_or_insert("GLBX.OPTION",  "ESZ6 C6500")
weekly option: lookup_or_insert("GLBX.OPTION",  "ESZ6-W1 C6500")
spot:          lookup_or_insert("COINBASE.SPOT", "BTC-USD")

Newer IDGenerator

The namespace result is similar, but the third argument is chosen by the adapter. Examples using a description-oriented adapter are:

get_or_add(VENUE_GLBX, SECURITY_TYPE_FUTURE, "ESZ6");
// store: ns="GLBX.FUTURE", key="ESZ6"

get_or_add(VENUE_GLBX, SECURITY_TYPE_OPTION, "ESZ6 C6500");
// store: ns="GLBX.OPTION", key="ESZ6 C6500"

get_or_add(VENUE_GLBX, SECURITY_TYPE_OPTION, "ESZ6-W1 C6500");
// store: ns="GLBX.OPTION", key="ESZ6-W1 C6500"

get_or_add(VENUE_COINBASE, SECURITY_TYPE_SPOT, "BTC-USD");
// store: ns="COINBASE.SPOT", key="BTC-USD"

Databento intentionally produces a different logical row even when the human description is ESZ6. If its instrument ID were 12345678, it would call the equivalent of:

get_or_add(VENUE_GLBX, SECURITY_TYPE_FUTURE, "12345678");
// store: ns="GLBX.FUTURE", key="12345678"

Likewise, the Python bridge uses exchange_security_id. A spot definition with exchange_security_id="BTC-USD" produces the COINBASE.SPOT / BTC-USD row, while a provider that sends exchange_security_id="BTCUSD" produces a different row and therefore a different Spark key.

These examples show why the tuple is only logical when the adapter's naming contract is also known. GLBX.FUTURE / ESZ6 and GLBX.FUTURE / 12345678 may describe the same contract to a human but are distinct rows to key_ids.sqlite.

Both enum-prefix removals currently produce the same namespace spelling (for example, GLBX.OPTION), so both mechanisms allocate from one global SQLite rowid space. Live rows cannot share a rowid. The table does not declare AUTOINCREMENT, however, so deleting rows—especially the highest row—can allow SQLite to reuse a number later. Security-key rows should therefore be treated as append-only operational data.

SqliteStore uses WAL mode and PRAGMA synchronous = 0. This favors startup and feed latency over maximum durability against a host/power failure. The unique (ns, key) index prevents two different rows for the same logical security.

Other places the key is stored

Location Purpose Authoritative for allocation?
Hot and cold shared-memory segments Live publication and fast consumer lookup No; rebuilt at process startup
CMESecurityIndex / GeneralSecurityIndex and adapter maps In-process routing from an incoming ID to a producer handle No; in-memory only
<data_path>/databases/md/securities.sqlite Serialized legacy api::Security definitions keyed by Spark key; used by security cache and MDP3 Sunday preload No; it stores metadata after a key exists
Relay/market-data recordings Preserve the key embedded in recorded securities, books, and trades for deterministic replay No; replay normally reuses the recorded value
Orders, fills, positions, APIs Reference a security by the published Spark key No

The SecurityIDRecorder::work_db() comment says a published security is also recorded in the security database, but that function does not write securities.sqlite in the current code. Full security definitions are persisted separately by SecurityCache::flush_to_disk() through db::insert_or_replace().

CME MDP3 / OnixS pipeline in detail

1. Construction and thread setup

MDP3Main::PImpl owns one CMESecurityIndex, one SecurityIDRecorder, and one MDProducer. It constructs the recorder with:

  • a lookup callback that maps ExchangeKey through the CME index to the handle's current Spark key; and
  • Dispatcher::current(), which drives a 200 ms recorder timer.

In the current MarketDataGateway route, OnixsGlbxSubscription creates a dedicated dispatcher_thread, constructs and starts MDP3Main on that thread, then calls that thread's dispatcher run(). The older md run onixs command uses its command/dispatcher thread similarly.

The OnixS feed engine has its own configured thread pool. Instrument and market data callbacks therefore originate on OnixS feed-engine threads, while recorder database work and final key assignment execute on the MDP3 dispatcher thread.

MDP3 stage Executing thread DB call?
Decode definition and set CME numeric_id OnixS feed-engine callback thread No
Allocate hot/cold entry and queue handle Same OnixS callback thread No
Add CME-ID index entry Same OnixS callback thread No
Resolve spread legs MDP3 dispatcher thread, on 200 ms timer No
Select/insert Spark key MDP3 dispatcher thread, on 200 ms timer Yes
handle.set_key() and public visibility MDP3 dispatcher thread, in DB callback No additional call

2. Definition callback: exchange ID is set, Spark key is not

For futures, options, and spreads, MDP3Listener::set_security_common() does:

fsec.numeric_id = def.securityId();

It also copies symbol/descriptor/venue and pricing fields. A newly constructed FixedSecurity already has key == invalid_security_key (0), and this helper does not change it.

For a spread, every leg starts with:

leg.exchange_key = ExchangeKey(definition_leg.legSecurityId());
leg.key = invalid_security_key;

Thus both the parent and legs initially carry exchange-routing identity, but no Spark identity.

3. Shared-memory allocation and exchange indexing

MDP3Listener::add_security() performs these operations in order:

  1. _producer.add_security(security) allocates a hot entry and a variable-sized cold entry and copies the security into the cold segment.
  2. Because the key is 0, MDProducer queues the handle through SecurityIDRecorder::ensure_key().
  3. _index.add_security(security.numeric_id, handle) immediately creates the CME-ID-to-handle mapping.
  4. The initial exchange status is written through the handle.

This ordering means incoming books can already be routed by CME ID while the Spark key is pending. Shared-memory consumers cannot discover the security yet. Also, MDProducerHandle::add_trade() only appends to the public trade queue when key() != 0, so a trade received in this short window updates the cold last-trade state but is not added to the keyed trade log.

4. Recorder tick: spreads, handles, then database

Every 200 ms the recorder calls, in this exact order:

  1. work_spreads() tries to fill previously unresolved spread leg keys by asking CMESecurityIndex for each leg exchange ID.
  2. work_handles() drains new handles. Outrights are converted to an API security and queued for DB lookup immediately. A spread without all leg keys is retained in pending_handle and its leg pointers are registered in unresolved_legs.
  3. work_db() resolves queued (venue, type, descriptor) requests.

Because spread work precedes new-handle work, a newly seen unresolved spread normally waits until at least the next timer tick to resolve legs. It waits longer if any outright leg has not yet received its Spark key. A spread's own key is not requested until every leg key is non-zero.

5. Lookup or generation

work_db() first queries (ns, key):

  • Existing row: use its rowid as the key and invoke the callback.
  • Missing row: batch it into insert_many_resolve_ids(). SQLite assigns a new rowid inside a transaction, the row object receives that ID, and the callback runs after the transaction completes.

New inserts are throttled to roughly 200 per timer pass (the current > 200 check can admit 201) to avoid delaying heartbeats during startup. The queues are TBB concurrent queues, but all SQLite access is serialized onto the recorder's dispatcher thread. check_access() asserts this single-thread rule after the first work pass records the dispatcher thread ID.

6. Final assignment and visibility

The queued callback captures a copy of MDProducerHandle and calls handle.set_key(id) on the dispatcher thread. set_key() writes hot, then cold, then increments security_count. From this point:

  • shared-memory scans report the security;
  • CMESecurityIndex::get_key(CME_ID) returns the Spark key;
  • trades and RFQs can be emitted with a non-zero key;
  • resolved spreads can copy this key into their leg structures.

7. Sunday preload is a fast path

For configured symbols on Sunday, MDP3Main::prepare_listener() loads stored definitions from securities.sqlite. It converts them with set_security_key=true, so both the security key and spread leg keys come from the serialized API security. MDProducer::add_security() calls set_key() immediately, and no recorder lookup is required for that preloaded handle. The CME exchange ID is still restored into numeric_id, allowing the normal CME index to be built.

Generalized/newer pipeline in detail

1. Topology and ownership

Run::run() starts a detached thread named MarketDataGateway. On that thread, run_md_gwy() obtains the process-wide IDGenerator, constructs MarketDataGateway, and starts the configured providers. For each (venue, source) returned by a provider, the gateway creates an MdSink containing:

  • a venue-local GeneralSecurityIndex;
  • a default MDProducer (not wired to a SecurityIDRecorder); and
  • a GeneralMarketDataListener connecting the source to that producer/index.

The source decides its callback thread. Some sources call the listener while starting; others own network or replay threads. The generalized listener calls the producer inline on that source thread. The process-wide IDGenerator serializes its own calls with a mutex, but the listener's transient cc_sec_id_map is an ordinary unordered_map, so each source/listener pair is expected to serialize security/event callbacks.

Generalized stage Executing thread DB call?
Build/parse domain definition Source/adaptor thread No
IDGenerator::get_or_add() Same source/adaptor thread Yes, synchronously
GeneralMarketDataListener::on_securities() Same source callback thread No in current use-given mode
Copy ID to FixedSecurity::key and numeric_id Same source callback thread No
MDProducer::add_security() / set_key() Same source callback thread No additional call
Add GeneralSecurityIndex entry Same source callback thread No

There are two important compatibility exceptions inside the provider framework:

  • OnixsGlbxSubscription::start() ignores the supplied generalized listener and starts MDP3Main; CME/OnixS therefore still follows the asynchronous legacy flow documented above.
  • EmumbaDeribitSubscription::start() also ignores that listener and uses EmMDListener, which allocates via IDGenerator and writes through its own producer/index. Its assignment is still synchronous and pre-populated, so it resembles the newer ID flow even though it does not pass through GeneralMarketDataListener.

2. How a new ID is constructed

For a normal live generalized adapter, ID creation happens in the adapter, before GeneralMarketDataListener::on_securities():

logical identity = (domain venue, domain security type, adapter-chosen name)
Spark ID         = SQLite rowid for that identity in key_ids.sqlite

IDGenerator::get_or_add(venue, type, name) formats a namespace by stripping the enum prefixes:

VENUE_TAIFEX + SECURITY_TYPE_FUTURE -> TAIFEX.FUTURE

It then calls retrieve_one(ns, name). If the row exists, it returns that row's rowid. If absent, it inserts (ns, name, ""), and SQLite assigns the next rowid. That integer is wrapped in domain::SecurityID and placed in SecurityCommon::id.

The third identity component is adapter-specific, and this matters when trying to reproduce a mapping:

Source/adaptor family Typical name sent to IDGenerator
Binance, Coinbase, Deribit, OKX and similar native/CCAPI paths Usually sec.desc() / exchange instrument name
Taifex product/instrument ID string
Databento std::to_string(msg.hd.instrument_id), not the human-readable description
Python bridge exchange_security_id (and it asserts Python supplied ID is zero for definitions)
Emumba Deribit extracted Emumba security descriptor
Replay no new allocation for a recorded definition; it reuses the recorded SecurityCommon::id

So “security name” alone is not a universal logical key. Reproduction requires the exact venue enum, exact security-type enum, and the exact adapter-specific name used at insertion time.

3. What GeneralMarketDataListener does with that ID

on_securities() first calls _find_sec(sec, true). Its current, compile-time behavior is controlled by:

const bool use_given_sec_key = true;

Consequently, for a newly seen incoming domain::SecurityID it does not call the DB. It copies security_common.id.value() into its transient cc_sec_id_map. _add_future(), _add_option(), _add_spread(), or _add_spot() then calls _copy_security_common(), which sets:

to.key = incoming_id;
to.numeric_id = incoming_id;

This collapses the generalized source/correlation ID and Spark shared-memory key into the same number. The source's text exchange_sec_id remains on the domain security but is not copied into FixedSecurity::numeric_id by this path.

_commit_security() then:

  1. calls MDProducer::add_security() with an already non-zero key;
  2. causes MDProducerHandle::set_key() to run immediately during that call;
  3. inserts the handle in GeneralSecurityIndex under numeric_id (the same number); and
  4. returns with the security already visible in shared memory.

Because the sink's producer has no recorder, supplying id == 0 would allocate the shared-memory entry but never assign or publish its key. Generalized live sources must therefore provide a valid non-zero ID before on_securities().

The listener contains an alternative branch for use_given_sec_key == false. That branch would call IDGenerator::get_or_add(me->venue, sec.type(), sec.desc()) itself. It is currently disabled, and _generate_security_key() is also effectively unused. Turning the flag off is not a mechanical change: an adapter such as Databento allocates using numeric instrument ID while the listener branch would use sec.desc(), potentially creating a second key for the same instrument.

There is also a spread failure edge case. _find_sec(sec, true) records the parent ID in cc_sec_id_map before _add_spread() resolves its legs. If any leg ID is absent, _add_spread() logs and returns without committing the spread. A later repeat of the parent definition is then classified as a duplicate and ignored, so definition ordering is a hard requirement in the current code.

4. How later books, trades, and statistics find the security

Every later domain event carries a domain::SecurityID:

  • Book::security_id();
  • TradeRecord::sec_id; or
  • SecurityMarketDataStatistics::security_id.

The listener first uses cc_sec_id_map to translate that event ID to the value stored in GeneralSecurityIndex, then obtains the MDProducerHandle. With use_given_sec_key == true, this mapping is numerically identity-to-identity, but it still serves as the admission check that the definition was seen.

Most high-throughput adapters avoid SQLite on market-data events:

  • Databento allocates on the definition and caches instrument_id -> domain::SecurityID in sec_id_map for books/trades.
  • Taifex keeps the resolved ID in its security-details mapper.
  • Other native subscriptions maintain comparable descriptor/ID maps.

The Python bridge is a notable exception: to_bts_security_(), to_bts_trade_(), and to_book_() all call get_sec_id_(), which calls IDGenerator::get_id(..., add_if_new=true). As written, that performs at least one SQLite SELECT for each converted Python definition, trade, or book. It returns the existing ID after the first insert, but it is still a DB call and takes the global IDGenerator mutex on every event.

5. Persistence timing and latency

The newer allocator is synchronous:

  • Existing security: mutex acquisition plus one SQLite SELECT on the caller's thread.
  • New security: mutex acquisition, one SELECT, then one SQLite INSERT (SQLite autocommit), also on the caller's thread.
  • Listener publication: no further DB call under the current use_given_sec_key setting.

The ID therefore exists in key_ids.sqlite before the adapter sends the definition to the listener. There is no intentional 200 ms publication delay, but the source callback/startup path directly pays DB and mutex latency. Bulk definition discovery performs one lookup and possibly one insert per security; IDGenerator does not batch those operations.

WAL mode helps concurrency, while synchronous=0 reduces fsync cost. It does not eliminate SQL parsing/stepping, mutex contention, page-cache misses, or occasional file-system work. The impact is usually a startup/definition cost when adapters cache the result, but it becomes hot-path latency for code such as the current Python bridge that resolves every event through the DB.

GeneralMarketDataListener itself does not persist the full contract definition to securities.sqlite; that remains the separate security-cache/metadata persistence path described earlier.

Comparison: MDP3 recorder vs generalized allocator

Property MDP3 / SecurityIDRecorder Generalized / IDGenerator
Logical input venue + type + legacy security_desc venue + type + adapter-chosen name
Numeric output SQLite rowid, checked/read as 32-bit security_key SQLite rowid returned as 64-bit domain::SecurityID, then narrowed into 32-bit shared-memory key
Allocation point After definition is copied to shared memory Normally inside adapter before listener sees definition
DB execution Asynchronous on dispatcher timer Synchronous on caller/source thread
Existing ID cost One deferred SELECT One immediate SELECT per call
New ID cost Deferred, batched transaction (about 200/pass) Immediate SELECT + individual autocommit INSERT
Time until public Usually 0–200+ ms; spreads may take extra ticks Immediate after synchronous lookup/insert completes
Feed callback blocking Queue push only for ID work Yes, if the adapter invokes IDGenerator there
Spread legs Parent waits asynchronously until all leg keys resolve by exchange ID Definition processing requires every leg's incoming ID already exist in cc_sec_id_map; otherwise the spread is aborted
Runtime ID relationship Exchange ID and Spark key remain distinct Incoming/canonical ID, numeric_id, index ID, and Spark key are currently the same number
Persistence database key_ids.sqlite The same key_ids.sqlite
Batching Yes for new rows No
Failure behavior Security has an internal handle with key 0 while normal queued work is pending; DB work does not block the feed callback DB errors/zero results occur in the synchronous source path; a zero ID can leave an unpublishable entry, with no pending-key state machine

Neither algorithm generates a UUID or a content hash. Both ultimately use an opaque, database-assigned SQLite rowid. It is normally increasing and looks like a sequence, but this schema does not request SQLite AUTOINCREMENT. The tuple used to look up the row is logical; the resulting number is not derivable from that tuple.

Can another system reconstruct the same keys?

Not from (venue, type, name) alone. Another system can reconstruct the logical lookup tuple, but it cannot calculate the numeric Spark key because the key depends on the historical insertion order of every row in key_ids.sqlite.

For example, two empty systems can insert the exact same GLBX.FUTURE / ESZ6 row and receive different IDs if other securities were inserted before it on one system. They are sequential opaque IDs, not deterministic hashes and not UUIDs.

Ways to preserve cross-system identity are:

  1. Copy/replicate key_ids.sqlite (including rowids) and keep descriptor rules identical. This preserves existing numeric keys but requires coordinated allocation for new ones.
  2. Export an explicit mapping of (namespace, adapter key) -> Spark key, then import rows with their original rowids before either system allocates new securities.
  3. Treat a stable external identity (venue + type + exchange instrument ID, or another canonical instrument identifier) as the cross-system key and carry a translation table to each system's local Spark key.
  4. Introduce a deterministic ID scheme only as a versioned migration. A hash can be computed independently, but collision handling, descriptor canonicalization, aliases/renames, the existing 32-bit ABI, and compatibility with orders/recordings/databases all need explicit design.

Copying only securities.sqlite is insufficient as an allocator strategy. It contains security blobs keyed by already-assigned IDs, but key_ids.sqlite is the mapping used to find or allocate those IDs. Conversely, copying only key_ids.sqlite preserves the numbers but not all contract metadata.

Other assignment paths in sparkd

The two main pipelines reduce to four ways a producer handle can acquire a key:

  1. Already keyed input. MDProducer::add_security() sees a non-zero FixedSecurity::key and calls set_key() immediately. Generalized feeds, Sunday preload, relay playback, and other replay paths use this mechanism.
  2. Recorder-backed live input. The input key is zero and the producer owns a SecurityIDRecorder; ensure_key() queues asynchronous assignment. CME MDP3 and legacy OnixS ICE use this mechanism. ICE uses marketId as numeric_id and otherwise follows the same recorder/index design.
  3. Service assignment. SyntheticSecurityService calls SecurityIDRecorder::assign_key() directly on a background dispatcher. The callback receives a copied api::Security with its top-level id set, then publishes it to synthetic subscribers. It uses the same key_ids.sqlite rowid allocator.
  4. Unkeyed input with no recorder. The entry is allocated with key zero and never becomes public. This is not a valid production assignment path, but it is the failure mode if a generalized source omits its ID.

Relay and replay deserve care when reasoning about persistence:

  • The generalized MDReplayer forwards the recorded domain security ID and emits later books/statistics with the recorded sec_key, so the listener's current use-given mode preserves the recording's numeric identity.
  • The legacy relay producer converts a recorded api::Security with set_security_key=true, so its shared-memory publication also preserves the embedded key.
  • The legacy relay recorder additionally calls bulk_insert(..., "REPLAY: ") for logical mappings. Those rows use a REPLAY: namespace prefix; this is a separate mapping population step, not a deterministic derivation of the key.

Operational implications

  • Back up key_ids.sqlite and securities.sqlite together if numeric ID and definition continuity matter.
  • Do not delete/rebuild key_ids.sqlite on one host and assume IDs will match another host, even when all instruments are identical.
  • Before moving a feed between adapters, compare the exact name each adapter supplies to IDGenerator. Matching venue/type is insufficient.
  • Cache generated IDs in the source after definition processing. Avoid calling IDGenerator from book/trade hot paths; the Python bridge currently violates this guideline.
  • Keep generated rowids below the 32-bit shared-memory ceiling until the security_key ABI is widened or explicit bounds checking is added to the generalized path.
  • Ensure spread legs are announced before generalized spread parents.
  • Treat synchronous=0 as a durability decision. A copied DB should be taken with SQLite-aware backup/checkpoint handling rather than by casually copying active WAL files.

Useful inspection queries (run against the configured DB path) are:

SELECT rowid AS spark_key, ns, key
FROM store
ORDER BY rowid;

SELECT rowid AS spark_key, ns, key
FROM store
WHERE ns = 'GLBX.FUTURE' AND key = '<exact-adapter-key>';

Source map