Add: simple doc to expplain security handling in sparkd
This commit is contained in:
1 parent
2853af9e0f
commit
a4ff7743da
1 file changed
+942
@@ -0,0 +1,942 @@
|
||||
# Security ID handling in `sparkd`
|
||||
|
||||
This document traces how security identifiers are created, translated, persisted,
|
||||
indexed, and published by the `sparkd` market-data paths. It focuses on the
|
||||
legacy CME MDP3/OnixS pipeline and the newer generalized market-data pipeline.
|
||||
|
||||
## Executive summary
|
||||
|
||||
`sparkd` uses two identifiers that must not be confused:
|
||||
|
||||
| Identifier | Meaning | Typical field | Lifetime |
|
||||
|---|---|---|---|
|
||||
| Exchange/feed ID | The venue's identifier used to route incoming definitions, books, and trades | `FixedSecurity::numeric_id`, `ExchangeKey`, or `domain::Security::security_common.exchange_sec_id` | Feed/session-specific unless the source guarantees stability |
|
||||
| Spark security key | Spark's internal non-zero identifier used in shared memory, APIs, orders, positions, and downstream services | `FixedSecurity::key`, `api::Security::id()`, `SecurityKey`; historically also called `security_id` | Intended to be stable across restarts |
|
||||
|
||||
`invalid_security_key` is `0`; a security with key `0` is not yet publicly
|
||||
discoverable through the shared-memory consumer path.
|
||||
|
||||
The two principal paths differ:
|
||||
|
||||
- **MDP3/OnixS:** the definition callback sets only the CME ID in
|
||||
`numeric_id`. `MDProducer::add_security()` allocates the shared-memory entry
|
||||
with key `0`. `SecurityIDRecorder`, driven by a 200 ms dispatcher timer,
|
||||
resolves or inserts a stable key in `key_ids.sqlite`; its callback finally
|
||||
calls `MDProducerHandle::set_key()`.
|
||||
- **Generalized/newer pipeline:** `GeneralMarketDataListener::on_securities()`
|
||||
currently uses the ID already present on the incoming domain security
|
||||
(`use_given_sec_key == true`). Live adapters normally obtain that ID
|
||||
synchronously from `IDGenerator` first. `_copy_security_common()` then copies
|
||||
it to both `FixedSecurity::key` and `numeric_id`, so
|
||||
`MDProducer::add_security()` publishes it synchronously. Only the listener's
|
||||
own fallback `IDGenerator` call is bypassed by the current constant.
|
||||
|
||||
## End-to-end diagrams
|
||||
|
||||
### CME MDP3 / OnixS
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant FE as OnixS feed-engine thread
|
||||
participant L as MDP3Listener
|
||||
participant P as MDProducer / shared memory
|
||||
participant Q as SecurityIDRecorder queues
|
||||
participant D as sparkd dispatcher thread
|
||||
participant DB as key_ids.sqlite
|
||||
|
||||
FE->>L: instrument definition (CME SecurityID)
|
||||
L->>L: set_security_common() - numeric_id = CME SecurityID, key remains 0
|
||||
L->>P: add_security(FixedSecurity)
|
||||
P->>P: allocate hot + cold entries with key 0
|
||||
P->>Q: ensure_key(handle)
|
||||
L->>L: index[CME SecurityID] = handle
|
||||
Note over FE,L: Books can route by CME ID while the Spark key is pending
|
||||
D->>Q: every 200 ms - work_spreads(), work_handles(), work_db()
|
||||
Q->>DB: find (namespace, descriptor), or insert a new row
|
||||
DB-->>Q: SQLite rowid = Spark security key
|
||||
Q->>P: handle.set_key(rowid)
|
||||
P->>P: hot.key first, then cold security.key, then increment security_count
|
||||
Note over P: The security is now public to shared-memory consumers
|
||||
```
|
||||
|
||||
### Generalized/newer pipeline (current behavior)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant A as Venue adapter / plugin
|
||||
participant DB as IDGenerator / key_ids.sqlite
|
||||
participant G as GeneralMarketDataListener
|
||||
participant I as GeneralSecurityIndex
|
||||
participant P as MDProducer / shared memory
|
||||
|
||||
A->>DB: get_or_add(venue, type, adapter key)
|
||||
DB-->>A: SQLite rowid as domain SecurityID
|
||||
A->>G: on_securities(domain Security list with ID)
|
||||
G->>G: _find_sec() - use incoming security_common.id
|
||||
G->>G: _copy_security_common() - key = id, numeric_id = id
|
||||
G->>P: add_security(FixedSecurity with non-zero key)
|
||||
P->>P: set_key() during add_security()
|
||||
G->>I: index[id] = handle
|
||||
Note over G,P: No asynchronous ID assignment in the current configuration
|
||||
```
|
||||
|
||||
## Data-flow charts
|
||||
|
||||
The sequence diagrams above emphasize timing and thread transitions. These
|
||||
flowcharts emphasize which identifier is carried through each representation
|
||||
and where data is retained.
|
||||
|
||||
### MDP3 / legacy data flow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
CME["CME MDP3 packets<br/>CME SecurityID"] --> ONIXS["OnixS feed engine<br/>decode definition and updates"]
|
||||
|
||||
ONIXS --> DEF["MDP3Listener definition callback"]
|
||||
DEF --> FIXED["FixedSecurity<br/>numeric_id = CME SecurityID<br/>key = 0"]
|
||||
|
||||
FIXED --> PRODUCER["MDProducer::add_security"]
|
||||
PRODUCER --> PENDING["Hot and cold shared memory<br/>allocated but not public<br/>key = 0"]
|
||||
PRODUCER --> QUEUE["SecurityIDRecorder<br/>pending_handle queue"]
|
||||
|
||||
FIXED --> CMEINDEX["CMESecurityIndex<br/>CME SecurityID to handle"]
|
||||
|
||||
QUEUE --> TIMER["Dispatcher timer<br/>every 200 ms"]
|
||||
TIMER --> LEGS{"Spread legs<br/>all resolved?"}
|
||||
LEGS -- No --> UNRESOLVED["unresolved_legs<br/>exchange IDs awaiting keys"]
|
||||
UNRESOLVED --> CMEINDEX
|
||||
CMEINDEX --> LEGS
|
||||
LEGS -- Yes or outright --> LOGICAL["Logical identity<br/>venue + type + security_desc"]
|
||||
|
||||
LOGICAL --> SELECT["SQLite SELECT<br/>key_ids.sqlite"]
|
||||
SELECT -- Existing row --> ROWID["SQLite rowid<br/>Spark security key"]
|
||||
SELECT -- Missing row --> INSERT["Batched SQLite INSERT"]
|
||||
INSERT --> ROWID
|
||||
|
||||
ROWID --> SETKEY["MDProducerHandle::set_key"]
|
||||
SETKEY --> HOT["Write hot-entry key"]
|
||||
HOT --> COLD["Write cold security key"]
|
||||
COLD --> COUNT["Increment security_count"]
|
||||
COUNT --> PUBLIC["Public security<br/>visible to MDConsumer"]
|
||||
|
||||
ONIXS --> UPDATE["Book, trade, status, or RFQ<br/>carrying CME SecurityID"]
|
||||
UPDATE --> CMEINDEX
|
||||
CMEINDEX --> HANDLE["MDProducerHandle"]
|
||||
HANDLE --> LIVE["Update live shared-memory data"]
|
||||
PUBLIC --> LIVE
|
||||
```
|
||||
|
||||
The CME index is usable before the Spark key is assigned, so feed updates can
|
||||
reach the internal handle while the security is still hidden from consumers.
|
||||
The persistent rowid crosses the publication boundary only at `set_key()`.
|
||||
|
||||
### Generalized/newer data flow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
FEED["Venue feed, Databento,<br/>Python plugin, or replay"] --> ADAPTER["Source adapter<br/>build domain::Security"]
|
||||
|
||||
ADAPTER --> IDTYPE{"Live definition or<br/>recorded/pre-keyed input?"}
|
||||
IDTYPE -- Live --> IDINPUT["Logical identity<br/>venue + type + adapter-chosen name"]
|
||||
IDINPUT --> MUTEX["IDGenerator global mutex"]
|
||||
MUTEX --> SELECT["Synchronous SQLite SELECT<br/>key_ids.sqlite"]
|
||||
SELECT -- Existing row --> ROWID["SQLite rowid"]
|
||||
SELECT -- Missing row --> INSERT["Synchronous SQLite INSERT<br/>autocommit"]
|
||||
INSERT --> ROWID
|
||||
ROWID --> DOMAINID["domain::SecurityID<br/>used as Spark ID"]
|
||||
IDTYPE -- Recorded or pre-keyed --> DOMAINID
|
||||
|
||||
DOMAINID --> DOMAINSEC["SecurityCommon::id = Spark ID"]
|
||||
DOMAINSEC --> LISTENER["GeneralMarketDataListener::on_securities"]
|
||||
LISTENER --> CCMAP["cc_sec_id_map<br/>incoming ID to Spark ID"]
|
||||
LISTENER --> CONVERT["_copy_security_common<br/>FixedSecurity.key = ID<br/>numeric_id = ID"]
|
||||
|
||||
CONVERT --> PRODUCER["MDProducer::add_security"]
|
||||
PRODUCER --> SETKEY["Immediate set_key<br/>no recorder timer"]
|
||||
SETKEY --> SHM["Hot and cold shared memory<br/>immediately public"]
|
||||
CONVERT --> GENERALINDEX["GeneralSecurityIndex<br/>ID to handle"]
|
||||
|
||||
FEED --> EVENT["Book, trade, or statistics<br/>with exchange/source identity"]
|
||||
EVENT --> EVENTMAP["Adapter map<br/>source identity to domain::SecurityID"]
|
||||
EVENTMAP --> EVENTID["Event carries domain::SecurityID"]
|
||||
EVENT --> PYHOT["Python bridge only<br/>synchronous IDGenerator lookup"]
|
||||
PYHOT --> EVENTID
|
||||
PYHOT -. uses same mutex and SQLite path .-> MUTEX
|
||||
EVENTID --> CCMAP
|
||||
CCMAP --> GENERALINDEX
|
||||
GENERALINDEX --> HANDLE["MDProducerHandle"]
|
||||
HANDLE --> SHMUPDATE["Update book, trade,<br/>statistics, or status"]
|
||||
SHM --> SHMUPDATE
|
||||
|
||||
```
|
||||
|
||||
In the normal newer live path, persistence occurs before the definition enters
|
||||
the generalized listener. The DB call therefore adds synchronous adapter-side
|
||||
latency, but publication needs no later timer. Most adapters use an in-memory
|
||||
map for subsequent events; the dotted Python path is the documented exception.
|
||||
|
||||
## The publication boundary
|
||||
|
||||
`MDProducerHandle::set_key()` is the definitive publication point for the
|
||||
legacy shared-memory representation. Its write ordering is intentional:
|
||||
|
||||
1. Write the key into the hot book entry (`HotEntry` or `HotDepthEntry`).
|
||||
2. Write the key into the cold security entry.
|
||||
3. Increment the cold header's public `security_count`.
|
||||
|
||||
The cold security key is written last because consumers treat a non-zero cold
|
||||
key as visibility. At that moment the corresponding hot entry must already be
|
||||
addressable. `set_key()` only permits the transition from `0` to one key; it is
|
||||
not a key-update mechanism.
|
||||
|
||||
## Identifier representations
|
||||
|
||||
The names are historically inconsistent. In particular, many protobuf fields
|
||||
named `security_id` contain the **Spark key**, not the exchange's ID.
|
||||
|
||||
| Layer | Exchange/feed identifier | Spark identifier |
|
||||
|---|---|---|
|
||||
| MDP3 wire/OnixS | `securityId()` | not present |
|
||||
| Legacy fixed security | `FixedSecurity::numeric_id` (`uint64_t`) | `FixedSecurity::key` / `security_key` (`uint32_t`) |
|
||||
| Legacy API security | venue-specific `exchange_id` | top-level `Security.id()` |
|
||||
| New domain model | `SecurityCommon::exchange_sec_id` (string) is descriptive; `SecurityCommon::id` is also used to correlate source events | `domain::SecurityID` (`uint64_t`), currently copied into the Spark key |
|
||||
| Shared-memory hot book | none | `HotEntry::key` or `HotDepthEntry::key` |
|
||||
| Shared-memory cold metadata | `FixedSecurity::numeric_id` | `FixedSecurity::key` |
|
||||
| CME runtime lookup | `CMESecurityIndex` key | `SecurityData::handle.key()` |
|
||||
| General runtime lookup | incoming/canonical `domain::SecurityID` | `GeneralSecurityIndex` key and `handle.key()` (currently the same value) |
|
||||
|
||||
The generalized model declares `domain::SecurityID` as 64-bit, but the legacy
|
||||
shared-memory `security_key` remains 32-bit. `_copy_security_common()` assigns a
|
||||
64-bit value to both `numeric_id` and the 32-bit `key`. This is safe only while
|
||||
allocated IDs fit in 32 bits; unlike the legacy lookup path, this assignment has
|
||||
no explicit checked conversion.
|
||||
|
||||
## Which parts use exchange IDs, descriptors, or Spark security IDs?
|
||||
|
||||
The three identifier classes serve different jobs:
|
||||
|
||||
- **Exchange/external ID:** routes data inside a venue or vendor feed adapter.
|
||||
- **Descriptor/natural key:** recognizes a definition logically at the
|
||||
persistence, discovery, or human-facing boundary.
|
||||
- **Spark security ID/key:** joins the instrument to books, orders, fills,
|
||||
positions, and services after publication.
|
||||
|
||||
The core downstream system does not repeatedly identify a security from its
|
||||
description. The description is primarily used to obtain a Spark key; after
|
||||
that boundary, most joins use the numeric Spark key.
|
||||
|
||||
### Component-level usage matrix
|
||||
|
||||
| Component or data path | Exchange/external ID | Descriptor or name | Spark security ID/key |
|
||||
|---|---|---|---|
|
||||
| CME MDP3 definition decoder | Reads `securityId()` and stores it in `FixedSecurity::numeric_id` | Copies definition symbol/tag 55 into `security_desc` | Leaves `FixedSecurity::key` at `0` initially |
|
||||
| `CMESecurityIndex` | Primary map key; CME ID resolves to a handle | Not used for update routing | Returns `handle.key()` when the Spark key has been assigned |
|
||||
| MDP3 books, trades, status, statistics, and RFQs | Incoming CME ID selects the handle through `CMESecurityIndex` | Not used to locate each update | The selected handle writes keyed shared-memory/trade data after publication |
|
||||
| MDP3 spread definitions | Each leg initially contains `legSecurityId()` as `exchange_key` | Parent `security_desc` identifies the spread's DB row | Each leg's Spark key and then the parent Spark key must be resolved before publication |
|
||||
| Legacy ICE OnixS | Uses ICE `marketId` as `numeric_id` and index key | ICE security description becomes the logical lookup name | Uses the same recorder-assigned Spark key pattern as MDP3 |
|
||||
| `SecurityIDRecorder` | Does not allocate from the external ID | Uses `(venue, security type, securities::descriptor())` to find/create a row | Treats the SQLite rowid as the resulting Spark key |
|
||||
| Native generalized adapters | Use exchange strings/numeric IDs to parse definitions and correlate updates | Usually pass a description or exchange instrument name to `IDGenerator` | Put the returned ID into `SecurityCommon::id` and later events |
|
||||
| Databento adapter | Uses `msg.hd.instrument_id` for feed correlation and its in-memory map | Uses the decimal instrument ID string—not the human description—as the `IDGenerator` name | Caches `instrument_id -> domain::SecurityID` and places that ID on books/trades |
|
||||
| Python bridge | Reads `exchange_security_id` from definitions, books, and trades | Uses that external-ID string as the `IDGenerator` name | Resolves it to `domain::SecurityID`; currently performs this lookup for every event |
|
||||
| `IDGenerator` | Does not interpret whether `name` came from an exchange ID | Uses exactly `(venue, type, caller-supplied name)` | Returns the SQLite rowid as a 64-bit ID |
|
||||
| `GeneralMarketDataListener` | Does not route on `exchange_sec_id` directly | Copies `sec.desc()` into fixed metadata; its disabled fallback allocator would use `sec.desc()` | Uses incoming `SecurityCommon::id`, maps events through `cc_sec_id_map`, and copies the value into both `key` and `numeric_id` |
|
||||
| `GeneralSecurityIndex` | No separate external-ID concept in the current use-given path | Not used | Maps the generalized ID/Spark key to `MDProducerHandle` |
|
||||
| `MDProducer` / shared memory | Retains `numeric_id`; in legacy feeds this is the exchange ID, but in the generalized path it is currently the Spark ID | Stores `security_desc` in the cold definition | Uses the key in hot/cold entries; non-zero key is the visibility boundary |
|
||||
| `MDConsumer` | Not normally used to find live books | Can enumerate/filter converted security definitions | Finds handles and book data by `security_key` |
|
||||
| `SecurityCache` / `securities.sqlite` | Preserves venue-specific exchange data inside the serialized security blob | Maintains descriptor/name lookup indexes and supports venue/type/name queries | Primary persisted definition key is `api::Security.id()` |
|
||||
| MBO processor and MBO service | May retain exchange **order** IDs, which are separate from security exchange IDs | Not used to find the instrument | Indexes snapshots and requests by `SecurityKey`; MBO protobuf `security_id` means Spark key |
|
||||
| Order manager and algorithms | Execution gateways eventually use venue-specific fields from the resolved security to encode orders | Descriptions appear in logs and errors, not as the primary join | Order definitions, subscriptions, book events, native orders, spread legs, and fills use `security_key` / `Security.id()` |
|
||||
| Position, PNL, and risk services | Not used as their position join | May resolve metadata for display/reporting | Positions, fills, overrides, and PNL handles are keyed by Spark security ID |
|
||||
| Relay and market-data recordings | Serialized security definitions may preserve venue exchange IDs | Preserve full descriptors in recorded definitions | Books, updates, trades, and replay correlation preserve the recorded Spark key |
|
||||
| Synthetic security service | No venue exchange ID; synthetics use `VENUE_BTS` | Synthetic name is the recorder's logical descriptor | Recorder assigns the SQLite rowid and synthetic metadata is keyed by it |
|
||||
|
||||
### Exchange/external IDs: where they stop
|
||||
|
||||
In MDP3 the separation is explicit:
|
||||
|
||||
```text
|
||||
CME SecurityID
|
||||
-> FixedSecurity.numeric_id
|
||||
-> CMESecurityIndex lookup
|
||||
-> MDProducerHandle
|
||||
-> Spark security_key for downstream publication
|
||||
```
|
||||
|
||||
The external CME ID continues to route feed messages but does not become the
|
||||
order/position key. This permits the same downstream Spark key to survive a
|
||||
restart as long as the logical descriptor mapping remains in
|
||||
`key_ids.sqlite`, even if the feed must rediscover the CME routing ID.
|
||||
|
||||
The generalized path collapses more of these roles:
|
||||
|
||||
```text
|
||||
exchange/vendor identity
|
||||
-> adapter calls IDGenerator
|
||||
-> domain::SecurityID
|
||||
-> FixedSecurity.key and FixedSecurity.numeric_id
|
||||
-> GeneralSecurityIndex and downstream Spark key
|
||||
```
|
||||
|
||||
`SecurityCommon::exchange_sec_id` may still contain the original textual
|
||||
external identifier, but `GeneralMarketDataListener` does not use it to route
|
||||
events and does not put it in `numeric_id`. The adapter must translate later
|
||||
events to `domain::SecurityID` before invoking the listener.
|
||||
|
||||
### Descriptor/name users
|
||||
|
||||
Descriptors are used in three main places:
|
||||
|
||||
1. **Key allocation:** `SecurityIDRecorder` uses the legacy venue-specific
|
||||
descriptor, while `IDGenerator` uses its caller-provided `name`.
|
||||
2. **Definition lookup and discovery:** `SecurityCache` supports
|
||||
venue/type/name lookup; symbol filters and administrative tools use security
|
||||
metadata to select or display instruments.
|
||||
3. **Diagnostics:** logs include symbols and descriptions so humans can
|
||||
understand a numeric key.
|
||||
|
||||
Descriptions generally do not drive the high-rate book, trade, order, fill, or
|
||||
position paths. Those paths use numeric IDs after definition resolution.
|
||||
|
||||
It is important that “descriptor” does not mean one universal canonical field:
|
||||
|
||||
- Legacy CME uses Globex `security_desc` from tag 55.
|
||||
- Many generalized adapters use `sec.desc()` or an exchange instrument name.
|
||||
- Databento uses its numeric instrument ID converted to a string.
|
||||
- Python uses `exchange_security_id`.
|
||||
|
||||
Therefore, two adapters may describe the same economic contract but request
|
||||
different logical rows and receive different Spark IDs.
|
||||
|
||||
### Spark security-ID users
|
||||
|
||||
Once a security is published, its Spark key is the dominant system identifier:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
KEY["Spark security key"] --> SHM["Hot/cold shared memory"]
|
||||
KEY --> MDC["MDConsumer and book subscriptions"]
|
||||
KEY --> MBO["MBO snapshots and queries"]
|
||||
KEY --> OMS["Orders, algorithms, and spread legs"]
|
||||
KEY --> FILLS["Fills and execution records"]
|
||||
KEY --> POS["Positions, PNL, and risk"]
|
||||
KEY --> API["Services and client APIs"]
|
||||
KEY --> REC["Relay and market-data recordings"]
|
||||
KEY --> SECDB["Serialized security definitions"]
|
||||
```
|
||||
|
||||
Fields named `security_id`, `security_key`, or top-level `Security.id()` in
|
||||
these downstream areas almost always mean the Spark key. They should not be
|
||||
compared directly with CME `SecurityID`, Databento `instrument_id`, or another
|
||||
venue/vendor identifier without going through the relevant adapter/index
|
||||
mapping.
|
||||
|
||||
## Persistence and allocation
|
||||
|
||||
### The authoritative key mapping: `key_ids.sqlite`
|
||||
|
||||
Both allocators use `SqliteStore("key_ids")`. With the normal configuration the
|
||||
file is:
|
||||
|
||||
```text
|
||||
<SystemConfig.data_path>/databases/key_ids/key_ids.sqlite
|
||||
```
|
||||
|
||||
The database contains one table and two indexes:
|
||||
|
||||
```sql
|
||||
CREATE TABLE store (ns TEXT, key TEXT, value TEXT);
|
||||
CREATE UNIQUE INDEX key_index ON store(ns, key);
|
||||
CREATE INDEX ns_index ON store(ns);
|
||||
```
|
||||
|
||||
The SQLite `rowid` is the generated Spark key. `ns` is normally
|
||||
`<VENUE>.<SECURITY_TYPE>` (for example `GLBX.FUTURE`), and `key` is the security
|
||||
descriptor chosen by the caller. `value` is normally empty. Therefore ID
|
||||
stability depends on all three of these remaining stable:
|
||||
|
||||
- the configured `data_path` and preservation of this database;
|
||||
- the venue/type namespace;
|
||||
- the descriptor supplied by the feed adapter.
|
||||
|
||||
The two allocator implementations share the file and convention:
|
||||
|
||||
- `SecurityIDRecorder` (legacy): derives the namespace from legacy API enums and
|
||||
uses `securities::descriptor()`. For CME this is the Globex
|
||||
`security_desc`, populated from the definition's symbol/tag 55.
|
||||
- `IDGenerator` (newer): derives the namespace from new domain enums and uses
|
||||
the caller's `name`. Most adapters pass `sec.desc()` or an exchange instrument
|
||||
identifier; Databento deliberately passes the decimal instrument ID, and the
|
||||
Python bridge uses the exchange security ID.
|
||||
|
||||
### Logical-row examples
|
||||
|
||||
The following names are illustrative. Production uses the exact string emitted
|
||||
by the selected feed/adapter, including its case, spacing, and weekly-series
|
||||
notation.
|
||||
|
||||
#### Legacy `SecurityIDRecorder`
|
||||
|
||||
For CME, `set_security_common()` copies the MDP3 definition's symbol/tag 55 into
|
||||
Globex `security_desc`. After converting the fixed security to
|
||||
`api::Security`, `securities::descriptor()` returns that same
|
||||
`glbx.security_desc()` string.
|
||||
|
||||
| Instrument | Legacy enums | Example descriptor from `security_desc` | Row searched/inserted in `store` |
|
||||
|---|---|---|---|
|
||||
| Future | `VENUE_GLBX`, `SECURITYTYPE_FUTURE` | `ESZ6` | `ns='GLBX.FUTURE', key='ESZ6'` |
|
||||
| Standard option | `VENUE_GLBX`, `SECURITYTYPE_OPTION` | `ESZ6 C6500` | `ns='GLBX.OPTION', key='ESZ6 C6500'` |
|
||||
| Weekly option | `VENUE_GLBX`, `SECURITYTYPE_OPTION` | `ESZ6-W1 C6500` | `ns='GLBX.OPTION', key='ESZ6-W1 C6500'` |
|
||||
| Spot from a legacy-compatible venue path | `VENUE_COINBASE`, `SECURITYTYPE_SPOT` | `BTC-USD` | `ns='COINBASE.SPOT', key='BTC-USD'` |
|
||||
|
||||
A weekly option does not get a special namespace. Standard and weekly options
|
||||
are both `SECURITYTYPE_OPTION`, so both live in `GLBX.OPTION`; the exact
|
||||
`security_desc` distinguishes the weekly series. For example, if the database
|
||||
assigned rowids `42001` through `42004` to the rows above, those integers—not
|
||||
the descriptor strings—would be the Spark security keys.
|
||||
|
||||
Conceptually, legacy allocation is:
|
||||
|
||||
```text
|
||||
future: lookup_or_insert("GLBX.FUTURE", "ESZ6")
|
||||
option: lookup_or_insert("GLBX.OPTION", "ESZ6 C6500")
|
||||
weekly option: lookup_or_insert("GLBX.OPTION", "ESZ6-W1 C6500")
|
||||
spot: lookup_or_insert("COINBASE.SPOT", "BTC-USD")
|
||||
```
|
||||
|
||||
#### Newer `IDGenerator`
|
||||
|
||||
The namespace result is similar, but the third argument is chosen by the
|
||||
adapter. Examples using a description-oriented adapter are:
|
||||
|
||||
```cpp
|
||||
get_or_add(VENUE_GLBX, SECURITY_TYPE_FUTURE, "ESZ6");
|
||||
// store: ns="GLBX.FUTURE", key="ESZ6"
|
||||
|
||||
get_or_add(VENUE_GLBX, SECURITY_TYPE_OPTION, "ESZ6 C6500");
|
||||
// store: ns="GLBX.OPTION", key="ESZ6 C6500"
|
||||
|
||||
get_or_add(VENUE_GLBX, SECURITY_TYPE_OPTION, "ESZ6-W1 C6500");
|
||||
// store: ns="GLBX.OPTION", key="ESZ6-W1 C6500"
|
||||
|
||||
get_or_add(VENUE_COINBASE, SECURITY_TYPE_SPOT, "BTC-USD");
|
||||
// store: ns="COINBASE.SPOT", key="BTC-USD"
|
||||
```
|
||||
|
||||
Databento intentionally produces a different logical row even when the human
|
||||
description is `ESZ6`. If its instrument ID were `12345678`, it would call the
|
||||
equivalent of:
|
||||
|
||||
```cpp
|
||||
get_or_add(VENUE_GLBX, SECURITY_TYPE_FUTURE, "12345678");
|
||||
// store: ns="GLBX.FUTURE", key="12345678"
|
||||
```
|
||||
|
||||
Likewise, the Python bridge uses `exchange_security_id`. A spot definition with
|
||||
`exchange_security_id="BTC-USD"` produces the `COINBASE.SPOT / BTC-USD` row,
|
||||
while a provider that sends `exchange_security_id="BTCUSD"` produces a
|
||||
different row and therefore a different Spark key.
|
||||
|
||||
These examples show why the tuple is only logical when the adapter's naming
|
||||
contract is also known. `GLBX.FUTURE / ESZ6` and `GLBX.FUTURE / 12345678` may
|
||||
describe the same contract to a human but are distinct rows to `key_ids.sqlite`.
|
||||
|
||||
Both enum-prefix removals currently produce the same namespace spelling (for
|
||||
example, `GLBX.OPTION`), so both mechanisms allocate from one global SQLite
|
||||
rowid space. Live rows cannot share a rowid. The table does not declare
|
||||
`AUTOINCREMENT`, however, so deleting rows—especially the highest row—can allow
|
||||
SQLite to reuse a number later. Security-key rows should therefore be treated
|
||||
as append-only operational data.
|
||||
|
||||
`SqliteStore` uses WAL mode and `PRAGMA synchronous = 0`. This favors startup
|
||||
and feed latency over maximum durability against a host/power failure. The
|
||||
unique `(ns, key)` index prevents two different rows for the same logical
|
||||
security.
|
||||
|
||||
### Other places the key is stored
|
||||
|
||||
| Location | Purpose | Authoritative for allocation? |
|
||||
|---|---|---|
|
||||
| Hot and cold shared-memory segments | Live publication and fast consumer lookup | No; rebuilt at process startup |
|
||||
| `CMESecurityIndex` / `GeneralSecurityIndex` and adapter maps | In-process routing from an incoming ID to a producer handle | No; in-memory only |
|
||||
| `<data_path>/databases/md/securities.sqlite` | Serialized legacy `api::Security` definitions keyed by Spark key; used by security cache and MDP3 Sunday preload | No; it stores metadata after a key exists |
|
||||
| Relay/market-data recordings | Preserve the key embedded in recorded securities, books, and trades for deterministic replay | No; replay normally reuses the recorded value |
|
||||
| Orders, fills, positions, APIs | Reference a security by the published Spark key | No |
|
||||
|
||||
The `SecurityIDRecorder::work_db()` comment says a published security is also
|
||||
recorded in the security database, but that function does **not** write
|
||||
`securities.sqlite` in the current code. Full security definitions are persisted
|
||||
separately by `SecurityCache::flush_to_disk()` through
|
||||
`db::insert_or_replace()`.
|
||||
|
||||
## CME MDP3 / OnixS pipeline in detail
|
||||
|
||||
### 1. Construction and thread setup
|
||||
|
||||
`MDP3Main::PImpl` owns one `CMESecurityIndex`, one `SecurityIDRecorder`, and one
|
||||
`MDProducer`. It constructs the recorder with:
|
||||
|
||||
- a lookup callback that maps `ExchangeKey` through the CME index to the
|
||||
handle's current Spark key; and
|
||||
- `Dispatcher::current()`, which drives a 200 ms recorder timer.
|
||||
|
||||
In the current `MarketDataGateway` route, `OnixsGlbxSubscription` creates a
|
||||
dedicated `dispatcher_thread`, constructs and starts `MDP3Main` on that thread,
|
||||
then calls that thread's dispatcher `run()`. The older `md run onixs` command
|
||||
uses its command/dispatcher thread similarly.
|
||||
|
||||
The OnixS feed engine has its own configured thread pool. Instrument and market
|
||||
data callbacks therefore originate on OnixS feed-engine threads, while recorder
|
||||
database work and final key assignment execute on the MDP3 dispatcher thread.
|
||||
|
||||
| MDP3 stage | Executing thread | DB call? |
|
||||
|---|---|---|
|
||||
| Decode definition and set CME `numeric_id` | OnixS feed-engine callback thread | No |
|
||||
| Allocate hot/cold entry and queue handle | Same OnixS callback thread | No |
|
||||
| Add CME-ID index entry | Same OnixS callback thread | No |
|
||||
| Resolve spread legs | MDP3 dispatcher thread, on 200 ms timer | No |
|
||||
| Select/insert Spark key | MDP3 dispatcher thread, on 200 ms timer | Yes |
|
||||
| `handle.set_key()` and public visibility | MDP3 dispatcher thread, in DB callback | No additional call |
|
||||
|
||||
### 2. Definition callback: exchange ID is set, Spark key is not
|
||||
|
||||
For futures, options, and spreads, `MDP3Listener::set_security_common()` does:
|
||||
|
||||
```cpp
|
||||
fsec.numeric_id = def.securityId();
|
||||
```
|
||||
|
||||
It also copies symbol/descriptor/venue and pricing fields. A newly constructed
|
||||
`FixedSecurity` already has `key == invalid_security_key` (`0`), and this helper
|
||||
does not change it.
|
||||
|
||||
For a spread, every leg starts with:
|
||||
|
||||
```cpp
|
||||
leg.exchange_key = ExchangeKey(definition_leg.legSecurityId());
|
||||
leg.key = invalid_security_key;
|
||||
```
|
||||
|
||||
Thus both the parent and legs initially carry exchange-routing identity, but no
|
||||
Spark identity.
|
||||
|
||||
### 3. Shared-memory allocation and exchange indexing
|
||||
|
||||
`MDP3Listener::add_security()` performs these operations in order:
|
||||
|
||||
1. `_producer.add_security(security)` allocates a hot entry and a variable-sized
|
||||
cold entry and copies the security into the cold segment.
|
||||
2. Because the key is `0`, `MDProducer` queues the handle through
|
||||
`SecurityIDRecorder::ensure_key()`.
|
||||
3. `_index.add_security(security.numeric_id, handle)` immediately creates the
|
||||
CME-ID-to-handle mapping.
|
||||
4. The initial exchange status is written through the handle.
|
||||
|
||||
This ordering means incoming books can already be routed by CME ID while the
|
||||
Spark key is pending. Shared-memory consumers cannot discover the security yet.
|
||||
Also, `MDProducerHandle::add_trade()` only appends to the public trade queue
|
||||
when `key() != 0`, so a trade received in this short window updates the cold
|
||||
last-trade state but is not added to the keyed trade log.
|
||||
|
||||
### 4. Recorder tick: spreads, handles, then database
|
||||
|
||||
Every 200 ms the recorder calls, in this exact order:
|
||||
|
||||
1. `work_spreads()` tries to fill previously unresolved spread leg keys by
|
||||
asking `CMESecurityIndex` for each leg exchange ID.
|
||||
2. `work_handles()` drains new handles. Outrights are converted to an API
|
||||
security and queued for DB lookup immediately. A spread without all leg keys
|
||||
is retained in `pending_handle` and its leg pointers are registered in
|
||||
`unresolved_legs`.
|
||||
3. `work_db()` resolves queued `(venue, type, descriptor)` requests.
|
||||
|
||||
Because spread work precedes new-handle work, a newly seen unresolved spread
|
||||
normally waits until at least the next timer tick to resolve legs. It waits
|
||||
longer if any outright leg has not yet received its Spark key. A spread's own
|
||||
key is not requested until every leg key is non-zero.
|
||||
|
||||
### 5. Lookup or generation
|
||||
|
||||
`work_db()` first queries `(ns, key)`:
|
||||
|
||||
- Existing row: use its `rowid` as the key and invoke the callback.
|
||||
- Missing row: batch it into `insert_many_resolve_ids()`. SQLite assigns a new
|
||||
rowid inside a transaction, the row object receives that ID, and the callback
|
||||
runs after the transaction completes.
|
||||
|
||||
New inserts are throttled to roughly 200 per timer pass (the current `> 200`
|
||||
check can admit 201) to avoid delaying heartbeats during startup. The queues are
|
||||
TBB concurrent queues, but all SQLite access is serialized onto the recorder's
|
||||
dispatcher thread. `check_access()` asserts this single-thread rule after the
|
||||
first work pass records the dispatcher thread ID.
|
||||
|
||||
### 6. Final assignment and visibility
|
||||
|
||||
The queued callback captures a copy of `MDProducerHandle` and calls
|
||||
`handle.set_key(id)` on the dispatcher thread. `set_key()` writes hot, then cold,
|
||||
then increments `security_count`. From this point:
|
||||
|
||||
- shared-memory scans report the security;
|
||||
- `CMESecurityIndex::get_key(CME_ID)` returns the Spark key;
|
||||
- trades and RFQs can be emitted with a non-zero key;
|
||||
- resolved spreads can copy this key into their leg structures.
|
||||
|
||||
### 7. Sunday preload is a fast path
|
||||
|
||||
For configured symbols on Sunday, `MDP3Main::prepare_listener()` loads stored
|
||||
definitions from `securities.sqlite`. It converts them with
|
||||
`set_security_key=true`, so both the security key and spread leg keys come from
|
||||
the serialized API security. `MDProducer::add_security()` calls `set_key()`
|
||||
immediately, and no recorder lookup is required for that preloaded handle. The
|
||||
CME exchange ID is still restored into `numeric_id`, allowing the normal CME
|
||||
index to be built.
|
||||
|
||||
## Generalized/newer pipeline in detail
|
||||
|
||||
### 1. Topology and ownership
|
||||
|
||||
`Run::run()` starts a detached thread named `MarketDataGateway`. On that thread,
|
||||
`run_md_gwy()` obtains the process-wide `IDGenerator`, constructs
|
||||
`MarketDataGateway`, and starts the configured providers. For each `(venue,
|
||||
source)` returned by a provider, the gateway creates an `MdSink` containing:
|
||||
|
||||
- a venue-local `GeneralSecurityIndex`;
|
||||
- a default `MDProducer` (not wired to a `SecurityIDRecorder`); and
|
||||
- a `GeneralMarketDataListener` connecting the source to that producer/index.
|
||||
|
||||
The source decides its callback thread. Some sources call the listener while
|
||||
starting; others own network or replay threads. The generalized listener calls
|
||||
the producer inline on that source thread. The process-wide `IDGenerator`
|
||||
serializes its own calls with a mutex, but the listener's transient
|
||||
`cc_sec_id_map` is an ordinary `unordered_map`, so each source/listener pair is
|
||||
expected to serialize security/event callbacks.
|
||||
|
||||
| Generalized stage | Executing thread | DB call? |
|
||||
|---|---|---|
|
||||
| Build/parse domain definition | Source/adaptor thread | No |
|
||||
| `IDGenerator::get_or_add()` | Same source/adaptor thread | Yes, synchronously |
|
||||
| `GeneralMarketDataListener::on_securities()` | Same source callback thread | No in current use-given mode |
|
||||
| Copy ID to `FixedSecurity::key` and `numeric_id` | Same source callback thread | No |
|
||||
| `MDProducer::add_security()` / `set_key()` | Same source callback thread | No additional call |
|
||||
| Add `GeneralSecurityIndex` entry | Same source callback thread | No |
|
||||
|
||||
There are two important compatibility exceptions inside the provider framework:
|
||||
|
||||
- `OnixsGlbxSubscription::start()` ignores the supplied generalized listener
|
||||
and starts `MDP3Main`; CME/OnixS therefore still follows the asynchronous
|
||||
legacy flow documented above.
|
||||
- `EmumbaDeribitSubscription::start()` also ignores that listener and uses
|
||||
`EmMDListener`, which allocates via `IDGenerator` and writes through its own
|
||||
producer/index. Its assignment is still synchronous and pre-populated, so it
|
||||
resembles the newer ID flow even though it does not pass through
|
||||
`GeneralMarketDataListener`.
|
||||
|
||||
### 2. How a new ID is constructed
|
||||
|
||||
For a normal live generalized adapter, ID creation happens **in the adapter,
|
||||
before** `GeneralMarketDataListener::on_securities()`:
|
||||
|
||||
```text
|
||||
logical identity = (domain venue, domain security type, adapter-chosen name)
|
||||
Spark ID = SQLite rowid for that identity in key_ids.sqlite
|
||||
```
|
||||
|
||||
`IDGenerator::get_or_add(venue, type, name)` formats a namespace by stripping
|
||||
the enum prefixes:
|
||||
|
||||
```text
|
||||
VENUE_TAIFEX + SECURITY_TYPE_FUTURE -> TAIFEX.FUTURE
|
||||
```
|
||||
|
||||
It then calls `retrieve_one(ns, name)`. If the row exists, it returns that
|
||||
row's `rowid`. If absent, it inserts `(ns, name, "")`, and SQLite assigns the
|
||||
next rowid. That integer is wrapped in `domain::SecurityID` and placed in
|
||||
`SecurityCommon::id`.
|
||||
|
||||
The third identity component is adapter-specific, and this matters when trying
|
||||
to reproduce a mapping:
|
||||
|
||||
| Source/adaptor family | Typical `name` sent to `IDGenerator` |
|
||||
|---|---|
|
||||
| Binance, Coinbase, Deribit, OKX and similar native/CCAPI paths | Usually `sec.desc()` / exchange instrument name |
|
||||
| Taifex | product/instrument ID string |
|
||||
| Databento | `std::to_string(msg.hd.instrument_id)`, not the human-readable description |
|
||||
| Python bridge | `exchange_security_id` (and it asserts Python supplied ID is zero for definitions) |
|
||||
| Emumba Deribit | extracted Emumba security descriptor |
|
||||
| Replay | no new allocation for a recorded definition; it reuses the recorded `SecurityCommon::id` |
|
||||
|
||||
So “security name” alone is not a universal logical key. Reproduction requires
|
||||
the exact venue enum, exact security-type enum, and the exact adapter-specific
|
||||
name used at insertion time.
|
||||
|
||||
### 3. What `GeneralMarketDataListener` does with that ID
|
||||
|
||||
`on_securities()` first calls `_find_sec(sec, true)`. Its current, compile-time
|
||||
behavior is controlled by:
|
||||
|
||||
```cpp
|
||||
const bool use_given_sec_key = true;
|
||||
```
|
||||
|
||||
Consequently, for a newly seen incoming `domain::SecurityID` it does **not**
|
||||
call the DB. It copies `security_common.id.value()` into its transient
|
||||
`cc_sec_id_map`. `_add_future()`, `_add_option()`, `_add_spread()`, or
|
||||
`_add_spot()` then calls `_copy_security_common()`, which sets:
|
||||
|
||||
```cpp
|
||||
to.key = incoming_id;
|
||||
to.numeric_id = incoming_id;
|
||||
```
|
||||
|
||||
This collapses the generalized source/correlation ID and Spark shared-memory
|
||||
key into the same number. The source's text `exchange_sec_id` remains on the
|
||||
domain security but is not copied into `FixedSecurity::numeric_id` by this
|
||||
path.
|
||||
|
||||
`_commit_security()` then:
|
||||
|
||||
1. calls `MDProducer::add_security()` with an already non-zero key;
|
||||
2. causes `MDProducerHandle::set_key()` to run immediately during that call;
|
||||
3. inserts the handle in `GeneralSecurityIndex` under `numeric_id` (the same
|
||||
number); and
|
||||
4. returns with the security already visible in shared memory.
|
||||
|
||||
Because the sink's producer has no recorder, supplying `id == 0` would allocate
|
||||
the shared-memory entry but never assign or publish its key. Generalized live
|
||||
sources must therefore provide a valid non-zero ID before `on_securities()`.
|
||||
|
||||
The listener contains an alternative branch for `use_given_sec_key == false`.
|
||||
That branch would call `IDGenerator::get_or_add(me->venue, sec.type(),
|
||||
sec.desc())` itself. It is currently disabled, and `_generate_security_key()` is
|
||||
also effectively unused. Turning the flag off is not a mechanical change: an
|
||||
adapter such as Databento allocates using numeric instrument ID while the
|
||||
listener branch would use `sec.desc()`, potentially creating a second key for
|
||||
the same instrument.
|
||||
|
||||
There is also a spread failure edge case. `_find_sec(sec, true)` records the
|
||||
parent ID in `cc_sec_id_map` before `_add_spread()` resolves its legs. If any leg
|
||||
ID is absent, `_add_spread()` logs and returns without committing the spread. A
|
||||
later repeat of the parent definition is then classified as a duplicate and
|
||||
ignored, so definition ordering is a hard requirement in the current code.
|
||||
|
||||
### 4. How later books, trades, and statistics find the security
|
||||
|
||||
Every later domain event carries a `domain::SecurityID`:
|
||||
|
||||
- `Book::security_id()`;
|
||||
- `TradeRecord::sec_id`; or
|
||||
- `SecurityMarketDataStatistics::security_id`.
|
||||
|
||||
The listener first uses `cc_sec_id_map` to translate that event ID to the value
|
||||
stored in `GeneralSecurityIndex`, then obtains the `MDProducerHandle`. With
|
||||
`use_given_sec_key == true`, this mapping is numerically identity-to-identity,
|
||||
but it still serves as the admission check that the definition was seen.
|
||||
|
||||
Most high-throughput adapters avoid SQLite on market-data events:
|
||||
|
||||
- Databento allocates on the definition and caches
|
||||
`instrument_id -> domain::SecurityID` in `sec_id_map` for books/trades.
|
||||
- Taifex keeps the resolved ID in its security-details mapper.
|
||||
- Other native subscriptions maintain comparable descriptor/ID maps.
|
||||
|
||||
The Python bridge is a notable exception: `to_bts_security_()`,
|
||||
`to_bts_trade_()`, and `to_book_()` all call `get_sec_id_()`, which calls
|
||||
`IDGenerator::get_id(..., add_if_new=true)`. As written, that performs at least
|
||||
one SQLite `SELECT` for each converted Python definition, trade, or book. It
|
||||
returns the existing ID after the first insert, but it is still a DB call and
|
||||
takes the global `IDGenerator` mutex on every event.
|
||||
|
||||
### 5. Persistence timing and latency
|
||||
|
||||
The newer allocator is synchronous:
|
||||
|
||||
- Existing security: mutex acquisition plus one SQLite `SELECT` on the caller's
|
||||
thread.
|
||||
- New security: mutex acquisition, one `SELECT`, then one SQLite `INSERT`
|
||||
(SQLite autocommit), also on the caller's thread.
|
||||
- Listener publication: no further DB call under the current
|
||||
`use_given_sec_key` setting.
|
||||
|
||||
The ID therefore exists in `key_ids.sqlite` before the adapter sends the
|
||||
definition to the listener. There is no intentional 200 ms publication delay,
|
||||
but the source callback/startup path directly pays DB and mutex latency. Bulk
|
||||
definition discovery performs one lookup and possibly one insert per security;
|
||||
`IDGenerator` does not batch those operations.
|
||||
|
||||
WAL mode helps concurrency, while `synchronous=0` reduces fsync cost. It does
|
||||
not eliminate SQL parsing/stepping, mutex contention, page-cache misses, or
|
||||
occasional file-system work. The impact is usually a startup/definition cost
|
||||
when adapters cache the result, but it becomes hot-path latency for code such as
|
||||
the current Python bridge that resolves every event through the DB.
|
||||
|
||||
`GeneralMarketDataListener` itself does not persist the full contract definition
|
||||
to `securities.sqlite`; that remains the separate security-cache/metadata
|
||||
persistence path described earlier.
|
||||
|
||||
## Comparison: MDP3 recorder vs generalized allocator
|
||||
|
||||
| Property | MDP3 / `SecurityIDRecorder` | Generalized / `IDGenerator` |
|
||||
|---|---|---|
|
||||
| Logical input | venue + type + legacy `security_desc` | venue + type + adapter-chosen name |
|
||||
| Numeric output | SQLite rowid, checked/read as 32-bit `security_key` | SQLite rowid returned as 64-bit `domain::SecurityID`, then narrowed into 32-bit shared-memory key |
|
||||
| Allocation point | After definition is copied to shared memory | Normally inside adapter before listener sees definition |
|
||||
| DB execution | Asynchronous on dispatcher timer | Synchronous on caller/source thread |
|
||||
| Existing ID cost | One deferred `SELECT` | One immediate `SELECT` per call |
|
||||
| New ID cost | Deferred, batched transaction (about 200/pass) | Immediate `SELECT` + individual autocommit `INSERT` |
|
||||
| Time until public | Usually 0–200+ ms; spreads may take extra ticks | Immediate after synchronous lookup/insert completes |
|
||||
| Feed callback blocking | Queue push only for ID work | Yes, if the adapter invokes `IDGenerator` there |
|
||||
| Spread legs | Parent waits asynchronously until all leg keys resolve by exchange ID | Definition processing requires every leg's incoming ID already exist in `cc_sec_id_map`; otherwise the spread is aborted |
|
||||
| Runtime ID relationship | Exchange ID and Spark key remain distinct | Incoming/canonical ID, `numeric_id`, index ID, and Spark key are currently the same number |
|
||||
| Persistence database | `key_ids.sqlite` | The same `key_ids.sqlite` |
|
||||
| Batching | Yes for new rows | No |
|
||||
| Failure behavior | Security has an internal handle with key 0 while normal queued work is pending; DB work does not block the feed callback | DB errors/zero results occur in the synchronous source path; a zero ID can leave an unpublishable entry, with no pending-key state machine |
|
||||
|
||||
Neither algorithm generates a UUID or a content hash. Both ultimately use an
|
||||
opaque, database-assigned SQLite rowid. It is normally increasing and looks like
|
||||
a sequence, but this schema does not request SQLite `AUTOINCREMENT`. The tuple
|
||||
used to *look up* the row is logical; the resulting number is not derivable from
|
||||
that tuple.
|
||||
|
||||
## Can another system reconstruct the same keys?
|
||||
|
||||
Not from `(venue, type, name)` alone. Another system can reconstruct the
|
||||
**logical lookup tuple**, but it cannot calculate the numeric Spark key because
|
||||
the key depends on the historical insertion order of every row in
|
||||
`key_ids.sqlite`.
|
||||
|
||||
For example, two empty systems can insert the exact same `GLBX.FUTURE / ESZ6`
|
||||
row and receive different IDs if other securities were inserted before it on
|
||||
one system. They are sequential opaque IDs, not deterministic hashes and not
|
||||
UUIDs.
|
||||
|
||||
Ways to preserve cross-system identity are:
|
||||
|
||||
1. Copy/replicate `key_ids.sqlite` (including rowids) and keep descriptor rules
|
||||
identical. This preserves existing numeric keys but requires coordinated
|
||||
allocation for new ones.
|
||||
2. Export an explicit mapping of `(namespace, adapter key) -> Spark key`, then
|
||||
import rows with their original rowids before either system allocates new
|
||||
securities.
|
||||
3. Treat a stable external identity (venue + type + exchange instrument ID, or
|
||||
another canonical instrument identifier) as the cross-system key and carry a
|
||||
translation table to each system's local Spark key.
|
||||
4. Introduce a deterministic ID scheme only as a versioned migration. A hash can
|
||||
be computed independently, but collision handling, descriptor
|
||||
canonicalization, aliases/renames, the existing 32-bit ABI, and compatibility
|
||||
with orders/recordings/databases all need explicit design.
|
||||
|
||||
Copying only `securities.sqlite` is insufficient as an allocator strategy. It
|
||||
contains security blobs keyed by already-assigned IDs, but `key_ids.sqlite` is
|
||||
the mapping used to find or allocate those IDs. Conversely, copying only
|
||||
`key_ids.sqlite` preserves the numbers but not all contract metadata.
|
||||
|
||||
## Other assignment paths in `sparkd`
|
||||
|
||||
The two main pipelines reduce to four ways a producer handle can acquire a key:
|
||||
|
||||
1. **Already keyed input.** `MDProducer::add_security()` sees a non-zero
|
||||
`FixedSecurity::key` and calls `set_key()` immediately. Generalized feeds,
|
||||
Sunday preload, relay playback, and other replay paths use this mechanism.
|
||||
2. **Recorder-backed live input.** The input key is zero and the producer owns a
|
||||
`SecurityIDRecorder`; `ensure_key()` queues asynchronous assignment. CME
|
||||
MDP3 and legacy OnixS ICE use this mechanism. ICE uses `marketId` as
|
||||
`numeric_id` and otherwise follows the same recorder/index design.
|
||||
3. **Service assignment.** `SyntheticSecurityService` calls
|
||||
`SecurityIDRecorder::assign_key()` directly on a background dispatcher. The
|
||||
callback receives a copied `api::Security` with its top-level `id` set, then
|
||||
publishes it to synthetic subscribers. It uses the same `key_ids.sqlite`
|
||||
rowid allocator.
|
||||
4. **Unkeyed input with no recorder.** The entry is allocated with key zero and
|
||||
never becomes public. This is not a valid production assignment path, but it
|
||||
is the failure mode if a generalized source omits its ID.
|
||||
|
||||
Relay and replay deserve care when reasoning about persistence:
|
||||
|
||||
- The generalized `MDReplayer` forwards the recorded domain security ID and
|
||||
emits later books/statistics with the recorded `sec_key`, so the listener's
|
||||
current use-given mode preserves the recording's numeric identity.
|
||||
- The legacy relay producer converts a recorded `api::Security` with
|
||||
`set_security_key=true`, so its shared-memory publication also preserves the
|
||||
embedded key.
|
||||
- The legacy relay recorder additionally calls `bulk_insert(..., "REPLAY: ")`
|
||||
for logical mappings. Those rows use a `REPLAY: ` namespace prefix; this is a
|
||||
separate mapping population step, not a deterministic derivation of the key.
|
||||
|
||||
## Operational implications
|
||||
|
||||
- Back up `key_ids.sqlite` and `securities.sqlite` together if numeric ID and
|
||||
definition continuity matter.
|
||||
- Do not delete/rebuild `key_ids.sqlite` on one host and assume IDs will match
|
||||
another host, even when all instruments are identical.
|
||||
- Before moving a feed between adapters, compare the exact `name` each adapter
|
||||
supplies to `IDGenerator`. Matching venue/type is insufficient.
|
||||
- Cache generated IDs in the source after definition processing. Avoid calling
|
||||
`IDGenerator` from book/trade hot paths; the Python bridge currently violates
|
||||
this guideline.
|
||||
- Keep generated rowids below the 32-bit shared-memory ceiling until the
|
||||
`security_key` ABI is widened or explicit bounds checking is added to the
|
||||
generalized path.
|
||||
- Ensure spread legs are announced before generalized spread parents.
|
||||
- Treat `synchronous=0` as a durability decision. A copied DB should be taken
|
||||
with SQLite-aware backup/checkpoint handling rather than by casually copying
|
||||
active WAL files.
|
||||
|
||||
Useful inspection queries (run against the configured DB path) are:
|
||||
|
||||
```sql
|
||||
SELECT rowid AS spark_key, ns, key
|
||||
FROM store
|
||||
ORDER BY rowid;
|
||||
|
||||
SELECT rowid AS spark_key, ns, key
|
||||
FROM store
|
||||
WHERE ns = 'GLBX.FUTURE' AND key = '<exact-adapter-key>';
|
||||
```
|
||||
|
||||
## Source map
|
||||
|
||||
- [`SecurityIDRecorder.cpp`](../src/lib_spark/spark/md/SecurityIDRecorder.cpp):
|
||||
legacy queues, 200 ms work ordering, logical row construction, DB callbacks,
|
||||
and spread-leg resolution.
|
||||
- [`IDGenerator.cpp`](../src/lib_spark/spark/persistence/IDGenerator.cpp):
|
||||
synchronous generalized lookup/allocation and process-wide mutex.
|
||||
- [`SqliteStore.cpp`](../src/lib_spark/spark/persistence/SqliteStore.cpp): DB
|
||||
location, schema, WAL/synchronous pragmas, individual inserts, and batch
|
||||
inserts.
|
||||
- [`MDProducer.cpp`](../src/lib_spark/spark/md/MDProducer.cpp) and
|
||||
[`MDProducerHandle.cpp`](../src/lib_spark/spark/md/MDProducerHandle.cpp):
|
||||
shared-memory allocation, recorder handoff, and ordered publication.
|
||||
- [`FixedSecurity.h`](../src/lib_spark/spark/md/Shm/FixedSecurity.h) and
|
||||
[`MDStructure.h`](../src/lib_spark/spark/md/Shm/MDStructure.h): identifier
|
||||
widths and hot/cold storage fields.
|
||||
- [`MDP3Main.cpp`](../src/app_sparkd/sparkd/md/platforms/mdp3_onixs/MDP3Main.cpp)
|
||||
and
|
||||
[`MDP3Listener.h`](../src/app_sparkd/sparkd/md/platforms/mdp3_onixs/MDP3Listener.h):
|
||||
MDP3 construction, definition callbacks, CME ID extraction, and spread legs.
|
||||
- [`MDP3Listener.cpp`](../src/app_sparkd/sparkd/md/platforms/mdp3_onixs/MDP3Listener.cpp):
|
||||
producer/index insertion order.
|
||||
- [`OnixsGlbxSubscription.cpp`](../src/app_sparkd/sparkd/md/glbx/OnixsGlbxSubscription.cpp):
|
||||
current dedicated MDP3 dispatcher-thread ownership.
|
||||
- [`MarketDataGateway.cpp`](../src/app_sparkd/sparkd/md/MarketDataGateway.cpp)
|
||||
and
|
||||
[`GeneralMarketDataListener.cpp`](../src/lib_spark/spark/md/GeneralMarketDataListener.cpp):
|
||||
generalized source/sink topology, incoming-ID mapping, fixed-security
|
||||
conversion, and publication.
|
||||
- [`DatabentoDataHandler.cpp`](../src/app_sparkd/sparkd/md/databento/DatabentoDataHandler.cpp),
|
||||
[`TaifexDataProcessors.h`](../src/app_sparkd/sparkd/md/taifex/TaifexDataProcessors.h),
|
||||
and
|
||||
[`PyMarketDataSubscription.cpp`](../src/app_sparkd/sparkd/md/PyMarketDataSubscription.cpp):
|
||||
representative adapter-side allocation and caching behavior.
|
||||
- [`SecurityDB.cpp`](../src/lib_spark/spark/md/SecurityDB.cpp) and
|
||||
[`SecurityCache.cpp`](../src/lib_spark/spark/md/SecurityCache.cpp): separate
|
||||
persistence of full security definitions.
|
||||
- [`SecurityIndex.h`](../src/lib_spark/spark/md/SecurityIndex.h),
|
||||
[`MBOServiceServer.cpp`](../src/lib_spark/spark/md/MBOServiceServer.cpp), and
|
||||
[`MDConsumer.cpp`](../src/lib_spark/spark/md/MDConsumer.cpp): exchange-index
|
||||
translation followed by Spark-keyed market-data consumption and MBO access.
|
||||
- [`OrderFactory.cpp`](../src/lib_spark/spark/orderman/order_factory/OrderFactory.cpp),
|
||||
[`FillStore.cpp`](../src/lib_spark/spark/orderman/actors/FillStore.cpp), and
|
||||
[`PositionMap.cpp`](../src/app_sparkd/sparkd/position/PositionMap.cpp):
|
||||
representative downstream use of Spark security keys.
|
||||
- [`MDReplayer.cpp`](../src/app_sparkd/sparkd/md/recording/MDReplayer.cpp) and
|
||||
[`RelayClient.cpp`](../src/lib_spark/spark/md/relay/RelayClient.cpp): recorded
|
||||
ID reuse.
|
||||
Reference in new issue
Block a user