Overview
The core abstraction that everything else is built on is the MDProducer+MDProducerHandle / MDConsumer+MDConsumerHandle.
A new market data source constructs an MDProducer and adds securities to it. When a security is added you get back an MDProducerHandle representing that security in shared memory. The memory is fully allocated and books can be written immidiately but the security will not be visible to the MDConsumer until the security key is stored on disk (SecurityIDRecorder). The MDProducerHandle assumes a single writer for most operations and may break silently if this constraint is violated.
The MDConsumer is the dual of the MDProducer. It provides an on_security() callback which is driven by the call to scan_securities(). No threads are created by MDConsumer/MDProducer instead they get cpu time from periodic calls to scan_securities (consumer), SecurityIDRecorder::do_work (producer). When a security is discovered by scan_securities the callback is called with the Security and the MDConsumerHandle. The MDConsumerHandle is thread safe and can be safely copied and read from concurrently.
Single Writer Principle
Core Classes
'*' = New in spark
- MBOProcessor*
- MBOServiceServer*
- MDClient*
- MDConsumer
- MDConsumerHandle
- MDProducer
- MDProducerHandle
- PlatformLoader
- PriceSize/IntPrice
- SecurityCache*
- SecurityOracle*
- SecurityDB*
- SecurityIDRecorder
- SecurityMapper*
- SettlementRecorder
- SyntheticSecurityService
- FixedSecurity.h / MDStructure.h
- SharedList.h
MBOProcessor*
This class is an optional dependency of MDProducer. It takes the low level incremental updates to the MBO book and constructs the book on a different thread. This is optimized for the UI use case so it tries to get the mbo messages out of the critical path as soon as possible. This uses regular heap memory so is not compatible with out of process operation. It exposes a reader and writer interface.
Each update goes into a queue and gets processed by MBOProcessorImpl::process_incremental() on a single background thread. This thread drains the queue and constructs the books in place. Then it copies snapshots of the books and publishes them as immutable snapshots which are consumed by the MBOServiceServer/UI. If mbo load gets very high this function will stop publishing snapshots and only update books until the load reduces.
MBOServiceServer*
Reads the snapshots from the MBOProcessor/MBOReader and serves them for the UI.
MDClient*
New in spark.
This class handles constructing the MDConsumer, calling scan_securities on it and checking for bad heartbeats and re-constructing the MDConsumer if there was a failure. In edge those tasks were the responsibility of each consumer of the market data but it was mitigated because most things used streams rather than MDConsumer directly. MDConsumer is reference counted internally so when the consuming code gets an on_consumer() callback with a new MDConsumer it should keep a copy of the old MDConsumer until it has finished any asyncronous processes involving the MDConsumerHandles from that first consumer. Once an MDConsumer is destroyed the MDConsumerHandles it produced will cause an Access Violation on reading because the memory has been unmapped.
Spark has always used MD in process so, while based on existing functionality, this recovery process hasn't been well tested.
MDConsumer
An MDConsumer maps the shared memory sections and produces MDConsumerHandles. These handles are only valid as long as the MDConsumer exists. It would make some sense to store the MDConsumer reference in the MDConsumerHandle to avoid this but MDConsumerHandle is designed to be very small for fast iteration. The MDConsumer also holds a reference to the ISyntheticSecurityService and makes synthetic securities 'look like' regular shared memory securities using polymorphism in the MDConsumerHandle.
Internally MDConsumer is a collection of MDConsumerInst which each handle a single symbol+type+exchange and the corresponding shared memory blocks.
- scan_securities - in spark this is always called by MDClient
- get_handle - Low level way to get a handle. Does a linear walk though shared memory and can return a void handle (MDConsumerHandle::is_initialize() == false).
- get_managed_handle - returns a 'managed handle' this is a shared_ptr to an MDConsumerhandle. The only difference is that this is tracked by the MDConsumer and may transition from uninitialized to initialized. Storing these in a vector and walking them is almost 50% slower than doing the same thing with an unmanaged handle. If you use these it's probably best to store the uninitialized handles in a nursery and check them periodically. You can dereference a managed handle into a regular handle at any time.
- get_channel - Gets a MDConsumerChannel representing common data for a symbol+type+exchange. This is mostly for the trade ticker / rfq and also exposes various counters which get published to dev-stats.
On Linux MDConsumer maps the shared memory as read-only pages. On Windows it cannot do this because std::atomic uses a different instruction which is considered a write by the CPU so we map it read/write.
If we have multiple platforms that can produce md for an exchange (i.e. onixs vs emumba) the MDConsumer consults the PlatformMap config key to determine which to read from. The md replayer impersonates the producer it was recorded from.
MDConsumerChannel
read_trades() and read_rfqs() read from a ring buffer stored in the header section of the shared memory. They read directly into a google::protobuf::RepeatedPtrField to avoid extra copies in the QueryServer but can just as easily read into a std::vector. They expose a 'cursor' representing the position in the ring buffer and this cursor is sent all the way to the client so it can request updates.
MDConsumerHandle
An MDConsumerHandle 4 states and would proably be impemented as an std::variant<> if done today. If not for the high performance requirements it could also be a virtual class but I didn't want to pay for a vtable. It is the size of 4 pointers plus the enum tag.
- VOID - an uninitialized handle
- BBO - Legacy from Edge. This represents an option book where we only kept the top to save space. I think this has rotted by now and wont work but I kept it as an example of how to add polymorphism to the shared memory types.
- DEPTH - Used for all exchange securities in Spark.
- SYNTHETIC - A Synthetic handle. These are bigger so they get stored as a pointer.
Reading from a BBO/DEPTH handle is done with an optimistic copy. If you inrease the size of the book data this copy can fail more often and get significantly worse perf. The result of the read can be changed, unchanged or busy (timeout). In edge 'busy' happened 1-2 times per day across all reads and gets logged by QueryServer.
MDProducer
Handle to writeable shared memory. Each market data producer "platform" (onixs_cme, onixs_ice, emumba, etc) constructs one.
MDProducerHandle
All updates to a single security go through here. Assumes a single writer thread.
PlatformLoader
In Edge this was the entry point to the run() for the market data producer process. It loaded the bts_md_*.so so each producer process only loaded the third party objects for a particular producer. In spark this is all in process so this is defunct. It looks like it still exists because it has some utility functions that probably belong in another file.
PriceSize/IntPrice
Market data uses integer prices to avoid double problems.
SecurityCache*
Created/Owned by MDClient this owns the Securities.
It has methods to get a security from the live collection or load them from the SecurityDB on disk if someone asks for one that hasn't been seen yet.
- If a security is loaded from disk it 'freezes' it so when the live security arrives it will continue to serve the old security so we don't end up with 2 copies of a single security with different properties.
- This class is also responsible for sending the live securities to the SecurityDB where we always overwrite stored securities with what comes from the exchange.
- Survives beyond the MDConsumer in case of crash/recovery of out of process md.
SecurityOracle*
Interface on top of SecurityCache exposing the read get_ operations.
SecurityDB*
- Just a collection of methods for save/load of securities to sqlite.
- Stored in binary format so we can use the protobuf versioning
SecurityIDRecorder
- Manages security to unique 32 bit id mapping across multiple processes
- Provides a callback when ids are safely persisted.
SecurityMapper*
- Happens before the security gets stored in shared memory.
- Replaces the mapping.h from edge with a more organized mechanism.
- Owned by the MDProducer each security is passed though SecurityMapper::apply() before it is first stored in shared memory
- SecurityMapper is made up of multiple ISecurityMapper instances so each mapping can be in it's own class so it's easier to read each mapping. Currently all mappings are defined in SecurityMaper.cpp
- The mappings each have an opportunity to mutate the FixedSecurity.
SettlementRecorder
In Edge SettlementRecorder stored the settlements for a security so we could preserve yesterday/day before settlement. In Spark this broke and nobody seemed to care so it was never fixed.
SyntheticSecurityService
Each synthetic security is a api::Security and a chunk of metadata. This service handles API requests to create/update synthetics and fires callbacks for MDConsumer which does the work of producing synethic books.
MDStructure.h
Hot vs Cold
FixedSecurity.h
- variable sized structure in shared mem
- For FixedSpreadSecurity we stack allocate a lot of legs then memcpy() into shared memory only the correct number of legs
- When accessing FixedSpreadSecurity it is cast back to a large number of legs. Iterate legs based on leg_count() not the static leg count.
SharedList.h
- Primitive list-like abstraction over shared memory
- Allows variable sized entries so always accessed in O(N)
- Data never moves so we point into it for O(1)
Market Data Relay / Recorder
-
QueryServer
- Stateless request/response
- Uses multiple threads for throughput
- Uses change timestamps per channel instead of per security
- Will over-send books that change around the time of the request
-
RelayClient
- Runs in the shell (or outside)
- Replicates books from the server (QueryServer) to local shared memory
-
RelayRecorder
- Records Protobufs from QueryServer to a file
Diagnostic tools
- sparkd md dump [symbol] [type] [exchange] [hot?]
- Primary diagnostic tool
- ShmDumper.cpp
- Shows current contents of shared memory
- append 'hot' to the end to see hot data i.e. books
- If you get a "cannot open" check that the file exists in /run/shm/
- dumpsecurities
- Inspect security database
- sparkd md tail
- MDPeriodicDumper.cpp
- for validating new market data
- prints books at given sequence number interval
- rectool
- print recording to text / convert back to futures.coconut
- This is most useful to checking if a particular symbol is in a replay file
- Occasionally useful for testing very specific market data transitions
- print recording to text / convert back to futures.coconut
Tuning / Dealing with Loss
Packet loss
If a small number 1-5 is lost it's upstream
- Use tcpdump to capture
- Check the tcpdump for gaps
- Ask Pete to contact hosting provider
If large gap probably overflow
- make sure socket buffers sizes, rmem_max, spin_time, data_wait are all set
- check affinity is set and working
- check ethtool
Show ethernet ring buffer sizes
ethtool -g eth0
change ethernet ring buffer rx size
sudo ethtool -G [device] rx 4096
interface statistics
ethtool -S eth0
- particularly look at: ethtool -S p1p1 | egrep "(bad)|(nodesc)"
If increasing the ring buffer doesn't work you can tweak interrupt coalescing using:
-
/proc/sys/net/core/dev_weight
-
This is the maximum number of packets that can be dequeued in a single interrupt
-
NOTE: We've never used this and it's unlikely it will fix anything