diff --git a/src/client/spark_api/Co2/MDConsumerHandle.cs b/src/client/spark_api/Co2/MDConsumerHandle.cs index 36b560c1a..b29adbf10 100644 --- a/src/client/spark_api/Co2/MDConsumerHandle.cs +++ b/src/client/spark_api/Co2/MDConsumerHandle.cs @@ -259,6 +259,7 @@ namespace mbts.co2 { } } + // If Size is wrong remember to update from [StructLayout(LayoutKind.Sequential, Size = 704)] internal struct ColdEntry { public int exchange_status; diff --git a/src/lib/md/readme.md b/src/lib/md/readme.md index 2249426a0..a87dc8b47 100644 --- a/src/lib/md/readme.md +++ b/src/lib/md/readme.md @@ -5,7 +5,7 @@ The core abstraction that everything else is built on is the MDProducer+MDProduc A new market data source constructs an MDProducer and adds securities to it. When a security is added you get back an MDProducerHandle representing that security in shared memory. The memory is fully allocated and books can be written immidiately but the security will not be visible to the MDConsumer until the security key is stored on disk (SecurityIDRecorder). The MDProducerHandle assumes a **single writer** for most operations and *may break silently if this constraint is violated*. -The MDConsumer is the dual of the MDProducer (duh). It provides an on_security() callback which is driven by the call to scan_securities(). No threads are created by MDConsumer/MDProducer instead they get cpu time from periodic calls to scan_securities (consumer), SecurityIDRecorder::do_work (producer). When a security is discovered by scan_securities the callback is called with the Security and the MDConsumerHandle. The MDConsumerHandle is thread safe and can be safely copied and read from concurrently. +The MDConsumer is the dual of the MDProducer. It provides an on_security() callback which is driven by the call to scan_securities(). No threads are created by MDConsumer/MDProducer instead they get cpu time from periodic calls to scan_securities (consumer), SecurityIDRecorder::do_work (producer). When a security is discovered by scan_securities the callback is called with the Security and the MDConsumerHandle. The MDConsumerHandle is thread safe and can be safely copied and read from concurrently. ## Single Writer Principle - https://mechanical-sympathy.blogspot.com/2011/09/single-writer-principle.html @@ -118,6 +118,7 @@ Interface on top of SecurityCache exposing the read get_ operations. ## SecurityMapper* +* Happens *before* the security gets stored in shared memory. * Replaces the mapping.h from edge with a more organized mechanism. * Owned by the MDProducer each security is passed though SecurityMapper::apply() before it is first stored in shared memory * SecurityMapper is made up of multiple ISecurityMapper instances so each mapping can be in it's own class so it's easier to read each mapping. Currently all mappings are defined in SecurityMaper.cpp @@ -133,102 +134,75 @@ Each synthetic security is a api::Security and a chunk of metadata. This servic ## MDStructure.h +### Hot vs Cold ## FixedSecurity.h -- variable sized structure in shared mem +* variable sized structure in shared mem +* For FixedSpreadSecurity we stack allocate a lot of legs then memcpy() into shared memory only the correct number of legs +* When accessing FixedSpreadSecurity it is cast back to a large number of legs. Iterate legs based on leg_count() not the static leg count. ## SharedList.h - - - +* Primitive list-like abstraction over shared memory +* Allows variable sized entries so always accessed in O(N) +* Data never moves so we point into it for O(1) # Market Data Relay / Recorder * QueryServer + - Stateless request/response + - Uses multiple threads for throughput + - Uses change timestamps per channel instead of per security + - Will over-send books that change around the time of the request + * RelayClient + - Runs in the shell (or outside) + - Replicates books from the server (QueryServer) to local shared memory + * RelayRecorder + - Records Protobufs from QueryServer to a file # Diagnostic tools -* ShmDumper -* MDPeriodicDumper +* sparkd md dump [symbol] [type] [exchange] [hot?] + - **Primary diagnostic tool** + - ShmDumper.cpp + - Shows current contents of shared memory + - append 'hot' to the end to see hot data i.e. books + - If you get a "cannot open" check that the file exists in /run/shm/ +* dumpsecurities + - Inspect security database +* sparkd md tail + - MDPeriodicDumper.cpp + - for validating new market data + - prints books at given sequence number interval +* rectool + - print recording to text / convert back to futures.coconut + - Occasionally useful for testing very specific market data transitions + +# Tuning / Dealing with Loss + +## Packet loss +#### If a small number 1-5 is lost it's upstream +- Use tcpdump to capture +- Check the tcpdump for gaps +- Ask Pete to contact hosting provider +#### If large gap probably overflow +- make sure socket buffers sizes, rmem_max, spin_time, data_wait are all set +- check affinity is set and working +- check ethtool +#### Show ethernet ring buffer sizes +ethtool -g eth0 +#### change ethernet ring buffer rx size +sudo ethtool -G [device] rx 4096 +#### interface statistics +ethtool -S eth0 +- particularly look at: + ethtool -S p1p1 | egrep "(bad)|(nodesc)" +### If increasing the ring buffer doesn't work you can tweak interrupt coalescing using: +- /proc/sys/net/core/dev_weight +- This is the maximum number of packets that can be dequeued in a single interrupt +- https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-network-common-queue-issues.html - - -PlatformLoader: - * Loads the platform specific dll - * Calls ->start_marketdata() with a full vector - -SharedList / SharedListWriter - * Shared memory list object which spans across shared memory segments as necessary - * O(N) walk - * atomic add() - * Single adder / multiple readers - * no delete - * Custom header - -SecurityIDRecorder - * add_security(security, lambda callback) -> int returns security id instantly - * security_persisted callback when security id is safely persisted - -SecurityIndex - * Platform Implementation helper - * Maps exhange_id to (ProducerHandle + Data) - -CacheEntry / CacheEntryDepth (CacheStructure.h) - * CacheEntry(symbol, depth, # legs) -- size determined in constructor - * Simple inheritance tree for optional depth - * Has FixedSecurity - * Fits in SharedList - * CRITICAL fields are at the front cache aligned. bbo/depth. Security goes at the end - - * size, is_public(), is_persisted, is_resolved (spread) - * statistics like bbo/trade counters - * is_status_ok - * bbo, depth, trade, status, open, close - -FixedSecurity / FixsdSpreadSecurity - * Normalized version of our protobuf security - * Has methods to convert to pb Security - * Size known at construction w/ # legs - spread legs of type security_key - * Can use a union for option/future fields - -MDProducer(symbol, type) - * manages lifetime of SharedList instances - * Hands out MDProducerHandle - entry for one security - -MDProducerHandle - * pointers to shared memory - * update methods - -MDConsumer - * list of securities - * find_security_id(symbol, type, etc) - * get_cache_handle(security id#) - * Hands out MDCacheHandle - - if security is null add MDCacheHandle to a list to be updated - next time securities are polled - -MDCacheHandle - * If security is not present add this is "null" - * has pointers to relevant shared memory - - -# Cache Structure - -2 Shared memory segment for each (SYMBOL, TYPE) - md_hot (bbo,depth,status), md_cold (others) - -When a new security is discovered - 1. id is allocated to the security - 2. New CacheEntry is allocated in appropriate arena (is_public=false) - 3. md_cache entry count is incremented - 4. When id is persisted is_public=true, increment version on cache arena - -# Out of Process / Recovery - - -# Future Improvements - - +- **NOTE**: We've never used this and it's unlikely it will fix anything