Changes to make HNG/HIS/HEN/HHD to work

This is in response to the Pure debacle of May 17th. CME changed the
MSGW used for certain products. As we only route orders once, when
they are created, any orders that already existed never transitioned
to the new MSGW. This made it so that users could no longer control
these orders. With this change we recalculate routes when the orders
are first loaded from disk. Assuming we have up to date securities,
something that sometimes does not happen unfortunately, this should
result in the order being automatically relocated to its new MSGW. As
a bonus we also get the hability to change MSGW configuration even if
there are working for that MSGW.

.gitignore

@@ -35,3 +35,4 @@ trace_data/
 .ionide/*
 /.ignore
 /build
+.clangd/

.vscode/launch.json

@@ -1,6 +1,30 @@
 {
     "version": "0.2.0",
     "configurations": [
+        {
+            "name": "Spark Local",
+            "type": "cppdbg",
+            "request": "launch",
+            "program": "${workspaceRoot}/build/ninja_debug/server/sparkd",
+            "args": [
+                "run",
+                // "test",
+                // "-a",
+                // "[ilink3]"
+            ],
+            "stopAtEntry": false,
+            "cwd": "${workspaceRoot}/build/ninja_debug/server/",
+            "environment": [],
+            "externalConsole": false,
+            "MIMode": "gdb",
+            "setupCommands": [
+                {
+                    "description": "Enable pretty-printing for gdb",
+                    "text": "-enable-pretty-printing",
+                    "ignoreFailures": true
+                }
+            ]
+        },
         {
             "name": "SparkVM Launch",
             "type": "cppdbg",

doc/api/content/client/getting-started/install.md

@@ -5,7 +5,7 @@ weight = 10
 
 # C#/.NET
 
-We target [.NET Standard 2.0](https://docs.microsoft.com/en-us/dotnet/standard/net-standard) so you can use .NET Framework 4.6.1+ or .NET Core 2.0+.  Simply add a reference to the [Spark Nuget package](https://nuget.org) to get started.
+We target [.NET Standard 2.0](https://docs.microsoft.com/en-us/dotnet/standard/net-standard) so you can use .NET Framework 4.6.1+ or .NET Core 2.0+.  Simply add a reference to the [Spark Nuget package](https://www.nuget.org/packages/spark_sdk/) to get started.
 
 ```
 dotnet new console -lang c#

doc/api/content/client/getting-started/services.md

@@ -124,4 +124,17 @@ service OrderService {
 ```
 ### C++ Example
 ```cpp
+#include "lib/api/public/ServiceContext.h"
+#include "lib/api/services/OrderService.client.pb.h"
+using namespace bts::api;
+ServiceContext ctx;
+auto orders = ctx.locate<orderman::OrderService>();
+auto unsub = orders.subscribe_fills().subscribe([](const auto& update) {
+  std::cout 
+    << "Added " << update.added().size() << " items, "
+    << "Removed " << update.removed().size() << " items, "
+    << "Updated " << update.updated().size() << " items, "
+    << "Total items " << update.all().size() << std::endl;
+});
+std::this_thread::sleep_for(std::chrono::minutes(5));
 ```

doc/api/content/client/services/marketdata.md

@@ -1,3 +1,51 @@
 +++
 title = "Market Data"
-+++
++++
+
+
+Use the MarketDataService to subscribe enumerate all available securities.
+
+```cs
+using System;
+using System.Threading;
+using bts.api;
+using bts.api.core;
+
+namespace sample {
+   class Program {
+      static void Main(string[] args) {
+         using var context = new ServiceContext();
+         var md = context.locate<MarketDataService>();
+         using var sub = md.subscribe_securities().Subscribe(update => {
+            foreach (var security in update.added) {
+               Console.WriteLine($"Added security {security.Id}: {security.Symbol}");
+            }
+         });
+
+         Thread.Sleep(TimeSpan.FromMinutes(5));
+      }
+   }
+}
+```
+
+```cpp
+#include "lib/api/public/ServiceContext.h"
+#include "spark/lib/domain_model/Security.h"
+#include "lib/api/services/MarketDataService.client.pb.h"
+#include <thread>
+
+using namespace bts::api;
+
+int main(int argc, char** argv) {
+   ServiceContext ctx;
+   auto md_service = ctx.locate<bts::api::MarketDataService>();
+   auto sub = md_service.subscribe_securities().subscribe([](const auto& update) {
+      for (const auto& s : update.added()) {
+         std::cout << "Added security: " << s->id() << ": " << s->symbol() << std::endl;
+      }
+   });
+
+   std::this_thread::sleep_for(std::chrono::minutes(5));
+   return 0;
+}
+```

doc/api/content/server/getting-started/download.md

@@ -68,12 +68,21 @@ To update your Spark SDK simply download an unzip a new [spark_sdk.zip](https://
 
 To preserve the old image so you can go back to it later run `docker tag spark_sdk spark_sdk:your_custom_tag` before loading the new SDK image.
 
-<!--
-Then stop, remove and restart spark_sdk.
+### Re-build the Dev Container
+
+If you are using Visual Studio Code you also need to rebuild the dev-container so it picks up the new image.
+
+#### Inside VSCode
+To re-build from inside Visual Studio Code.  Click the 'Remote Explorer' icon on the left hand side. 
+You should see your spark_sdk project listed under the Containers section.  Right click on it and select **rebuild-container**.
+
+#### Command Line
+To re-build from the command line run `docker ps -a` and find the row with the spark_sdk image. 
 ```
-c:\bts> docker stop spark_sdk
-c:\bts> docker rm spark_sdk
-c:\bts> docker run -d -v %cd%:/sdk --name spark_sdk bluetradingsytems/spark_sdk
+CONTAINER ID        IMAGE                          COMMAND                  CREATED              STATUS                      PORTS                                NAMES
+54d96cfa3888        spark_sdk                      "/bin/sh -c 'echo Co…"   About a minute ago   Up About a minute           127.0.0.1:9000-9001->9000-9001/tcp   quizzical_lichterman
+```
+Then run `docker rm 54d96cfa3888` (use your container id instead of 54d96cfa3888)  or `docker rm quizzical_lichterman` (use your container name).
+
 
-``` -->
 

doc/api/templates/index.html

@@ -73,7 +73,7 @@
                             <p>
                                 The Spark client SDK allows you to integrate spark with your existing ecosystem. The Spark Client is built entirely on this simple, powerful API.
                             </p>
-                            <a href="" class="button">Get started</a>
+                            <a href="{{ get_url(path="@/client/_index.md") }}" class="button">Get started</a>
                         </div>
                         
                     </div>

doc/user/content/release_notes/_index.md

@@ -4,6 +4,13 @@ weight = 3
 sort_by = "weight"
 insert_anchor_links = "right"
 +++
+## 2.15.10
+### Bug Fixes
+  - Solved an issue that could cause the user interface to fail when playing some alert sounds
+### API Changes
+  - Added settlement and position information to the API
+  - Added the ability for user algos to report custom state to the user interface
+  - Added the ability for algos to discover all active securities
 ## 2.15.3
 ### Bug Fixes
   - Fixed a bug where some CME currency spreads exhibited incorrect tick values

src/client/spark/Modules/Market/MarketView/MarketViewRow.cs

@@ -1,370 +1,372 @@
-using bts.api;
-using bts.api.orderman;
-using bts.collections;
-using mbts.co2;
-using mbts.spark.utilities;
-using mbts.ui;
-using mbts.ui.controls;
-using mbts.ui.spark;
-using mbts.ui.utility;
-using mbts.utils;
-using System;
-using System.Collections.Generic;
-using System.Collections.ObjectModel;
-using System.Collections.Specialized;
-using System.Linq;
-using System.Reactive.Linq;
-using System.Windows.Input;
-
-namespace mbts.spark {
-   partial class MarketViewVM {
-      public class Row : SecurityMDRow, IDisposable, IEquatable<Row>, DIYGrid.IRowVM, DIYGrid.IActionableRowVM {
-         readonly IBus _bus;
-         readonly OrdersModel _orders_model;
-         readonly AnimationManager _animation = new AnimationManager();
-         readonly ObservableCollection<OrderRecordImmutable> _orders = new ObservableCollection<OrderRecordImmutable>();
-         readonly ObservableFilterCollection<OrderRecordImmutable> _working_orders, _working_bids, _working_asks;
-         readonly string _security_text, _security_key_text;
-         readonly Property<double> _bid_price, _ask_price, _last_price, _day_change, _settlement, _session_high, _session_low;
-         readonly Property<bool> _last_price_from_spread;
-         readonly CellProperty<int> _position;
-         readonly CellProperty<uint> _last_size;
-         readonly CellProperty<ulong> _volume;
-         readonly CellProperty<bool> _inside_bids, _inside_asks;
-         readonly CellProperty<int> _working_bid_volume, _working_ask_volume;
-         readonly CellProperty<double> _working_best_bid, _working_best_ask;
-         readonly CellProperty<string> _working_bids_text, _working_asks_text;
-         readonly MyOrderSummarizer _order_summarizer = new MyOrderSummarizer();
-         readonly MDDisplayHelper.SizeFormatter _size_formatter = MDDisplayHelper.get_formatter(MarketSizeDisplay.get_default());
-         Property<uint> _bid_direct_size, _bid_merged_size, _ask_direct_size, _ask_merged_size;
-         double _raw_day_position;
-         string _bid_merged_size_text, _ask_merged_size_text;
-         ulong _bbo_ref, _last_ref;
-
-         public Row(IBus bus, SecurityMD smd, OrdersModel orders_model) : base(smd) {
-            _bus = bus;
-            _orders_model = orders_model;
-            _security_text = smd.security.ToString();
-            _security_key_text = security_key.ToString();
-            string format_price(double price) => security.ContractSpec.format(price);
-            _position = new CellProperty<int>(this, nameof(position), 0);
-            _bid_price = backing(nameof(bid_price), double.NaN);
-            _ask_price = backing(nameof(ask_price), double.NaN);
-            _last_price = backing(nameof(last_price), double.NaN);
-            _last_size = new CellProperty<uint>(this, nameof(last_size), 0);
-            _day_change = backing(nameof(day_change), double.NaN);
-            _session_low = backing(nameof(session_low), double.NaN);
-            _session_high = backing(nameof(session_high), double.NaN);
-            _last_price_from_spread = backing(nameof(last_price_from_spread), false);
-            _settlement = backing(nameof(settlement), double.NaN);
-            _volume = new CellProperty<ulong>(this, nameof(volume), 0);
-            _inside_bids = new CellProperty<bool>(this, nameof(inside_bids));
-            _inside_asks = new CellProperty<bool>(this, nameof(inside_asks));
-            _working_bid_volume = new CellProperty<int>(this, nameof(working_bid_volume));
-            _working_ask_volume = new CellProperty<int>(this, nameof(working_ask_volume));
-            //
-            _bid_direct_size = backing(nameof(bid_direct_size), default(uint));
-            _bid_merged_size = backing(nameof(bid_merged_size), default(uint));
-            _ask_direct_size = backing(nameof(ask_direct_size), default(uint));
-            _ask_merged_size = backing(nameof(ask_merged_size), default(uint));
-            //
-            _working_orders = new ObservableFilterCollection<OrderRecordImmutable>(_orders, o => o.is_active());
-            _working_bids = new ObservableFilterCollection<OrderRecordImmutable>(_working_orders, o => o.Def.Side == Side.Buy);
-            _working_asks = new ObservableFilterCollection<OrderRecordImmutable>(_working_orders, o => o.Def.Side == Side.Sell);
-            _working_best_bid = new CellProperty<double>(this, nameof(working_best_bid), format_price, double.NaN);
-            _working_best_ask = new CellProperty<double>(this, nameof(working_best_ask), format_price, double.NaN);
-            _working_bids_text = new CellProperty<string>(this, nameof(working_bids_text), string.Empty);
-            _working_asks_text = new CellProperty<string>(this, nameof(working_asks_text), string.Empty);
-            //
-            default_command = new DelegateCommand(_show_ladder);
-            bid_price_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.Bid));
-            bid_merged_size_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.BidMergedSize));
-            bid_direct_size_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.BidDirectSize));
-            bid_implied_size_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.BidImpliedSize));
-            working_bids_click = new DelegateCommand(() => _show_order_book(MarketViewView.Columns.WorkingBids));
-            ask_price_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.Ask));
-            ask_merged_size_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.AskMergedSize));
-            ask_direct_size_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.AskDirectSize));
-            ask_implied_size_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.AskImpliedSize));
-            working_asks_click = new DelegateCommand(() => _show_order_book(MarketViewView.Columns.WorkingAsks));
-            _animation.AnimationsChanged += on_animations_changed;
-            void on_animations_changed() => RaisePropertyChanged(nameof(animations));
-            _dispose = Disposable.Create(
-               _orders_model.on_security_root_orders(security.Id).OnDispatcher().Subscribe(_on_orders),
-               _position.OnDispatcher().Subscribe(v => _animation.animate_position(v.newval > v.oldval)),
-               _orders_model.on_security_positions(security.Id).OnDispatcher().Subscribe(t => {
-                  if (Set(ref _raw_day_position, t.state.Values.Sum(p => p.Item2.raw_day_pos()), nameof(raw_day_position)))
-                     raise_invalidated();
-               }),
-               _inside_bids.OnDispatcher().Subscribe(v => _animation.animate_working_bids_inside(v.newval)),
-               _inside_asks.OnDispatcher().Subscribe(v => _animation.animate_working_asks_inside(v.newval)),
-               _day_change.OnDispatcher().Subscribe(v => _animation.animate_day_change(v.newval > v.oldval)),
-               Disposable.Create(() => _animation.AnimationsChanged -= on_animations_changed)
-            );
-         }
-
-         public event Action invalidated;
-
-         #region Bound Properties
-         public bool inside_bids => _inside_bids.value;
-         public bool inside_asks => _inside_asks.value;
-         public int position => _position.value;
-         public double raw_day_position => _raw_day_position;
-         public double last_price => _last_price.Value;
-         public bool last_price_from_spread => _last_price_from_spread.Value;
-         public uint last_size => _last_size.value;
-         public double bid_price => _bid_price.Value;
-         public uint bid_direct_size => _bid_direct_size.Value;
-         public uint bid_implied_size => _bid_merged_size.Value - _bid_direct_size.Value;
-         public uint bid_merged_size => _bid_merged_size.Value;
-         public string bid_merged_size_text => _bid_merged_size_text ?? (_bid_merged_size_text = _size_formatter(_bid_merged_size.Value, bid_implied_size));
-         public double ask_price => _ask_price.Value;
-         public uint ask_direct_size => _ask_direct_size.Value;
-         public uint ask_implied_size => _ask_merged_size.Value - _ask_direct_size.Value;
-         public uint ask_merged_size => _ask_merged_size.Value;
-         public string ask_merged_size_text => _ask_merged_size_text ?? (_ask_merged_size_text = _size_formatter(_ask_merged_size.Value, ask_implied_size));
-         public ulong volume => _volume.value;
-         public double settlement => _settlement.Value;
-         public double day_change => _day_change.Value;
-         public double session_low => _session_low.Value;
-         public double session_high => _session_high.Value;
-         public int working_bid_volume => _working_bid_volume.value;
-         public int working_ask_volume => _working_ask_volume.value;
-         public string working_bids_text => OrderUtils.order_collection_summary(_order_summarizer.best_working_bid, _order_summarizer.best_working_bid_qty, _order_summarizer.working_bid_qty, security.ContractSpec);
-         public string working_asks_text => OrderUtils.order_collection_summary(_order_summarizer.best_working_ask, _order_summarizer.best_working_ask_qty, _order_summarizer.working_ask_qty, security.ContractSpec);
-         public double working_best_bid => _working_best_bid.value;
-         public double working_best_ask => _working_best_ask.value;
-         public IList<ui.spark.CellAnimation> animations => _animation.get_animations().ToList();
-         public ICommand bid_price_click { get; }
-         public ICommand bid_merged_size_click { get; }
-         public ICommand bid_direct_size_click { get; }
-         public ICommand bid_implied_size_click { get; }
-         public ICommand working_bids_click { get; }
-         public ICommand ask_price_click { get; }
-         public ICommand ask_merged_size_click { get; }
-         public ICommand ask_direct_size_click { get; }
-         public ICommand ask_implied_size_click { get; }
-         public ICommand working_asks_click { get; }
-         public ICommand default_command { get; }
-         public string market_description {get;}
-         public string cleared_alias { get; }
-         public string product_description { get; }
-         public string hub_alias { get; }
-         public string strip_name{ get; }
-         public string mic_code { get; }
-
+using bts.api;
+using bts.api.orderman;
+using bts.collections;
+using mbts.co2;
+using mbts.spark.utilities;
+using mbts.ui;
+using mbts.ui.controls;
+using mbts.ui.spark;
+using mbts.ui.utility;
+using mbts.utils;
+using System;
+using System.Collections.Generic;
+using System.Collections.ObjectModel;
+using System.Collections.Specialized;
+using System.Linq;
+using System.Reactive.Linq;
+using System.Windows.Input;
+
+namespace mbts.spark {
+   partial class MarketViewVM {
+      public class Row : SecurityMDRow, IDisposable, IEquatable<Row>, DIYGrid.IRowVM, DIYGrid.IActionableRowVM {
+         readonly IBus _bus;
+         readonly OrdersModel _orders_model;
+         readonly AnimationManager _animation = new AnimationManager();
+         readonly ObservableCollection<OrderRecordImmutable> _orders = new ObservableCollection<OrderRecordImmutable>();
+         readonly ObservableFilterCollection<OrderRecordImmutable> _working_orders, _working_bids, _working_asks;
+         readonly string _security_text, _security_key_text;
+         readonly Property<double> _bid_price, _ask_price, _last_price, _day_change, _settlement, _session_high, _session_low;
+         readonly Property<bool> _last_price_from_spread;
+         readonly CellProperty<int> _position;
+         readonly CellProperty<uint> _last_size;
+         readonly CellProperty<ulong> _volume;
+         readonly CellProperty<bool> _inside_bids, _inside_asks;
+         readonly CellProperty<int> _working_bid_volume, _working_ask_volume;
+         readonly CellProperty<double> _working_best_bid, _working_best_ask;
+         readonly CellProperty<string> _working_bids_text, _working_asks_text;
+         readonly MyOrderSummarizer _order_summarizer = new MyOrderSummarizer();
+         readonly MDDisplayHelper.SizeFormatter _size_formatter = MDDisplayHelper.get_formatter(MarketSizeDisplay.get_default());
+         Property<uint> _bid_direct_size, _bid_merged_size, _ask_direct_size, _ask_merged_size;
+         double _raw_day_position;
+         string _bid_merged_size_text, _ask_merged_size_text;
+         ulong _bbo_ref, _last_ref;
+
+         public Row(IBus bus, SecurityMD smd, OrdersModel orders_model) : base(smd) {
+            _bus = bus;
+            _orders_model = orders_model;
+            _security_text = smd.security.ToString();
+            _security_key_text = security_key.ToString();
+            string format_price(double price) => security.ContractSpec.format(price);
+            _position = new CellProperty<int>(this, nameof(position), 0);
+            _bid_price = backing(nameof(bid_price), double.NaN);
+            _ask_price = backing(nameof(ask_price), double.NaN);
+            _last_price = backing(nameof(last_price), double.NaN);
+            _last_size = new CellProperty<uint>(this, nameof(last_size), 0);
+            _day_change = backing(nameof(day_change), double.NaN);
+            _session_low = backing(nameof(session_low), double.NaN);
+            _session_high = backing(nameof(session_high), double.NaN);
+            _last_price_from_spread = backing(nameof(last_price_from_spread), false);
+            _settlement = backing(nameof(settlement), double.NaN);
+            _volume = new CellProperty<ulong>(this, nameof(volume), 0);
+            _inside_bids = new CellProperty<bool>(this, nameof(inside_bids));
+            _inside_asks = new CellProperty<bool>(this, nameof(inside_asks));
+            _working_bid_volume = new CellProperty<int>(this, nameof(working_bid_volume));
+            _working_ask_volume = new CellProperty<int>(this, nameof(working_ask_volume));
+            //
+            _bid_direct_size = backing(nameof(bid_direct_size), default(uint));
+            _bid_merged_size = backing(nameof(bid_merged_size), default(uint));
+            _ask_direct_size = backing(nameof(ask_direct_size), default(uint));
+            _ask_merged_size = backing(nameof(ask_merged_size), default(uint));
+            //
+            _working_orders = new ObservableFilterCollection<OrderRecordImmutable>(_orders, o => o.is_active());
+            _working_bids = new ObservableFilterCollection<OrderRecordImmutable>(_working_orders, o => o.Def.Side == Side.Buy);
+            _working_asks = new ObservableFilterCollection<OrderRecordImmutable>(_working_orders, o => o.Def.Side == Side.Sell);
+            _working_best_bid = new CellProperty<double>(this, nameof(working_best_bid), format_price, double.NaN);
+            _working_best_ask = new CellProperty<double>(this, nameof(working_best_ask), format_price, double.NaN);
+            _working_bids_text = new CellProperty<string>(this, nameof(working_bids_text), string.Empty);
+            _working_asks_text = new CellProperty<string>(this, nameof(working_asks_text), string.Empty);
+            //
+            default_command = new DelegateCommand(_show_ladder);
+            bid_price_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.Bid));
+            bid_merged_size_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.BidMergedSize));
+            bid_direct_size_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.BidDirectSize));
+            bid_implied_size_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.BidImpliedSize));
+            working_bids_click = new DelegateCommand(() => _show_order_book(MarketViewView.Columns.WorkingBids));
+            ask_price_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.Ask));
+            ask_merged_size_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.AskMergedSize));
+            ask_direct_size_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.AskDirectSize));
+            ask_implied_size_click = new DelegateCommand(() => _show_order_ticket(MarketViewView.Columns.AskImpliedSize));
+            working_asks_click = new DelegateCommand(() => _show_order_book(MarketViewView.Columns.WorkingAsks));
+            _animation.AnimationsChanged += on_animations_changed;
+            void on_animations_changed() => RaisePropertyChanged(nameof(animations));
+            _dispose = Disposable.Create(
+               _orders_model.on_security_root_orders(security.Id).OnDispatcher().Subscribe(_on_orders),
+               _position.OnDispatcher().Subscribe(v => _animation.animate_position(v.newval > v.oldval)),
+               _orders_model.on_security_positions(security.Id).OnDispatcher().Subscribe(t => {
+                  if (Set(ref _raw_day_position, t.state.Values.Sum(p => p.Item2.raw_day_pos()), nameof(raw_day_position)))
+                     raise_invalidated();
+               }),
+               _inside_bids.OnDispatcher().Subscribe(v => _animation.animate_working_bids_inside(v.newval)),
+               _inside_asks.OnDispatcher().Subscribe(v => _animation.animate_working_asks_inside(v.newval)),
+               _day_change.OnDispatcher().Subscribe(v => _animation.animate_day_change(v.newval > v.oldval)),
+               Disposable.Create(() => _animation.AnimationsChanged -= on_animations_changed)
+            );
+         }
+
+         public event Action invalidated;
+
+         #region Bound Properties
+         public bool inside_bids => _inside_bids.value;
+         public bool inside_asks => _inside_asks.value;
+         public int position => _position.value;
+         public double raw_day_position => _raw_day_position;
+         public double last_price => _last_price.Value;
+         public bool last_price_from_spread => _last_price_from_spread.Value;
+         public uint last_size => _last_size.value;
+         public double bid_price => _bid_price.Value;
+         public uint bid_direct_size => _bid_direct_size.Value;
+         public uint bid_implied_size => _bid_merged_size.Value - _bid_direct_size.Value;
+         public uint bid_merged_size => _bid_merged_size.Value;
+         public string bid_merged_size_text => _bid_merged_size_text ?? (_bid_merged_size_text = _size_formatter(_bid_merged_size.Value, bid_implied_size));
+         public double ask_price => _ask_price.Value;
+         public uint ask_direct_size => _ask_direct_size.Value;
+         public uint ask_implied_size => _ask_merged_size.Value - _ask_direct_size.Value;
+         public uint ask_merged_size => _ask_merged_size.Value;
+         public string ask_merged_size_text => _ask_merged_size_text ?? (_ask_merged_size_text = _size_formatter(_ask_merged_size.Value, ask_implied_size));
+         public ulong volume => _volume.value;
+         public double settlement => _settlement.Value;
+         public double day_change => _day_change.Value;
+         public double session_low => _session_low.Value;
+         public double session_high => _session_high.Value;
+         public int working_bid_volume => _working_bid_volume.value;
+         public int working_ask_volume => _working_ask_volume.value;
+         public string working_bids_text => OrderUtils.order_collection_summary(_order_summarizer.best_working_bid, _order_summarizer.best_working_bid_qty, _order_summarizer.working_bid_qty, security.ContractSpec);
+         public string working_asks_text => OrderUtils.order_collection_summary(_order_summarizer.best_working_ask, _order_summarizer.best_working_ask_qty, _order_summarizer.working_ask_qty, security.ContractSpec);
+         public double working_best_bid => _working_best_bid.value;
+         public double working_best_ask => _working_best_ask.value;
+         // ice props
+         public string market_description => security.market_description();
+         public string cleared_alias => security.cleared_alias();
+         public string product_description => security.product_name();
+         public string hub_alias => security.hub_alias();
+         public string strip_name => security.strip_name();
+         public string mic_code => security.mic_code();
+         // animations
+         public IList<ui.spark.CellAnimation> animations => _animation.get_animations().ToList();
          public ObservableCollection<RowAnimation> row_animations { get; } = new ObservableCollection<RowAnimation>();
-
-         public IReadOnlyList<DIYGrid.IGridCellStore> cell_stores => new DIYGrid.IGridCellStore[0];
-         #endregion
-
-         public void raise_invalidated() => invalidated?.Invoke();
-
-         public void refresh() {
-            bool dirty = false;
-            if (md.md.get_bbo(ref _bbo_ref, out NativePriceSize b, out NativePriceSize a)) {
-               _bid_price.set(b.price);
-               refresh_size(ref b, _bid_direct_size, _bid_merged_size, ref _bid_merged_size_text, nameof(bid_merged_size_text));
-               _ask_price.set(a.price);
-               refresh_size(ref a, _ask_direct_size, _ask_merged_size, ref _ask_merged_size_text, nameof(ask_merged_size_text));
-               _inside_bids.on_next_distinct(_bid_price.Value >= _working_best_bid.value, ref dirty);
-               _inside_asks.on_next_distinct(_ask_price.Value <= _working_best_ask.value, ref dirty);
-            }
-            if (md.md.get_last_trade(ref _last_ref, out NativePriceSize ps, out DateTime ts, out ulong v, out bool from_spread)) {               
-               _last_price.set(ps.price);
-               _last_price_from_spread.set(from_spread);
-               _last_size.on_next_distinct(ps.size, ref dirty);
-               _volume.on_next_distinct(v, ref dirty);
-            }
-            dirty |= _settlement.set_new(md.md.settlement_price);
-            dirty |= _session_low.set_new(md.md.session_low);
-            dirty |= _session_high.set_new(md.md.session_high);
-            dirty |= _day_change.set_new(last_price - settlement);
-            _volume.on_next_distinct(md.md.total_volume, ref dirty);
-            if (dirty)
-               invalidated?.Invoke();
-
-            void refresh_size(ref NativePriceSize nps, Property<uint> direct, Property<uint> merged, ref string text, string text_prop) {
-               if (nps.size != merged.Value || nps.direct_size != direct.Value) {
-                  merged.set(nps.size);
-                  direct.set(nps.direct_size);
-                  text = null;
-                  dirty = true;
-                  RaisePropertyChanged(text_prop);
-               }
-            }
-         }
-
-         public bool Equals(Row other) =>
-           other != null &&
-           other.security_key == security_key;
-
-         public override bool Equals(object obj) =>
-            Equals(obj as Row);
-
-         public override int GetHashCode() =>
-            security_key.GetHashCode();
-
-         #region Show Windows
-         void _show_ladder() => _bus.post(SparkTags.ShowLadderTag, md);
-
-         void _show_order_book(MarketViewView.Columns col) {
-            switch (col) {
-               case MarketViewView.Columns.WorkingBids: _show_order_book(Side.Buy); break;
-               case MarketViewView.Columns.WorkingAsks: _show_order_book(Side.Sell); break;
-            }
-         }
-
-         void _show_order_book(Side side) {
-            var ob_settings = bts.shell.orderbook.OrderBookStartupData.Default;
-            ob_settings.filters = new bts.shell.orderbook.FilterStates {
-               statuses = new[] { OrderStatus.Working },
-               sides = new[] { side },
-               security_criteria = new SecurityCriteria(security)
-            };
-            _bus.post(SparkTags.ShowOrderBookTag, ob_settings);
-         }
-
-         void _show_order_ticket(MarketViewView.Columns col) {
-            var order = _orders_model.factory.order(security);
-            switch (col) {
-               case MarketViewView.Columns.Bid:
-                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Sell), bid_price, 0u));
-                  break;
-               case MarketViewView.Columns.BidMergedSize:
-                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Sell), bid_price, bid_merged_size));
-                  break;
-               case MarketViewView.Columns.BidDirectSize:
-                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Sell), bid_price, bid_direct_size));
-                  break;
-               case MarketViewView.Columns.BidImpliedSize:
-                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Sell), bid_price, bid_implied_size));
-                  break;
-               case MarketViewView.Columns.Ask:
-                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Buy), ask_price, 0u));
-                  break;
-               case MarketViewView.Columns.AskMergedSize:
-                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Buy), ask_price, ask_merged_size));
-                  break;
-               case MarketViewView.Columns.AskDirectSize:
-                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Sell), ask_price, ask_direct_size));
-                  break;
-               case MarketViewView.Columns.AskImpliedSize:
-                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Sell), ask_price, ask_implied_size));
-                  break;
-               case MarketViewView.Columns.Last:
-                  _bus.post(SparkTags.ShowOrderTicketTag, (order, last_price, 0u));
-                  break;
-               case MarketViewView.Columns.LastSize:
-                  _bus.post(SparkTags.ShowOrderTicketTag, (order, last_price, last_size));
-                  break;
-            }
-         }
-         #endregion
-
-         #region Callbacks
-         void _on_orders(ITableUpdateTransform<OrderRecordImmutable, uint> orders) {
-            foreach (var o in orders.removed)
-               _order_summarizer.remove_order(o);
-            foreach (var o in orders.added)
-               _order_summarizer.add_order(o);
-            bool dirty = false;
-            _refresh_orders(ref dirty);
-            if (dirty)
-               invalidated?.Invoke();
-         }
-
-         void _refresh_orders(ref bool dirty) {
-            _position.on_next_distinct(_order_summarizer.position, ref dirty);
-            _working_bid_volume.on_next_distinct(_order_summarizer.working_bid_qty, ref dirty);
-            _working_ask_volume.on_next_distinct(_order_summarizer.working_ask_qty, ref dirty);
-            _working_best_bid.on_next_distinct(_order_summarizer.best_working_bid, ref dirty);
-            _working_best_ask.on_next_distinct(_order_summarizer.best_working_ask, ref dirty);
-            _inside_bids.on_next_distinct(_order_summarizer.best_working_bid >= bid_price, ref dirty);
-            _inside_asks.on_next_distinct(_order_summarizer.best_working_ask <= ask_price, ref dirty);
-
-            RaisePropertyChanged(
-               // Implied might not be ever set, so it probably does not RaisePropertyChanged either
-               // This might be part of the filter funk?
-               //nameof(bid_implied_size),
-               //nameof(ask_implied_size),
-               nameof(working_bids_text), 
-               nameof(working_asks_text)
-            );
-         }
-         #endregion
-
-         class MyOrderSummarizer {
-            SortedSetMap<double, OrderRecordImmutable> _bids, _asks;
-            int _best_bid_working_qty, _best_ask_working_qty, _position, _working_bid_qty, _working_ask_qty;
-
-            public MyOrderSummarizer() {
-               _bids = new SortedSetMap<double, OrderRecordImmutable>(
-                  OrderDataHelper.CompareUtils.bid_price_comparer,
-                  OrderDataHelper.CompareUtils.order_record_id_eq_comparer);
-               _asks = new SortedSetMap<double, OrderRecordImmutable>(
-                  OrderDataHelper.CompareUtils.ask_price_comparer,
-                  OrderDataHelper.CompareUtils.order_record_id_eq_comparer);
-            }
-
-            // bid
-            public double best_working_bid => _bids.Count > 0 ? _bids.at_index(0).Key : double.NaN;
-            public int best_working_bid_qty => _best_bid_working_qty;
-            public int working_bid_qty => _working_bid_qty;
-            // ask
-            public double best_working_ask => _asks.Count > 0 ? _asks.at_index(0).Key : double.NaN;
-            public int best_working_ask_qty => _best_ask_working_qty;
-            public int working_ask_qty => _working_ask_qty;
-            // pos
-            public int position => _position;
-
-            public void add_order(OrderRecordImmutable o) {
-               switch (o.side()) {
-                  case Side.Buy:
-                     _position += (int)o.filled();
-                     if (o.is_active()) {
-                        _working_bid_qty += o.working_quantity();
-                        _bids.add(o.price(), o);
-                     }
-                     break;
-                  case Side.Sell:
-                     _position -= (int)o.filled();
-                     if (o.is_active()) {
-                        _working_ask_qty += o.working_quantity();
-                        _asks.add(o.price(), o);
-                     }
-                     break;
-                  default: return;
-               }
-               _refresh();
-            }
-
-            public void remove_order(OrderRecordImmutable o) {
-               switch (o.side()) {
-                  case Side.Buy:
-                     _position -= (int)o.filled();
-                     if (o.is_active()) {
-                        _working_bid_qty -= o.working_quantity();
-                        _bids.remove(o.price(), o);
-                     }
-                     break;
-                  case Side.Sell:
-                     _position += (int)o.filled();
-                     if (o.is_active()) {
-                        _working_ask_qty -= o.working_quantity();
-                        _asks.remove(o.price(), o);
-                     }
-                     break;
-                  default: return;
-               }
-               _refresh();
-            }
-
-            void _refresh() {
-               _best_bid_working_qty = _bids.Count > 0 ? _bids.at_index(0).Sum(o => o.working_quantity()) : 0;
-               _best_ask_working_qty = _asks.Count > 0 ? _asks.at_index(0).Sum(o => o.working_quantity()) : 0;
-            }
-         }
-      }
-   }
+         // commands
+         public ICommand bid_price_click { get; }
+         public ICommand bid_merged_size_click { get; }
+         public ICommand bid_direct_size_click { get; }
+         public ICommand bid_implied_size_click { get; }
+         public ICommand working_bids_click { get; }
+         public ICommand ask_price_click { get; }
+         public ICommand ask_merged_size_click { get; }
+         public ICommand ask_direct_size_click { get; }
+         public ICommand ask_implied_size_click { get; }
+         public ICommand working_asks_click { get; }
+         public ICommand default_command { get; }
+
+         public IReadOnlyList<DIYGrid.IGridCellStore> cell_stores => new DIYGrid.IGridCellStore[0];
+         #endregion
+
+         public void raise_invalidated() => invalidated?.Invoke();
+
+         public void refresh() {
+            bool dirty = false;
+            if (md.md.get_bbo(ref _bbo_ref, out NativePriceSize b, out NativePriceSize a)) {
+               _bid_price.set(b.price);
+               refresh_size(ref b, _bid_direct_size, _bid_merged_size, ref _bid_merged_size_text, nameof(bid_merged_size_text));
+               _ask_price.set(a.price);
+               refresh_size(ref a, _ask_direct_size, _ask_merged_size, ref _ask_merged_size_text, nameof(ask_merged_size_text));
+               _inside_bids.on_next_distinct(_bid_price.Value >= _working_best_bid.value, ref dirty);
+               _inside_asks.on_next_distinct(_ask_price.Value <= _working_best_ask.value, ref dirty);
+            }
+            if (md.md.get_last_trade(ref _last_ref, out NativePriceSize ps, out DateTime ts, out ulong v, out bool from_spread)) {               
+               _last_price.set(ps.price);
+               _last_price_from_spread.set(from_spread);
+               _last_size.on_next_distinct(ps.size, ref dirty);
+               _volume.on_next_distinct(v, ref dirty);
+            }
+            dirty |= _settlement.set_new(md.md.settlement_price);
+            dirty |= _session_low.set_new(md.md.session_low);
+            dirty |= _session_high.set_new(md.md.session_high);
+            dirty |= _day_change.set_new(last_price - settlement);
+            _volume.on_next_distinct(md.md.total_volume, ref dirty);
+            if (dirty)
+               invalidated?.Invoke();
+
+            void refresh_size(ref NativePriceSize nps, Property<uint> direct, Property<uint> merged, ref string text, string text_prop) {
+               if (nps.size != merged.Value || nps.direct_size != direct.Value) {
+                  merged.set(nps.size);
+                  direct.set(nps.direct_size);
+                  text = null;
+                  dirty = true;
+                  RaisePropertyChanged(text_prop);
+               }
+            }
+         }
+
+         public bool Equals(Row other) =>
+           other != null &&
+           other.security_key == security_key;
+
+         public override bool Equals(object obj) =>
+            Equals(obj as Row);
+
+         public override int GetHashCode() =>
+            security_key.GetHashCode();
+
+         #region Show Windows
+         void _show_ladder() => _bus.post(SparkTags.ShowLadderTag, md);
+
+         void _show_order_book(MarketViewView.Columns col) {
+            switch (col) {
+               case MarketViewView.Columns.WorkingBids: _show_order_book(Side.Buy); break;
+               case MarketViewView.Columns.WorkingAsks: _show_order_book(Side.Sell); break;
+            }
+         }
+
+         void _show_order_book(Side side) {
+            var ob_settings = bts.shell.orderbook.OrderBookStartupData.Default;
+            ob_settings.filters = new bts.shell.orderbook.FilterStates {
+               statuses = new[] { OrderStatus.Working },
+               sides = new[] { side },
+               security_criteria = new SecurityCriteria(security)
+            };
+            _bus.post(SparkTags.ShowOrderBookTag, ob_settings);
+         }
+
+         void _show_order_ticket(MarketViewView.Columns col) {
+            var order = _orders_model.factory.order(security);
+            switch (col) {
+               case MarketViewView.Columns.Bid:
+                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Sell), bid_price, 0u));
+                  break;
+               case MarketViewView.Columns.BidMergedSize:
+                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Sell), bid_price, bid_merged_size));
+                  break;
+               case MarketViewView.Columns.BidDirectSize:
+                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Sell), bid_price, bid_direct_size));
+                  break;
+               case MarketViewView.Columns.BidImpliedSize:
+                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Sell), bid_price, bid_implied_size));
+                  break;
+               case MarketViewView.Columns.Ask:
+                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Buy), ask_price, 0u));
+                  break;
+               case MarketViewView.Columns.AskMergedSize:
+                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Buy), ask_price, ask_merged_size));
+                  break;
+               case MarketViewView.Columns.AskDirectSize:
+                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Sell), ask_price, ask_direct_size));
+                  break;
+               case MarketViewView.Columns.AskImpliedSize:
+                  _bus.post(SparkTags.ShowOrderTicketTag, (order.with(side:Side.Sell), ask_price, ask_implied_size));
+                  break;
+               case MarketViewView.Columns.Last:
+                  _bus.post(SparkTags.ShowOrderTicketTag, (order, last_price, 0u));
+                  break;
+               case MarketViewView.Columns.LastSize:
+                  _bus.post(SparkTags.ShowOrderTicketTag, (order, last_price, last_size));
+                  break;
+            }
+         }
+         #endregion
+
+         #region Callbacks
+         void _on_orders(ITableUpdateTransform<OrderRecordImmutable, uint> orders) {
+            foreach (var o in orders.removed)
+               _order_summarizer.remove_order(o);
+            foreach (var o in orders.added)
+               _order_summarizer.add_order(o);
+            bool dirty = false;
+            _refresh_orders(ref dirty);
+            if (dirty)
+               invalidated?.Invoke();
+         }
+
+         void _refresh_orders(ref bool dirty) {
+            _position.on_next_distinct(_order_summarizer.position, ref dirty);
+            _working_bid_volume.on_next_distinct(_order_summarizer.working_bid_qty, ref dirty);
+            _working_ask_volume.on_next_distinct(_order_summarizer.working_ask_qty, ref dirty);
+            _working_best_bid.on_next_distinct(_order_summarizer.best_working_bid, ref dirty);
+            _working_best_ask.on_next_distinct(_order_summarizer.best_working_ask, ref dirty);
+            _inside_bids.on_next_distinct(_order_summarizer.best_working_bid >= bid_price, ref dirty);
+            _inside_asks.on_next_distinct(_order_summarizer.best_working_ask <= ask_price, ref dirty);
+
+            RaisePropertyChanged(
+               // Implied might not be ever set, so it probably does not RaisePropertyChanged either
+               // This might be part of the filter funk?
+               //nameof(bid_implied_size),
+               //nameof(ask_implied_size),
+               nameof(working_bids_text), 
+               nameof(working_asks_text)
+            );
+         }
+         #endregion
+
+         class MyOrderSummarizer {
+            SortedSetMap<double, OrderRecordImmutable> _bids, _asks;
+            int _best_bid_working_qty, _best_ask_working_qty, _position, _working_bid_qty, _working_ask_qty;
+
+            public MyOrderSummarizer() {
+               _bids = new SortedSetMap<double, OrderRecordImmutable>(
+                  OrderDataHelper.CompareUtils.bid_price_comparer,
+                  OrderDataHelper.CompareUtils.order_record_id_eq_comparer);
+               _asks = new SortedSetMap<double, OrderRecordImmutable>(
+                  OrderDataHelper.CompareUtils.ask_price_comparer,
+                  OrderDataHelper.CompareUtils.order_record_id_eq_comparer);
+            }
+
+            // bid
+            public double best_working_bid => _bids.Count > 0 ? _bids.at_index(0).Key : double.NaN;
+            public int best_working_bid_qty => _best_bid_working_qty;
+            public int working_bid_qty => _working_bid_qty;
+            // ask
+            public double best_working_ask => _asks.Count > 0 ? _asks.at_index(0).Key : double.NaN;
+            public int best_working_ask_qty => _best_ask_working_qty;
+            public int working_ask_qty => _working_ask_qty;
+            // pos
+            public int position => _position;
+
+            public void add_order(OrderRecordImmutable o) {
+               switch (o.side()) {
+                  case Side.Buy:
+                     _position += (int)o.filled();
+                     if (o.is_active()) {
+                        _working_bid_qty += o.working_quantity();
+                        _bids.add(o.price(), o);
+                     }
+                     break;
+                  case Side.Sell:
+                     _position -= (int)o.filled();
+                     if (o.is_active()) {
+                        _working_ask_qty += o.working_quantity();
+                        _asks.add(o.price(), o);
+                     }
+                     break;
+                  default: return;
+               }
+               _refresh();
+            }
+
+            public void remove_order(OrderRecordImmutable o) {
+               switch (o.side()) {
+                  case Side.Buy:
+                     _position -= (int)o.filled();
+                     if (o.is_active()) {
+                        _working_bid_qty -= o.working_quantity();
+                        _bids.remove(o.price(), o);
+                     }
+                     break;
+                  case Side.Sell:
+                     _position += (int)o.filled();
+                     if (o.is_active()) {
+                        _working_ask_qty -= o.working_quantity();
+                        _asks.remove(o.price(), o);
+                     }
+                     break;
+                  default: return;
+               }
+               _refresh();
+            }
+
+            void _refresh() {
+               _best_bid_working_qty = _bids.Count > 0 ? _bids.at_index(0).Sum(o => o.working_quantity()) : 0;
+               _best_ask_working_qty = _asks.Count > 0 ? _asks.at_index(0).Sum(o => o.working_quantity()) : 0;
+            }
+         }
+      }
+   }
 }

src/client/spark/Modules/Market/MarketView/MarketViewVM.cs

@@ -1,286 +1,294 @@
-using bts.api;
-using bts.collections;
-using bts.utils;
-using mbts.persist;
-using mbts.spark.filters;
-using mbts.ui;
-using mbts.ui.settings;
-using mbts.ui.utility;
-using System;
-using System.Collections.Generic;
-using System.Collections.ObjectModel;
-using System.Reactive.Linq;
-using System.Windows.Data;
-using System.Windows.Media;
-using static bts.shell.MarketViewWindowFactory;
-using static mbts.ui.controls.DIYGrid;
-using static mbts.ui.spark.MarketViewView;
-
-namespace mbts.spark {
-   partial class MarketViewVM : SecurityMDGrid<MarketViewVM.Row, SecurityMD> {
-      readonly IBus _bus;
-      readonly OrdersModel _orders_model;
-      IWindowSettingsManager<GridSettings> _settings_manager;
-
-      public MarketViewVM(IBus bus, OrdersModel orders_model, IObservable<StartupData> state, IWindowSettingsManager<GridSettings> settings_manager )
-         : this(bus, orders_model, state, state.SelectManyInSeries(s => s.security_criteria.subscribe_md(bus)), settings_manager) { }
-
-      // securities_connectable should NOT be connected already.
-      MarketViewVM(IBus bus, OrdersModel orders_model, IObservable<StartupData> state, IObservable<IEnumerable<SecurityMD>> securities_source, IWindowSettingsManager<GridSettings> settings_manager)
-         : base(RXCollection.Create(securities_source, SecurityDataHelper.EqualityComparer.MDComparer), null) {
-         _bus = bus;
-         _orders_model = orders_model;
-         security_filter = new RegexFilterVM<Row>(r => r.security.ToString(), nameof(Row.security));
-         _settings_manager = settings_manager;
-         // Columns
-         const string merged_size_col_header = "Qty", direct_size_col_header = "Direct", implied_size_col_header = "Implied", working_col_header = "Working";
-         const double working_orders_col_width = 105;
-         IMultiValueConverter price_converter = new ui.spark.converters.TickDisplayConverter();
-         add_col(Columns.Security, "Security", r => r.security.ToString(), nameof(Row.security), DefaultColumnWidths.Security, security_filter, DataGridFilterKnownTemplates.Regex);
-         add_col(Columns.Volume, "Volume", r => r.volume.ToString(), nameof(Row.volume), DefaultColumnWidths.Volume, volume_filter, DataGridFilterKnownTemplates.ULongMinMax);
-         add_col(Columns.LastSpd, "Spd", r => r.last_price_from_spread ? "S" : "", nameof(Row.last_price_from_spread), DefaultColumnWidths.Side, last_price_from_spread_filter, DataGridFilterKnownTemplates.Regex);
-         add_col(Columns.Last, "Last", r => r.format_price(r.last_price), nameof(Row.last_price), DefaultColumnWidths.Price, last_price_filter, DataGridFilterKnownTemplates.DoubleMinMax);         
-         add_col(Columns.LastSize, merged_size_col_header, r => r.last_size.ToString(), nameof(Row.last_size), DefaultColumnWidths.Size, last_size_filter, DataGridFilterKnownTemplates.UIntMinMax, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_col(Columns.Position, "Pos", r => r.position.ToString(), nameof(Row.position), DefaultColumnWidths.Size, position_filter, DataGridFilterKnownTemplates.IntMinMax, true, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_col(Columns.RawDayPosition, "Raw Day Pos", r => SparkUtils.fmt_position(r.raw_day_position), nameof(Row.raw_day_position), DefaultColumnWidths.Size, raw_day_position_filter, DataGridFilterKnownTemplates.DoubleMinMax, true, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_hl_working_orders_col(Columns.WorkingBids, working_col_header, nameof(Row.working_bids_text), nameof(Row.working_best_bid), nameof(Row.working_bids_click), working_bid_volume_filter, BTSColors.MarketData.bid, System.Windows.HorizontalAlignment.Right);
-         add_col(Columns.BidImpliedSize, implied_size_col_header, r => r.bid_implied_size.ToString(), nameof(Row.bid_implied_size), DefaultColumnWidths.ImpliedDirectMergedSize, bid_implied_size_filter, DataGridFilterKnownTemplates.UIntMinMax, font_color: BTSColors.MarketData.bid, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_col(Columns.BidDirectSize, direct_size_col_header, r => r.bid_direct_size.ToString(), nameof(Row.bid_direct_size), DefaultColumnWidths.ImpliedDirectMergedSize, bid_direct_size_filter, DataGridFilterKnownTemplates.UIntMinMax, font_color: BTSColors.MarketData.bid, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_hl_merged_size_col(Columns.BidMergedSize, merged_size_col_header, nameof(Row.bid_merged_size), nameof(Row.bid_merged_size_text), nameof(Row.bid_merged_size_click), bid_merged_size_filter, BTSColors.MarketData.bid, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_hl_price_col(Columns.Bid, "Bid", nameof(Row.bid_price), nameof(Row.bid_price_click), bid_price_filter, BTSColors.header_text, BTSColors.MarketData.bid, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_hl_price_col(Columns.Ask, "Ask", nameof(Row.ask_price), nameof(Row.ask_price_click), ask_price_filter, BTSColors.header_text, BTSColors.MarketData.ask, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_hl_merged_size_col(Columns.AskMergedSize, merged_size_col_header, nameof(Row.ask_merged_size), nameof(Row.ask_merged_size_text), nameof(Row.ask_merged_size_click), ask_merged_size_filter, BTSColors.MarketData.ask, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_col(Columns.AskDirectSize, direct_size_col_header, r => r.ask_direct_size.ToString(), nameof(Row.ask_direct_size), DefaultColumnWidths.ImpliedDirectMergedSize, ask_direct_size_filter, DataGridFilterKnownTemplates.UIntMinMax, font_color: BTSColors.MarketData.ask, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_col(Columns.AskImpliedSize, implied_size_col_header, r => r.ask_implied_size.ToString(), nameof(Row.ask_implied_size), DefaultColumnWidths.ImpliedDirectMergedSize, ask_implied_size_filter, DataGridFilterKnownTemplates.UIntMinMax, font_color: BTSColors.MarketData.ask, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_hl_working_orders_col(Columns.WorkingAsks, working_col_header, nameof(Row.working_asks_text), nameof(Row.working_best_ask), nameof(Row.working_asks_click), working_ask_volume_filter, BTSColors.MarketData.ask, System.Windows.HorizontalAlignment.Right);
-         add_col(Columns.Settlement, "Settlement", r => r.format_price(r.settlement), nameof(Row.settlement), DefaultColumnWidths.Settlement, settlement_filter, DataGridFilterKnownTemplates.DoubleMinMax, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_col(Columns.DayChange, "Change", r => r.format_price(r.day_change), nameof(Row.day_change), DefaultColumnWidths.Price, day_change_filter, DataGridFilterKnownTemplates.DoubleMinMax, true, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_col(Columns.SessionLow, "Low", r => r.format_price(r.session_low), nameof(Row.session_low), DefaultColumnWidths.Price, session_low_filter, DataGridFilterKnownTemplates.DoubleMinMax, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_col(Columns.SessionHigh, "High", r => r.format_price(r.session_high), nameof(Row.session_high), DefaultColumnWidths.Price, session_high_filter, DataGridFilterKnownTemplates.DoubleMinMax, h_alignment: System.Windows.HorizontalAlignment.Right);
-         add_col(Columns.SecurityKey, "Security Key", r => r.security_key.ToString(), nameof(Row.security_key), DefaultColumnWidths.SecurityKey, security_key_filter, DataGridFilterKnownTemplates.Regex);
-
-         add_col(Columns.MarketDescription, "Market Description", r => r.security.market_description(), nameof(Row.market_description), DefaultColumnWidths.SecurityKey, market_description_filter, DataGridFilterKnownTemplates.Regex);
-         add_col(Columns.ClearedAlias, "Cleared Alias", r => r.security.cleared_alias(), nameof(Row.cleared_alias), DefaultColumnWidths.SecurityKey, cleared_alias_filter, DataGridFilterKnownTemplates.Regex);
-         add_col(Columns.ProductDescription, "Product Description", r => r.security.product_name(), nameof(Row.product_description), DefaultColumnWidths.SecurityKey, product_description_filter, DataGridFilterKnownTemplates.Regex);
-         add_col(Columns.HubAlias, "Hub Alias", r => r.security.hub_alias(), nameof(Row.hub_alias), DefaultColumnWidths.SecurityKey, hub_alias_filter, DataGridFilterKnownTemplates.Regex);
-         add_col(Columns.StripName, "Strip Name", r => r.security.strip_name(), nameof(Row.strip_name), DefaultColumnWidths.SecurityKey, strip_name_filter, DataGridFilterKnownTemplates.Regex);
-         add_col(Columns.MicCode, "Mic Code", r => r.security.mic_code(), nameof(Row.mic_code), DefaultColumnWidths.SecurityKey, mic_code_filter, DataGridFilterKnownTemplates.Regex);
-         //
-         void add_col(Columns col_key, string header, Func<Row, string> get_text, string prop, double width, IDataGridColumnFilter filter, string filter_template, bool is_animatable = false, Color? font_color = null, System.Windows.HorizontalAlignment h_alignment = System.Windows.HorizontalAlignment.Left) =>
-            columns.Add(new RenderTextColumn<Row, Columns>(col_key, get_text, filter, filter_template) {
-               // it's necessary to have this id populated for grid settings
-               id = bts.shell.MarketViewWindowFactory.col_id(col_key),
-               Header = header,
-               sort_member_path = prop,
-               Width = width,
-               padding = is_animatable ? AnimatableCellPadding : DefaultCellPadding,
-               font_color_override = font_color,
-               horizontal_alignment = h_alignment
-            });
-
-         void add_hl_price_col(Columns col_key, string header, string prop, string command_prop, IDataGridColumnFilter filter, Color foreground, Color? background, System.Windows.HorizontalAlignment h_alignment) =>
-            add_hl_col(col_key, header, price_binding(prop), prop, command_prop, filter, DataGridFilterKnownTemplates.DoubleMinMax, DefaultColumnWidths.MViewPrice, foreground, background, h_alignment);
-
-         void add_hl_merged_size_col(Columns col_key, string header, string sort_prop, string text_prop, string command_prop, IDataGridColumnFilter filter, Color foreground, System.Windows.HorizontalAlignment h_alignment) =>
-            add_hl_col(col_key, header, new Binding(text_prop), sort_prop, command_prop, filter, DataGridFilterKnownTemplates.UIntMinMax, DefaultColumnWidths.ImpliedDirectMergedSize, foreground, null, h_alignment);
-
-         void add_hl_working_orders_col(Columns col_key, string header, string text_prop, string sort_prop, string command_prop, IDataGridColumnFilter filter, Color foreground, System.Windows.HorizontalAlignment h_alignment) =>
-            add_hl_col(col_key, header, new Binding(text_prop), sort_prop, command_prop, filter, DataGridFilterKnownTemplates.IntMinMax, working_orders_col_width, foreground, null, h_alignment);
-
-         void add_hl_col(
-            Columns col_key, string header, BindingBase binding, string prop, string command_prop,
-            IDataGridColumnFilter filter, string filter_template, double width, Color foreground, Color? background, System.Windows.HorizontalAlignment h_alignment = System.Windows.HorizontalAlignment.Left) =>
-         columns.Add(new HyperlinkElementColumn<Row, Columns>(
-            col_key, binding, new Binding(command_prop), filter, filter_template) {
-            // it's necessary to have this id populated for grid settings
-            id = bts.shell.MarketViewWindowFactory.col_id(col_key),
-            Header = header,
-            sort_member_path = prop,
-            font_color_override = foreground,
-            background_color_override = background,
-            Width = DefaultColumnWidths.Price,
-            horizontal_alignment = h_alignment
-         });
-
-         BindingBase price_binding(string prop) {
-            var mb = new MultiBinding { Converter = price_converter };
-            mb.Bindings.Add(new Binding(prop) { Mode = BindingMode.OneWay });
-            mb.Bindings.Add(new Binding($"{nameof(Row.security)}.{nameof(SecurityImmutable.ContractSpec)}") { Mode = BindingMode.OneWay });
-            return mb;
-         }
-         //
-         // make font-color immutable for Side, Size, (the font-color rule is hard-coded: MarketData.bid for Buy orders, MarketData.ask for Sells)
-         var col_settings_flags = new Dictionary<string, GridColumnSettingsFeatureFlags> {
-            { Columns.Bid.ToString(), GridColumnSettingsFeatureFlags.ImmutableFontColor | GridColumnSettingsFeatureFlags.ImmutableBackgroundColor },
-            { Columns.Ask.ToString(), GridColumnSettingsFeatureFlags.ImmutableFontColor | GridColumnSettingsFeatureFlags.ImmutableBackgroundColor},
-            { Columns.WorkingAsks.ToString(), GridColumnSettingsFeatureFlags.ImmutableFontColor },
-            { Columns.WorkingBids.ToString(), GridColumnSettingsFeatureFlags.ImmutableFontColor }
-         };
-         grid_settings_manager = new SimpleGridSettingsManager<Row>(columns, col_settings_flags, row_vms);
-         // this will initialize the immutable properties that correspond to the col_settings_flags above (e.g. Bid/Ask font_color_override, etc.)
-         grid_settings_manager.load_from(settings_manager.current_settings());
-         //
-         _resources.AddRange(
-            state.OnDispatcher().Subscribe(s => {
-               using (rows_view.DeferRefresh()) {
-                  if (s.grid != null) {
-                     // this means we are not using defaults,
-                     // Note: settings_manager in turn notifies grid_settings_manager of this update
-                     _settings_manager.instance_settings = s.grid;
-                  }
-                  if (s.filters is FilterStates filters) {
-                     security_filter.load_state(filters.security);
-                     bid_price_filter.load_state(filters.bid_price);
-                     bid_merged_size_filter.load_state(filters.bid_merged_size);
-                     bid_direct_size_filter.load_state(filters.bid_direct_size);
-                     bid_implied_size_filter.load_state(filters.bid_implied_size);
-                     ask_price_filter.load_state(filters.ask_price);
-                     ask_merged_size_filter.load_state(filters.ask_merged_size);
-                     ask_direct_size_filter.load_state(filters.ask_direct_size);
-                     ask_implied_size_filter.load_state(filters.ask_implied_size);
-                     last_price_filter.load_state(filters.last_price);
-                     last_price_from_spread_filter.load_state(filters.last_price_from_spread);
-                     last_size_filter.load_state(filters.last_size);
-                     day_change_filter.load_state(filters.day_change);
-                     settlement_filter.load_state(filters.settlement);
-                     session_low_filter.load_state(filters.session_low);
-                     session_high_filter.load_state(filters.session_high);
-                     position_filter.load_state(filters.position);
-                     raw_day_position_filter.load_state(filters.raw_day_position);
-                     working_bid_volume_filter.load_state(filters.working_bid_volume);
-                     working_ask_volume_filter.load_state(filters.working_ask_volume);
-                     volume_filter.load_state(filters.volume);
-                     security_key_filter.load_state(filters.security_key);
-                  }
-                  rows_view.set_sort_descriptions(s.sorting_criterion, columns);
-               }
-            }), DataGridFilterHelper.hook_filters(rows_view, security_filter,
-            bid_price_filter, bid_merged_size_filter, bid_direct_size_filter, bid_implied_size_filter,
-            ask_price_filter, ask_merged_size_filter, ask_direct_size_filter, ask_implied_size_filter,
-            last_price_filter, last_price_from_spread_filter, last_size_filter, day_change_filter, settlement_filter,
-            session_low_filter, session_high_filter, position_filter, raw_day_position_filter, working_bid_volume_filter, working_ask_volume_filter, volume_filter, security_key_filter),
-            settings_manager.subscribe_current().OnDispatcher().Subscribe(grid_settings_manager),
-            WorkLoop.Start("MarketView", _refresh)
-         );
-      }
-
-      public ObservableCollection<ColumnBase> columns { get; } = new ObservableCollection<ColumnBase>();
-
-      public SimpleGridSettingsManager<Row> grid_settings_manager { get; }
-
-      public GridSettings get_grid_save_state() {
-         if (_settings_manager.instance_settings == null)
-            return null;
-         else
-            return grid_settings_manager.create();
-      }
-
-      public static IComparer<Row> get_column_comparer(Columns column) {
-         switch (column) {
-            case Columns.Security: return wrap(r => r.security);
-            case Columns.SecurityKey: return wrap(r => r.security.Id);
-            case Columns.Volume: return wrap(r => r.volume);
-            case Columns.Last: return wrap(r => r.last_price);
-            case Columns.LastSize: return wrap(r => r.last_size);
-            case Columns.Position: return wrap(r => r.position);
-            case Columns.WorkingBids: return wrap(r => r.working_bid_volume);
-            case Columns.BidImpliedSize: return wrap(r => r.bid_implied_size);
-            case Columns.BidDirectSize: return wrap(r => r.bid_direct_size);
-            case Columns.BidMergedSize: return wrap(r => r.bid_merged_size);
-            case Columns.Bid: return wrap(r => r.bid_price);
-            case Columns.Ask: return wrap(r => r.ask_price);
-            case Columns.AskMergedSize: return wrap(r => r.ask_merged_size);
-            case Columns.AskDirectSize: return wrap(r => r.ask_direct_size);
-            case Columns.AskImpliedSize: return wrap(r => r.ask_implied_size);
-            case Columns.WorkingAsks: return wrap(r => r.working_ask_volume);
-            case Columns.Settlement: return wrap(r => r.settlement);
-            case Columns.DayChange: return wrap(r => r.day_change);
-            case Columns.SessionLow: return wrap(r => r.session_low);
-            case Columns.SessionHigh: return wrap(r => r.session_high);
-            case Columns.MarketDescription: return wrap(r => r.market_description);
-            case Columns.ClearedAlias: return wrap(r => r.security.cleared_alias());
-            case Columns.ProductDescription: return wrap(r => r.security.product_name());
-            case Columns.HubAlias: return wrap(r => r.security.hub_alias());
-            case Columns.StripName: return wrap(r => r.security.strip_name());
-            case Columns.MicCode: return wrap(r => r.security.mic_code());
-            default: return null;
-         }
-         IComparer<Row> wrap<T>(Func<Row, T> selector) =>
-            Comparer<T>.Default.wrap(selector);
-      }
-
-      #region Filters
-      public IDataGridColumnPersistentFilter<Row, string> security_filter { get; }
-      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> bid_price_filter { get; } = new DoubleMinMax<Row>(r => r.bid_price, nameof(Row.bid_price));
-      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> bid_merged_size_filter { get; } = new UIntMinMax<Row>(r => r.bid_merged_size, nameof(Row.bid_merged_size));
-      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> bid_direct_size_filter { get; } = new UIntMinMax<Row>(r => r.bid_direct_size, nameof(Row.bid_direct_size));
-      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> bid_implied_size_filter { get; } = new UIntMinMax<Row>(r => r.bid_implied_size, nameof(Row.bid_implied_size));
-      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> ask_price_filter { get; } = new DoubleMinMax<Row>(r => r.ask_price, nameof(Row.ask_price));
-      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> ask_merged_size_filter { get; } = new UIntMinMax<Row>(r => r.ask_merged_size, nameof(Row.ask_merged_size));
-      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> ask_direct_size_filter { get; } = new UIntMinMax<Row>(r => r.ask_direct_size, nameof(Row.ask_direct_size));
-      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> ask_implied_size_filter { get; } = new UIntMinMax<Row>(r => r.ask_implied_size, nameof(Row.ask_implied_size));
-      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> last_price_filter { get; } = new DoubleMinMax<Row>(r => r.last_price, nameof(Row.last_price));
-      public IDataGridColumnPersistentFilter<Row, string> last_price_from_spread_filter { get; } = new RegexFilterVM<Row>(r => r.last_price_from_spread.ToString(), nameof(Row.last_price_from_spread));
-      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> last_size_filter { get; } = new UIntMinMax<Row>(r => r.last_size, nameof(Row.last_size));
-      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> day_change_filter { get; } = new DoubleMinMax<Row>(r => r.day_change, nameof(Row.day_change));
-      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> settlement_filter { get; } = new DoubleMinMax<Row>(r => r.settlement, nameof(Row.settlement));
-      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> session_low_filter { get; } = new DoubleMinMax<Row>(r => r.session_low, nameof(Row.session_low));
-      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> session_high_filter { get; } = new DoubleMinMax<Row>(r => r.session_high, nameof(Row.session_high));
-      public IDataGridColumnPersistentFilter<Row, IntMinMaxState> position_filter { get; } = new IntMinMax<Row>(r => r.position, nameof(Row.position));
-      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> raw_day_position_filter { get; } = new DoubleMinMax<Row>(r => r.raw_day_position, nameof(Row.raw_day_position));
-      public IDataGridColumnPersistentFilter<Row, IntMinMaxState> working_bid_volume_filter { get; } = new IntMinMax<Row>(r => r.working_bid_volume, nameof(Row.working_bid_volume));
-      public IDataGridColumnPersistentFilter<Row, IntMinMaxState> working_ask_volume_filter { get; } = new IntMinMax<Row>(r => r.working_ask_volume, nameof(Row.working_ask_volume));
-      public IDataGridColumnPersistentFilter<Row, ULongMinMaxState> volume_filter { get; } = new ULongMinMax<Row>(r => r.volume, nameof(Row.volume));
-      public IDataGridColumnPersistentFilter<Row, string> security_key_filter { get; } = new RegexFilterVM<Row>(r => r.security_key.ToString(), nameof(Row.security_key));
-      
-      public IDataGridColumnPersistentFilter<Row, string> market_description_filter { get; } = new RegexFilterVM<Row>(r => r.security.HasIce ? r.security.Ice.MarketDescription : "", nameof(Row.market_description));
-      public IDataGridColumnPersistentFilter<Row, string> cleared_alias_filter { get; } = new RegexFilterVM<Row>(r => r.security.HasIce ? r.security.Ice.ClearedAlias : "", nameof(Row.cleared_alias));
-      public IDataGridColumnPersistentFilter<Row, string> product_description_filter { get; } = new RegexFilterVM<Row>(r => r.security.HasIce ? r.security.Ice.ProductName : "", nameof(Row.product_description));
-      public IDataGridColumnPersistentFilter<Row, string> hub_alias_filter { get; } = new RegexFilterVM<Row>(r => r.security.HasIce ? r.security.Ice.HubAlias : "", nameof(Row.hub_alias));
-      public IDataGridColumnPersistentFilter<Row, string> strip_name_filter { get; } = new RegexFilterVM<Row>(r => r.security.HasIce ? r.security.Ice.StripName : "", nameof(Row.strip_name));
-      public IDataGridColumnPersistentFilter<Row, string> mic_code_filter { get; } = new RegexFilterVM<Row>(r => r.security.HasIce ? r.security.Ice.MicCode : "", nameof(Row.mic_code));
-      
-
-      public FilterStates get_filter_states() {
-         return new FilterStates {
-            security = security_filter.get_state(),
-            bid_price = bid_price_filter.get_state(),
-            bid_merged_size = bid_merged_size_filter.get_state(),
-            bid_direct_size = bid_direct_size_filter.get_state(),
-            bid_implied_size = bid_implied_size_filter.get_state(),
-            ask_price = ask_price_filter.get_state(),
-            ask_merged_size = ask_merged_size_filter.get_state(),
-            ask_direct_size = ask_direct_size_filter.get_state(),
-            ask_implied_size = ask_implied_size_filter.get_state(),
-            last_price = last_price_filter.get_state(),
-            last_price_from_spread = last_price_from_spread_filter.get_state(),
-            last_size = last_size_filter.get_state(),
-            day_change = day_change_filter.get_state(),
-            settlement = settlement_filter.get_state(),
-            session_low = session_low_filter.get_state(),
-            session_high = session_high_filter.get_state(),
-            position = position_filter.get_state(),
-            raw_day_position = raw_day_position_filter.get_state(),
-            working_bid_volume = working_bid_volume_filter.get_state(),
-            working_ask_volume = working_ask_volume_filter.get_state(),
-            volume = volume_filter.get_state(),
-            security_key = security_key_filter.get_state(),
-            market_description = market_description_filter.get_state(),
-            cleared_alias = cleared_alias_filter.get_state(),
-            product_description = product_description_filter.get_state(),
-            hub_alias = hub_alias_filter.get_state(),
-            strip_name = strip_name_filter.get_state(),
-            mic_code = mic_code_filter.get_state(),
-         };
-      }
-      #endregion
-
-      protected override Row row_factory(SecurityMD smd) => new Row(_bus, smd, _orders_model);
-      protected override void on_row_removed(Row row) => row.Dispose();
-      protected override IEnumerable<SortingCriteria> get_default_sorting() { yield return new SortingCriteria(nameof(Row.security), true); }
-
-      void _refresh() {
-         foreach (var r in _row_vms) r.refresh();
-      }
-   }
-}
+using bts.api;
+using bts.collections;
+using bts.utils;
+using mbts.persist;
+using mbts.spark.filters;
+using mbts.ui;
+using mbts.ui.settings;
+using mbts.ui.utility;
+using System;
+using System.Collections.Generic;
+using System.Collections.ObjectModel;
+using System.Linq;
+using System.Reactive.Linq;
+using System.Windows.Data;
+using System.Windows.Media;
+using static bts.shell.MarketViewWindowFactory;
+using static mbts.ui.controls.DIYGrid;
+using static mbts.ui.spark.MarketViewView;
+
+namespace mbts.spark {
+   partial class MarketViewVM : SecurityMDGrid<MarketViewVM.Row, SecurityMD> {
+      static IReadOnlyList<Func<MarketViewVM, IDataGridColumnFilter<Row>>> _filter_getters =
+         ObjectSnapshot.property_delegates_of_type<MarketViewVM, IDataGridColumnFilter<Row>>();
+
+      readonly IBus _bus;
+      readonly OrdersModel _orders_model;
+      IWindowSettingsManager<GridSettings> _settings_manager;
+
+      public MarketViewVM(IBus bus, OrdersModel orders_model, IObservable<StartupData> state, IWindowSettingsManager<GridSettings> settings_manager )
+         : this(bus, orders_model, state, state.SelectManyInSeries(s => s.security_criteria.subscribe_md(bus)), settings_manager) { }
+
+      // securities_connectable should NOT be connected already.
+      MarketViewVM(IBus bus, OrdersModel orders_model, IObservable<StartupData> state, IObservable<IEnumerable<SecurityMD>> securities_source, IWindowSettingsManager<GridSettings> settings_manager)
+         : base(RXCollection.Create(securities_source, SecurityDataHelper.EqualityComparer.MDComparer), null) {
+         _bus = bus;
+         _orders_model = orders_model;
+         security_filter = new RegexFilterVM<Row>(r => r.security.ToString(), nameof(Row.security));
+         _settings_manager = settings_manager;
+         // Columns
+         const string merged_size_col_header = "Qty", direct_size_col_header = "Direct", implied_size_col_header = "Implied", working_col_header = "Working";
+         const double working_orders_col_width = 105;
+         IMultiValueConverter price_converter = new ui.spark.converters.TickDisplayConverter();
+         add_col(Columns.Security, "Security", r => r.security.ToString(), nameof(Row.security), DefaultColumnWidths.Security, security_filter, DataGridFilterKnownTemplates.Regex);
+         add_col(Columns.Volume, "Volume", r => r.volume.ToString(), nameof(Row.volume), DefaultColumnWidths.Volume, volume_filter, DataGridFilterKnownTemplates.ULongMinMax);
+         add_col(Columns.LastSpd, "Spd", r => r.last_price_from_spread ? "S" : "", nameof(Row.last_price_from_spread), DefaultColumnWidths.Side, last_price_from_spread_filter, DataGridFilterKnownTemplates.Regex);
+         add_col(Columns.Last, "Last", r => r.format_price(r.last_price), nameof(Row.last_price), DefaultColumnWidths.Price, last_price_filter, DataGridFilterKnownTemplates.DoubleMinMax);         
+         add_col(Columns.LastSize, merged_size_col_header, r => r.last_size.ToString(), nameof(Row.last_size), DefaultColumnWidths.Size, last_size_filter, DataGridFilterKnownTemplates.UIntMinMax, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_col(Columns.Position, "Pos", r => r.position.ToString(), nameof(Row.position), DefaultColumnWidths.Size, position_filter, DataGridFilterKnownTemplates.IntMinMax, true, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_col(Columns.RawDayPosition, "Raw Day Pos", r => SparkUtils.fmt_position(r.raw_day_position), nameof(Row.raw_day_position), DefaultColumnWidths.Size, raw_day_position_filter, DataGridFilterKnownTemplates.DoubleMinMax, true, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_hl_working_orders_col(Columns.WorkingBids, working_col_header, nameof(Row.working_bids_text), nameof(Row.working_best_bid), nameof(Row.working_bids_click), working_bid_volume_filter, BTSColors.MarketData.bid, System.Windows.HorizontalAlignment.Right);
+         add_col(Columns.BidImpliedSize, implied_size_col_header, r => r.bid_implied_size.ToString(), nameof(Row.bid_implied_size), DefaultColumnWidths.ImpliedDirectMergedSize, bid_implied_size_filter, DataGridFilterKnownTemplates.UIntMinMax, font_color: BTSColors.MarketData.bid, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_col(Columns.BidDirectSize, direct_size_col_header, r => r.bid_direct_size.ToString(), nameof(Row.bid_direct_size), DefaultColumnWidths.ImpliedDirectMergedSize, bid_direct_size_filter, DataGridFilterKnownTemplates.UIntMinMax, font_color: BTSColors.MarketData.bid, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_hl_merged_size_col(Columns.BidMergedSize, merged_size_col_header, nameof(Row.bid_merged_size), nameof(Row.bid_merged_size_text), nameof(Row.bid_merged_size_click), bid_merged_size_filter, BTSColors.MarketData.bid, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_hl_price_col(Columns.Bid, "Bid", nameof(Row.bid_price), nameof(Row.bid_price_click), bid_price_filter, BTSColors.header_text, BTSColors.MarketData.bid, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_hl_price_col(Columns.Ask, "Ask", nameof(Row.ask_price), nameof(Row.ask_price_click), ask_price_filter, BTSColors.header_text, BTSColors.MarketData.ask, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_hl_merged_size_col(Columns.AskMergedSize, merged_size_col_header, nameof(Row.ask_merged_size), nameof(Row.ask_merged_size_text), nameof(Row.ask_merged_size_click), ask_merged_size_filter, BTSColors.MarketData.ask, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_col(Columns.AskDirectSize, direct_size_col_header, r => r.ask_direct_size.ToString(), nameof(Row.ask_direct_size), DefaultColumnWidths.ImpliedDirectMergedSize, ask_direct_size_filter, DataGridFilterKnownTemplates.UIntMinMax, font_color: BTSColors.MarketData.ask, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_col(Columns.AskImpliedSize, implied_size_col_header, r => r.ask_implied_size.ToString(), nameof(Row.ask_implied_size), DefaultColumnWidths.ImpliedDirectMergedSize, ask_implied_size_filter, DataGridFilterKnownTemplates.UIntMinMax, font_color: BTSColors.MarketData.ask, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_hl_working_orders_col(Columns.WorkingAsks, working_col_header, nameof(Row.working_asks_text), nameof(Row.working_best_ask), nameof(Row.working_asks_click), working_ask_volume_filter, BTSColors.MarketData.ask, System.Windows.HorizontalAlignment.Right);
+         add_col(Columns.Settlement, "Settlement", r => r.format_price(r.settlement), nameof(Row.settlement), DefaultColumnWidths.Settlement, settlement_filter, DataGridFilterKnownTemplates.DoubleMinMax, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_col(Columns.DayChange, "Change", r => r.format_price(r.day_change), nameof(Row.day_change), DefaultColumnWidths.Price, day_change_filter, DataGridFilterKnownTemplates.DoubleMinMax, true, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_col(Columns.SessionLow, "Low", r => r.format_price(r.session_low), nameof(Row.session_low), DefaultColumnWidths.Price, session_low_filter, DataGridFilterKnownTemplates.DoubleMinMax, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_col(Columns.SessionHigh, "High", r => r.format_price(r.session_high), nameof(Row.session_high), DefaultColumnWidths.Price, session_high_filter, DataGridFilterKnownTemplates.DoubleMinMax, h_alignment: System.Windows.HorizontalAlignment.Right);
+         add_col(Columns.SecurityKey, "Security Key", r => r.security_key.ToString(), nameof(Row.security_key), DefaultColumnWidths.SecurityKey, security_key_filter, DataGridFilterKnownTemplates.Regex);
+
+         add_col(Columns.MarketDescription, "Market Description", r => r.security.market_description(), nameof(Row.market_description), DefaultColumnWidths.SecurityKey, market_description_filter, DataGridFilterKnownTemplates.Regex);
+         add_col(Columns.ClearedAlias, "Cleared Alias", r => r.security.cleared_alias(), nameof(Row.cleared_alias), DefaultColumnWidths.SecurityKey, cleared_alias_filter, DataGridFilterKnownTemplates.Regex);
+         add_col(Columns.ProductDescription, "Product Description", r => r.security.product_name(), nameof(Row.product_description), DefaultColumnWidths.SecurityKey, product_description_filter, DataGridFilterKnownTemplates.Regex);
+         add_col(Columns.HubAlias, "Hub Alias", r => r.security.hub_alias(), nameof(Row.hub_alias), DefaultColumnWidths.SecurityKey, hub_alias_filter, DataGridFilterKnownTemplates.Regex);
+         add_col(Columns.StripName, "Strip Name", r => r.security.strip_name(), nameof(Row.strip_name), DefaultColumnWidths.SecurityKey, strip_name_filter, DataGridFilterKnownTemplates.Regex);
+         add_col(Columns.MicCode, "Mic Code", r => r.security.mic_code(), nameof(Row.mic_code), DefaultColumnWidths.SecurityKey, mic_code_filter, DataGridFilterKnownTemplates.Regex);
+         //
+         void add_col(Columns col_key, string header, Func<Row, string> get_text, string prop, double width, IDataGridColumnFilter filter, string filter_template, bool is_animatable = false, Color? font_color = null, System.Windows.HorizontalAlignment h_alignment = System.Windows.HorizontalAlignment.Left) =>
+            columns.Add(new RenderTextColumn<Row, Columns>(col_key, get_text, filter, filter_template) {
+               // it's necessary to have this id populated for grid settings
+               id = bts.shell.MarketViewWindowFactory.col_id(col_key),
+               Header = header,
+               sort_member_path = prop,
+               Width = width,
+               padding = is_animatable ? AnimatableCellPadding : DefaultCellPadding,
+               font_color_override = font_color,
+               horizontal_alignment = h_alignment
+            });
+
+         void add_hl_price_col(Columns col_key, string header, string prop, string command_prop, IDataGridColumnFilter filter, Color foreground, Color? background, System.Windows.HorizontalAlignment h_alignment) =>
+            add_hl_col(col_key, header, price_binding(prop), prop, command_prop, filter, DataGridFilterKnownTemplates.DoubleMinMax, DefaultColumnWidths.MViewPrice, foreground, background, h_alignment);
+
+         void add_hl_merged_size_col(Columns col_key, string header, string sort_prop, string text_prop, string command_prop, IDataGridColumnFilter filter, Color foreground, System.Windows.HorizontalAlignment h_alignment) =>
+            add_hl_col(col_key, header, new Binding(text_prop), sort_prop, command_prop, filter, DataGridFilterKnownTemplates.UIntMinMax, DefaultColumnWidths.ImpliedDirectMergedSize, foreground, null, h_alignment);
+
+         void add_hl_working_orders_col(Columns col_key, string header, string text_prop, string sort_prop, string command_prop, IDataGridColumnFilter filter, Color foreground, System.Windows.HorizontalAlignment h_alignment) =>
+            add_hl_col(col_key, header, new Binding(text_prop), sort_prop, command_prop, filter, DataGridFilterKnownTemplates.IntMinMax, working_orders_col_width, foreground, null, h_alignment);
+
+         void add_hl_col(
+            Columns col_key, string header, BindingBase binding, string prop, string command_prop,
+            IDataGridColumnFilter filter, string filter_template, double width, Color foreground, Color? background, System.Windows.HorizontalAlignment h_alignment = System.Windows.HorizontalAlignment.Left) =>
+         columns.Add(new HyperlinkElementColumn<Row, Columns>(
+            col_key, binding, new Binding(command_prop), filter, filter_template) {
+            // it's necessary to have this id populated for grid settings
+            id = bts.shell.MarketViewWindowFactory.col_id(col_key),
+            Header = header,
+            sort_member_path = prop,
+            font_color_override = foreground,
+            background_color_override = background,
+            Width = DefaultColumnWidths.Price,
+            horizontal_alignment = h_alignment
+         });
+
+         BindingBase price_binding(string prop) {
+            var mb = new MultiBinding { Converter = price_converter };
+            mb.Bindings.Add(new Binding(prop) { Mode = BindingMode.OneWay });
+            mb.Bindings.Add(new Binding($"{nameof(Row.security)}.{nameof(SecurityImmutable.ContractSpec)}") { Mode = BindingMode.OneWay });
+            return mb;
+         }
+         //
+         // make font-color immutable for Side, Size, (the font-color rule is hard-coded: MarketData.bid for Buy orders, MarketData.ask for Sells)
+         var col_settings_flags = new Dictionary<string, GridColumnSettingsFeatureFlags> {
+            { Columns.Bid.ToString(), GridColumnSettingsFeatureFlags.ImmutableFontColor | GridColumnSettingsFeatureFlags.ImmutableBackgroundColor },
+            { Columns.Ask.ToString(), GridColumnSettingsFeatureFlags.ImmutableFontColor | GridColumnSettingsFeatureFlags.ImmutableBackgroundColor},
+            { Columns.WorkingAsks.ToString(), GridColumnSettingsFeatureFlags.ImmutableFontColor },
+            { Columns.WorkingBids.ToString(), GridColumnSettingsFeatureFlags.ImmutableFontColor }
+         };
+         grid_settings_manager = new SimpleGridSettingsManager<Row>(columns, col_settings_flags, row_vms);
+         // this will initialize the immutable properties that correspond to the col_settings_flags above (e.g. Bid/Ask font_color_override, etc.)
+         grid_settings_manager.load_from(settings_manager.current_settings());
+         //
+         _resources.AddRange(
+            state.OnDispatcher().Subscribe(s => {
+               using (rows_view.DeferRefresh()) {
+                  if (s.grid != null) {
+                     // this means we are not using defaults,
+                     // Note: settings_manager in turn notifies grid_settings_manager of this update
+                     _settings_manager.instance_settings = s.grid;
+                  }
+                  if (s.filters is FilterStates filters) {
+                     security_filter.load_state(filters.security);
+                     bid_price_filter.load_state(filters.bid_price);
+                     bid_merged_size_filter.load_state(filters.bid_merged_size);
+                     bid_direct_size_filter.load_state(filters.bid_direct_size);
+                     bid_implied_size_filter.load_state(filters.bid_implied_size);
+                     ask_price_filter.load_state(filters.ask_price);
+                     ask_merged_size_filter.load_state(filters.ask_merged_size);
+                     ask_direct_size_filter.load_state(filters.ask_direct_size);
+                     ask_implied_size_filter.load_state(filters.ask_implied_size);
+                     last_price_filter.load_state(filters.last_price);
+                     last_price_from_spread_filter.load_state(filters.last_price_from_spread);
+                     last_size_filter.load_state(filters.last_size);
+                     day_change_filter.load_state(filters.day_change);
+                     settlement_filter.load_state(filters.settlement);
+                     session_low_filter.load_state(filters.session_low);
+                     session_high_filter.load_state(filters.session_high);
+                     position_filter.load_state(filters.position);
+                     raw_day_position_filter.load_state(filters.raw_day_position);
+                     working_bid_volume_filter.load_state(filters.working_bid_volume);
+                     working_ask_volume_filter.load_state(filters.working_ask_volume);
+                     volume_filter.load_state(filters.volume);
+                     security_key_filter.load_state(filters.security_key);
+                     market_description_filter.load_state(filters.market_description);
+                     cleared_alias_filter.load_state(filters.cleared_alias);
+                     product_description_filter.load_state(filters.product_description);
+                     hub_alias_filter.load_state(filters.hub_alias);
+                     strip_name_filter.load_state(filters.strip_name);
+                     mic_code_filter.load_state(filters.mic_code);
+                  }
+                  rows_view.set_sort_descriptions(s.sorting_criterion, columns);
+               }
+            }), DataGridFilterHelper.hook_filters(rows_view, _filters),
+            settings_manager.subscribe_current().OnDispatcher().Subscribe(grid_settings_manager),
+            WorkLoop.Start("MarketView", _refresh)
+         );
+      }
+
+      public ObservableCollection<ColumnBase> columns { get; } = new ObservableCollection<ColumnBase>();
+
+      public SimpleGridSettingsManager<Row> grid_settings_manager { get; }
+
+      public GridSettings get_grid_save_state() {
+         if (_settings_manager.instance_settings == null)
+            return null;
+         else
+            return grid_settings_manager.create();
+      }
+
+      public static IComparer<Row> get_column_comparer(Columns column) {
+         switch (column) {
+            case Columns.Security: return wrap(r => r.security);
+            case Columns.SecurityKey: return wrap(r => r.security.Id);
+            case Columns.Volume: return wrap(r => r.volume);
+            case Columns.Last: return wrap(r => r.last_price);
+            case Columns.LastSize: return wrap(r => r.last_size);
+            case Columns.Position: return wrap(r => r.position);
+            case Columns.WorkingBids: return wrap(r => r.working_bid_volume);
+            case Columns.BidImpliedSize: return wrap(r => r.bid_implied_size);
+            case Columns.BidDirectSize: return wrap(r => r.bid_direct_size);
+            case Columns.BidMergedSize: return wrap(r => r.bid_merged_size);
+            case Columns.Bid: return wrap(r => r.bid_price);
+            case Columns.Ask: return wrap(r => r.ask_price);
+            case Columns.AskMergedSize: return wrap(r => r.ask_merged_size);
+            case Columns.AskDirectSize: return wrap(r => r.ask_direct_size);
+            case Columns.AskImpliedSize: return wrap(r => r.ask_implied_size);
+            case Columns.WorkingAsks: return wrap(r => r.working_ask_volume);
+            case Columns.Settlement: return wrap(r => r.settlement);
+            case Columns.DayChange: return wrap(r => r.day_change);
+            case Columns.SessionLow: return wrap(r => r.session_low);
+            case Columns.SessionHigh: return wrap(r => r.session_high);
+            case Columns.MarketDescription: return wrap(r => r.market_description);
+            case Columns.ClearedAlias: return wrap(r => r.security.cleared_alias());
+            case Columns.ProductDescription: return wrap(r => r.security.product_name());
+            case Columns.HubAlias: return wrap(r => r.security.hub_alias());
+            case Columns.StripName: return wrap(r => r.security.strip_name());
+            case Columns.MicCode: return wrap(r => r.security.mic_code());
+            default: return null;
+         }
+         IComparer<Row> wrap<T>(Func<Row, T> selector) =>
+            Comparer<T>.Default.wrap(selector);
+      }
+
+      #region Filters
+      public IDataGridColumnPersistentFilter<Row, string> security_filter { get; }
+      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> bid_price_filter { get; } = new DoubleMinMax<Row>(r => r.bid_price, nameof(Row.bid_price));
+      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> bid_merged_size_filter { get; } = new UIntMinMax<Row>(r => r.bid_merged_size, nameof(Row.bid_merged_size));
+      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> bid_direct_size_filter { get; } = new UIntMinMax<Row>(r => r.bid_direct_size, nameof(Row.bid_direct_size));
+      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> bid_implied_size_filter { get; } = new UIntMinMax<Row>(r => r.bid_implied_size, nameof(Row.bid_implied_size));
+      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> ask_price_filter { get; } = new DoubleMinMax<Row>(r => r.ask_price, nameof(Row.ask_price));
+      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> ask_merged_size_filter { get; } = new UIntMinMax<Row>(r => r.ask_merged_size, nameof(Row.ask_merged_size));
+      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> ask_direct_size_filter { get; } = new UIntMinMax<Row>(r => r.ask_direct_size, nameof(Row.ask_direct_size));
+      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> ask_implied_size_filter { get; } = new UIntMinMax<Row>(r => r.ask_implied_size, nameof(Row.ask_implied_size));
+      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> last_price_filter { get; } = new DoubleMinMax<Row>(r => r.last_price, nameof(Row.last_price));
+      public IDataGridColumnPersistentFilter<Row, string> last_price_from_spread_filter { get; } = new RegexFilterVM<Row>(r => r.last_price_from_spread.ToString(), nameof(Row.last_price_from_spread));
+      public IDataGridColumnPersistentFilter<Row, UIntMinMaxState> last_size_filter { get; } = new UIntMinMax<Row>(r => r.last_size, nameof(Row.last_size));
+      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> day_change_filter { get; } = new DoubleMinMax<Row>(r => r.day_change, nameof(Row.day_change));
+      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> settlement_filter { get; } = new DoubleMinMax<Row>(r => r.settlement, nameof(Row.settlement));
+      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> session_low_filter { get; } = new DoubleMinMax<Row>(r => r.session_low, nameof(Row.session_low));
+      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> session_high_filter { get; } = new DoubleMinMax<Row>(r => r.session_high, nameof(Row.session_high));
+      public IDataGridColumnPersistentFilter<Row, IntMinMaxState> position_filter { get; } = new IntMinMax<Row>(r => r.position, nameof(Row.position));
+      public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> raw_day_position_filter { get; } = new DoubleMinMax<Row>(r => r.raw_day_position, nameof(Row.raw_day_position));
+      public IDataGridColumnPersistentFilter<Row, IntMinMaxState> working_bid_volume_filter { get; } = new IntMinMax<Row>(r => r.working_bid_volume, nameof(Row.working_bid_volume));
+      public IDataGridColumnPersistentFilter<Row, IntMinMaxState> working_ask_volume_filter { get; } = new IntMinMax<Row>(r => r.working_ask_volume, nameof(Row.working_ask_volume));
+      public IDataGridColumnPersistentFilter<Row, ULongMinMaxState> volume_filter { get; } = new ULongMinMax<Row>(r => r.volume, nameof(Row.volume));
+      public IDataGridColumnPersistentFilter<Row, string> security_key_filter { get; } = new RegexFilterVM<Row>(r => r.security_key.ToString(), nameof(Row.security_key));
+      
+      public IDataGridColumnPersistentFilter<Row, string> market_description_filter { get; } = new RegexFilterVM<Row>(r => r.market_description, nameof(Row.market_description));
+      public IDataGridColumnPersistentFilter<Row, string> cleared_alias_filter { get; } = new RegexFilterVM<Row>(r => r.cleared_alias, nameof(Row.cleared_alias));
+      public IDataGridColumnPersistentFilter<Row, string> product_description_filter { get; } = new RegexFilterVM<Row>(r => r.product_description, nameof(Row.product_description));
+      public IDataGridColumnPersistentFilter<Row, string> hub_alias_filter { get; } = new RegexFilterVM<Row>(r => r.hub_alias, nameof(Row.hub_alias));
+      public IDataGridColumnPersistentFilter<Row, string> strip_name_filter { get; } = new RegexFilterVM<Row>(r => r.strip_name, nameof(Row.strip_name));
+      public IDataGridColumnPersistentFilter<Row, string> mic_code_filter { get; } = new RegexFilterVM<Row>(r => r.mic_code, nameof(Row.mic_code));
+
+      IEnumerable<IDataGridColumnFilter<Row>> _filters =>
+         _filter_getters.Select(fn => fn(this));
+
+      public FilterStates get_filter_states() {
+         return new FilterStates {
+            security = security_filter.get_state(),
+            bid_price = bid_price_filter.get_state(),
+            bid_merged_size = bid_merged_size_filter.get_state(),
+            bid_direct_size = bid_direct_size_filter.get_state(),
+            bid_implied_size = bid_implied_size_filter.get_state(),
+            ask_price = ask_price_filter.get_state(),
+            ask_merged_size = ask_merged_size_filter.get_state(),
+            ask_direct_size = ask_direct_size_filter.get_state(),
+            ask_implied_size = ask_implied_size_filter.get_state(),
+            last_price = last_price_filter.get_state(),
+            last_price_from_spread = last_price_from_spread_filter.get_state(),
+            last_size = last_size_filter.get_state(),
+            day_change = day_change_filter.get_state(),
+            settlement = settlement_filter.get_state(),
+            session_low = session_low_filter.get_state(),
+            session_high = session_high_filter.get_state(),
+            position = position_filter.get_state(),
+            raw_day_position = raw_day_position_filter.get_state(),
+            working_bid_volume = working_bid_volume_filter.get_state(),
+            working_ask_volume = working_ask_volume_filter.get_state(),
+            volume = volume_filter.get_state(),
+            security_key = security_key_filter.get_state(),
+            market_description = market_description_filter.get_state(),
+            cleared_alias = cleared_alias_filter.get_state(),
+            product_description = product_description_filter.get_state(),
+            hub_alias = hub_alias_filter.get_state(),
+            strip_name = strip_name_filter.get_state(),
+            mic_code = mic_code_filter.get_state(),
+         };
+      }
+      #endregion
+
+      protected override Row row_factory(SecurityMD smd) => new Row(_bus, smd, _orders_model);
+      protected override void on_row_removed(Row row) => row.Dispose();
+      protected override IEnumerable<SortingCriteria> get_default_sorting() { yield return new SortingCriteria(nameof(Row.security), true); }
+
+      void _refresh() {
+         foreach (var r in _row_vms) r.refresh();
+      }
+   }
+}

src/client/spark/Modules/Trading/ManualFills/ManualFillsVM.cs

@@ -2,6 +2,7 @@ using bts.api;
 using bts.api.core;
 using bts.api.position;
 using bts.collections;
+using bts.utils;
 using mbts;
 using mbts.spark;
 using mbts.ui;

src/client/spark/Modules/Trading/OrderBook/OrderBookRowVM.cs

@@ -36,7 +36,6 @@ namespace mbts.spark {
       public OrderBookRowVM(OrderBookVM owner, OrderSecurityRootPair order_tuple, bool is_expanded, Action<bool> on_selection_changed) {
          _owner = owner;
          this.order_tuple = order_tuple;
-         is_visible = !is_child;
          _is_expanded = is_expanded;
          prospective_quantity = this.quantity;
          prospective_price = this.price;
@@ -113,7 +112,6 @@ namespace mbts.spark {
       // Tree
       public bool is_parent { get => _is_parent; set => Set(ref _is_parent, value); }
       public bool is_child => order.has_parent();
-      public bool is_visible { get; set; }
       public bool is_expanded {
          get => _is_expanded;
          set {

src/client/spark/Modules/Trading/OrderBook/OrderBookVM.cs

@@ -188,6 +188,56 @@ namespace mbts.spark {
                security_plurality = p;
                RaisePropertyChanged(nameof(security_plurality));
             }),
+            Observable.Create<OrderBookRowVM>(o => {
+               OrderBookRowVM last_fired = null;
+               selected_order_changed += on_selected_order_changed;
+               var sub = _current_row.Subscribe(ch => notify());
+               return Disposable.Create(dispose);
+               // local methods
+               void on_selected_order_changed(OrderRecordImmutable? _) =>
+                  notify();
+
+               void notify() {
+                  var cr = current_row_if_distinct_root_selected();
+                  if (!cr.safe_equals(last_fired))
+                     o.OnNext(last_fired = cr);
+               }
+
+               void dispose() {
+                  Disposable.dispose(ref sub);
+                  selected_order_changed -= on_selected_order_changed;
+               }
+            })
+            .SubscribeEach(r => {
+               if (r == null) {
+                  current_algo_custom_state.Clear();
+                  return Disposable.empty;
+               }
+               r.updated += on_updated;
+               refresh_custom_state();
+               return Disposable.Create(() => r.updated -= on_updated);
+               void on_updated((OrderSecurityRootPair oldval, OrderSecurityRootPair newval) _) =>
+                  refresh_custom_state();
+
+               void refresh_custom_state() {
+                  current_algo_custom_state.Clear();
+                  var dcs = r.order.dynamic_custom_state();
+                  // while dcs is not expected to be null, best to play it safe and double check.
+                  if (dcs.is_null_or_empty()) {
+                     var child_rows = _row_collection.get_descendent_rows(r.order_key);
+                     if (_row_collection.filter != null) {
+                        child_rows = child_rows.Where(cr => _row_collection.filter(cr.order_tuple));
+                     }
+                     foreach (var dr in child_rows) {
+                        dcs = dr.order.dynamic_custom_state();
+                        if (!dcs.is_null_or_empty())
+                           break;
+                     }
+                  }
+                  foreach (var p in dcs.empty_if_null().OrderBy(kv => kv.Key))
+                     current_algo_custom_state.Add(new NamedValueVM(p.Key, DynamicOrderHelper.fmt_param(p.Value)));
+               }
+            }),
             orders_source.OnDispatcher().Subscribe(t => {
                if (_selected_rows.Count == 0) {
                   _cached_orders = t.state;
@@ -352,6 +402,15 @@ namespace mbts.spark {
          }
          void on_end_attaching() => Disposable.dispose(ref attach_vm_hooks);
 
+         // the output grid (custom state) in the right panel
+         // will only display info for the 'current_row'
+         // if all selected rows are of the same root order.
+         OrderBookRowVM current_row_if_distinct_root_selected() =>
+            current_row != null &&
+            selected_rows.Select(r => r.order_tuple.root.order.Key).Distinct().TryGetSingle(out uint droot) &&
+            current_row.order_tuple.root.order.Key == droot ?
+            current_row : null;
+
          IDisposable connect_order_mod_vm() =>
             Disposable.Create(
                _selected_params_mon.subscribe_algos().OnDispatcher().Subscribe(order_mod_vm.algo_vm.load_many),
@@ -395,6 +454,7 @@ namespace mbts.spark {
          }
       }
       public OrderRecordImmutable? current_order => current_row?.order;
+      public ObservableCollection<NamedValueVM> current_algo_custom_state { get; } = new ObservableCollection<NamedValueVM>();
       public AlgoDefinitionVM algo_vm => order_mod_vm.algo_vm;
       public OrderParamsVM order_mod_vm { get; }
       public FillsVMContainer fills_vm { get; } = new FillsVMContainer();

src/client/spark/Modules/Trading/OrderBook/View/OrderBookView.xaml

@@ -484,6 +484,7 @@
           <Border Padding="12,12,12,6">
             <Grid>
               <Grid.RowDefinitions>
+                <RowDefinition Height="Auto"/>
                 <RowDefinition Height="Auto"/>
                 <RowDefinition Height="*"/>
               </Grid.RowDefinitions>
@@ -495,7 +496,45 @@
                 AlgoParamRows="{Binding param_rows}"
                 IsReadOnly="{Binding DataContext.can_user_modify_selected_orders, ElementName=Root, Converter={StaticResource NotConverter}}"
               />
-              <StackPanel Grid.Row="1" HorizontalAlignment="Right" VerticalAlignment="Top" Margin="0,8,0,0" Orientation="Horizontal">
+              <c:ExpanderToggleButton
+                x:Name="algo_state_toggle"
+                Grid.Row="1"
+                BorderThickness="0"
+                Content="Algo State"
+                HorizontalContentAlignment="Left"
+                Margin="0,4,0,0"
+                VerticalAlignment="Top"
+                Visibility="{Binding current_algo_custom_state.Count, Converter={StaticResource IntNotZeroVisibilityConverter}}"
+              />
+              <c:DIYGrid Grid.Row="1" Rows="{Binding current_algo_custom_state}" Margin="0,28,0,4">
+                <c:DIYGrid.Visibility>
+                  <MultiBinding>
+                    <MultiBinding.Converter>
+                      <cv:MultiValueConverterPipe>
+                        <cv:BooleanAndConverter/>
+                        <cv:BooleanToVisibilityConverter/>
+                      </cv:MultiValueConverterPipe>
+                    </MultiBinding.Converter>
+                    <Binding Path="current_algo_custom_state.Count" Mode="OneWay" Converter="{StaticResource IntNotZeroConverter}"/>
+                    <Binding Path="IsChecked" ElementName="algo_state_toggle"/>
+                  </MultiBinding>
+                </c:DIYGrid.Visibility>
+                <c:DIYGrid.Columns>
+                  <c:DIYGridColumnCollection>
+                    <c:DIYGridColumn id="Name" Header="Param" Width="176"/>
+                    <c:DIYGridColumn id="Value" Header="Value" Width="176"/>
+                  </c:DIYGridColumnCollection>
+                </c:DIYGrid.Columns>
+                <c:DIYGrid.RowTemplate>
+                  <DataTemplate>
+                    <Grid c:DIYGrid.SyncWithColumnGeometry="True">
+                      <c:TextCell c:DIYGrid.ColumnKey="Name" Text="{Binding name}"/>
+                      <c:TextCell c:DIYGrid.ColumnKey="Value" Text="{Binding value}"/>
+                    </Grid>
+                  </DataTemplate>
+                </c:DIYGrid.RowTemplate>
+              </c:DIYGrid>
+              <StackPanel Grid.Row="2" HorizontalAlignment="Right" VerticalAlignment="Top" Margin="0,8,0,0" Orientation="Horizontal">
                 <Button x:Name="modify_orders_button" Command="{Binding modify_selected_orders_cmd}">
                   <Button.Margin>
                     <Thickness Right="{StaticResource DialogSkirtButtonPadding.Right}"/>

src/client/spark/Modules/Trading/PNL/VM/PNLRowVM.cs

@@ -31,7 +31,7 @@ namespace bts.shell.pnl {
          security_type_cell = _cell_map.add(ColumnKeys.SecurityType, d => d.security?.ContractCase ?? Security.ContractOneofCase.None);
          exchange_cell = _cell_map.add(ColumnKeys.Exchange, d => d.security?.Exchange ?? Venue.Unset);
          symbol_cell = _cell_map.add(ColumnKeys.Symbol, d => d.security?.Symbol ?? string.Empty);
-         security_cell = _cell_map.add(ColumnKeys.Security, d => d.security, null, EqualityComparer<SecurityImmutable?>.Default, Comparer<SecurityImmutable>.Default.to_nullable());
+         security_cell = _cell_map.add(ColumnKeys.Security, d => (d.security), null, EqualityComparer<SecurityImmutable?>.Default, Comparer<SecurityImmutable>.Default.to_nullable());
          security_key = _cell_map.add(ColumnKeys.SecurityKey, d => d.key.security_id);
          cpp = _cell_map.add(ColumnKeys.CPP, d => d.cpp);
          // sod

src/client/spark/Notifications/INotifier.cs

@@ -2,6 +2,7 @@
 using bts.api;
 using bts.shell;
 using bts.spark;
+using bts.utils;
 using mbts.config;
 using mbts.persist;
 using mbts.spark;
@@ -114,6 +115,7 @@ namespace mbts.ui {
             void dispose_core() {
                Disposable.dispose(ref _settings_sub);
                Disposable.dispose(ref _toast_controller);
+               Disposable.dispose(ref _sound_player);
                if (_dispatcher != null && !_dispatcher.HasShutdownStarted) {
                   _dispatcher.BeginInvokeShutdown(DispatcherPriority.Background);
                }
@@ -158,15 +160,25 @@ namespace mbts.ui {
             }
          }
 
-         class SoundNotificationPlayer {
-            readonly IReadOnlyDictionary<SoundNotification.SystemSound, NAudioPlayer> _players;
-
-            public static SoundNotificationPlayer create() => new SoundNotificationPlayer(_all_players());
+         class SoundNotificationPlayer : IDisposable {
+            IReadOnlyDictionary<SoundNotification.SystemSound, NAudioPlayer> _players;
 
             SoundNotificationPlayer(IReadOnlyDictionary<SoundNotification.SystemSound, NAudioPlayer> players) =>
                _players = players;
 
+            public static SoundNotificationPlayer create() =>
+               new SoundNotificationPlayer(_all_players());
+
+            public void Dispose() {
+               if (_players != null) {
+                  _players.Values.DisposeAll();
+                  _players = null;
+               }
+            }
+
             public void play_sounds(IEnumerable<SoundNotification> sounds) {
+               if (_players == null)
+                  return;
                foreach (var s in sounds) {
                   switch (s.sound_type_case) {
                      case SoundNotification.SoundTypeOneofCase.System: _players.GetValueOrDefault(s.system_sound.Value)?.play(); break;
@@ -177,7 +189,8 @@ namespace mbts.ui {
             }
 
             static IReadOnlyDictionary<SoundNotification.SystemSound, NAudioPlayer> _all_players() =>
-               EnumHelper.get_values<SoundNotification.SystemSound>().ToReadOnlyDictionary(s => s, NAudioPlayer.create);
+               EnumHelper.get_values<SoundNotification.SystemSound>()
+               .ToReadOnlyDictionaryWhere<SoundNotification.SystemSound, NAudioPlayer>(NAudioPlayer.try_create);
          }
 
          // This class adopts to role of view-controller in the mvc paradigm, rather than view-model in mvvm.

src/client/spark/Notifications/NAudioPlayer.cs

@@ -1,4 +1,5 @@
-using bts.shell;
+using bts;
+using bts.shell;
 using mbts.ui.utility;
 using NAudio.Wave;
 using System;
@@ -6,29 +7,71 @@ using System.IO;
 using System.Threading.Tasks;
 
 namespace mbts.ui {
-   class NAudioPlayer {
+   class NAudioPlayer : IDisposable {
       // number of milliseconds to delay playing sound,
       // during which time all subsequent requests to play the same sound will be ignored.
       const int _delay_ms = 75;
       readonly int _start_position;
+      IDisposable _stream_in;
       WaveOutEvent _wave_out = new WaveOutEvent();
       WaveFileReader _wave_reader;
       bool _is_playing, _is_play_requested;
 
-      public NAudioPlayer(Stream uwave_stream, int start_position = 0) {
-         var ms = new MemoryStream();
-         uwave_stream.CopyTo(ms);
-         ms.Position = 0;
-         _wave_reader = new WaveFileReader(ms);
+      /// <paramref name="stream_in"/> should not be disposed by the caller,
+      /// as NAudioPlayer handles that in its own Dispose method;
+      /// this constructor is private is to prevent such misuse.
+      NAudioPlayer(Stream stream_in, int start_position = 0) {
+         _stream_in = stream_in;
+         _wave_reader = new WaveFileReader(stream_in);
          _wave_reader.Position = _start_position = (start_position < _wave_reader.Length ? start_position : 0);
          _wave_out.Init(_wave_reader);
          _wave_out.PlaybackStopped += _on_playback_stopped;
       }
 
-      public static NAudioPlayer create(SoundNotification.SystemSound system_sound) {
-         var stream = _get_stream(system_sound);
-         var sb = EnumHelper.get_attribute<SoundNotification.SystemSound, SilenceBufferAttribute>(system_sound);
-         return sb != null ? new NAudioPlayer(stream, sb.start_position) : new NAudioPlayer(stream);
+      /// <see cref="_playback_stopped"/> is a proxy/buffer for the _wave_out.PlaybackStopped event.
+      /// It serves a couple purposes:
+      ///   1) The main callback (<see cref="_on_playback_stopped(object, StoppedEventArgs)"/>)
+      ///      is guarenteed to be executed first (though it would be anyways, since that
+      ///      callback is hooked up in the constructor, the proxy event makes it obvious).
+      ///   2) <see cref="_playback_stopped"/> is only raised after playback is truly complete,
+      ///      meaning there are no pending requests to repeat the sound.
+      event Action _playback_stopped;
+
+      public static bool try_create(SoundNotification.SystemSound system_sound, out NAudioPlayer result) {
+         if (_try_get_stream(system_sound, out var stream)) {
+            var sb = EnumHelper.get_attribute<SoundNotification.SystemSound, SilenceBufferAttribute>(system_sound);
+            result = sb != null ? new NAudioPlayer(stream, sb.start_position) : new NAudioPlayer(stream);
+            return true;
+         }
+         result = null;
+         return false;
+      }
+
+      public static void play(SoundNotification.SystemSound system_sound) {
+         if (try_create(system_sound, out var player)) {
+            try {
+               var temp = player;
+               temp._playback_stopped += temp.Dispose;
+               player.play();
+               player = null;
+            } catch (Exception ex) {
+               Log.Ui.warn($"Encountered exception playing the system sound '{system_sound}': {ex}.");
+            } finally {
+               player?.Dispose();
+            }
+         } else {
+            Log.Ui.warn($"Unable to play the system sound '{system_sound}': unexpected sound type (did not find an audio stream that corresponds to this enum value).");
+         }
+      }
+
+      public void Dispose() {
+         if (_wave_out != null) {
+            _wave_out.PlaybackStopped -= _on_playback_stopped;
+            _wave_out.Dispose();
+            _wave_out = null;
+         }
+         Disposable.dispose(ref _wave_reader);
+         Disposable.dispose(ref _stream_in);
       }
 
       public void play() {
@@ -46,6 +89,8 @@ namespace mbts.ui {
          if (_is_play_requested) {
             _is_play_requested = false;
             _play_core();
+         } else {
+            _playback_stopped?.Invoke();
          }
       }
 
@@ -56,18 +101,26 @@ namespace mbts.ui {
          _is_playing = false;
       }
 
-      static Stream _get_stream(SoundNotification.SystemSound system_sound) {
+      static bool _try_get_stream(SoundNotification.SystemSound system_sound, out Stream stream) {
+         UnmanagedMemoryStream um_stream;
          switch (system_sound) {
-            case SoundNotification.SystemSound.GameCorrect: return bts.Properties.Resources.game_sound_correct;
-            case SoundNotification.SystemSound.GameIncorrect: return bts.Properties.Resources.game_sound_wrong;
-            case SoundNotification.SystemSound.GameOverArcade: return bts.Properties.Resources.game_over_arcade;
-            case SoundNotification.SystemSound.Whistle: return bts.Properties.Resources.whistle;
-            case SoundNotification.SystemSound.BellCounter: return bts.Properties.Resources.bell_counter;
-            case SoundNotification.SystemSound.Powerup: return bts.Properties.Resources.powerup;
-            case SoundNotification.SystemSound.CashRegister: return bts.Properties.Resources.cash_register_purchase;
-            case SoundNotification.SystemSound.Reload: return bts.Properties.Resources.reload_1911;
-            default: throw new NotImplementedException($"Unknown {nameof(SoundNotification.SystemSound)}: {system_sound}.");
+            case SoundNotification.SystemSound.GameCorrect: um_stream = bts.Properties.Resources.game_sound_correct; break;
+            case SoundNotification.SystemSound.GameIncorrect: um_stream = bts.Properties.Resources.game_sound_wrong; break;
+            case SoundNotification.SystemSound.GameOverArcade: um_stream = bts.Properties.Resources.game_over_arcade; break;
+            case SoundNotification.SystemSound.Whistle: um_stream = bts.Properties.Resources.whistle; break;
+            case SoundNotification.SystemSound.BellCounter: um_stream = bts.Properties.Resources.bell_counter; break;
+            case SoundNotification.SystemSound.Powerup: um_stream = bts.Properties.Resources.powerup; break;
+            case SoundNotification.SystemSound.CashRegister: um_stream = bts.Properties.Resources.cash_register_purchase; break;
+            case SoundNotification.SystemSound.Reload: um_stream = bts.Properties.Resources.reload_1911; break;
+            default:
+               stream = null;
+               return false;
          }
+         stream = new MemoryStream();
+         um_stream.CopyTo(stream);
+         um_stream.Dispose();
+         stream.Position = 0;
+         return true;
       }
    }
-}
+}

src/client/spark/Orders/OrderTypeVMs/DynamicOrderHelper.cs

@@ -23,6 +23,32 @@ namespace mbts.spark {
 
       public IEnumerable<string> dynamic_types => _dynamic_types.DynamicOrderDesc.Select(d => d.DynamicOrderType);
 
+      public static string fmt_param(DynamicOrderParam dyn_param) {
+         if (dyn_param == null)
+            return null;
+         switch (dyn_param.ValueCase) {
+            case DynamicOrderParam.ValueOneofCase.AsArray:
+               return string.Join(", ", dyn_param.AsArray.Element.Select(fmt_param));
+            case DynamicOrderParam.ValueOneofCase.AsBool:
+               return dyn_param.AsBool.ToString();
+            case DynamicOrderParam.ValueOneofCase.AsDouble:
+               return dyn_param.AsDouble.ToString();
+            case DynamicOrderParam.ValueOneofCase.AsEnumerated:
+               return dyn_param.AsEnumerated.EnumValue;
+            case DynamicOrderParam.ValueOneofCase.AsInt32:
+               return dyn_param.AsInt32.ToString();
+            case DynamicOrderParam.ValueOneofCase.AsQuantity:
+               return dyn_param.AsQuantity.ToString();
+            case DynamicOrderParam.ValueOneofCase.AsString:
+               return dyn_param.AsString;
+            default:
+               // implement additional cases as needed
+               // (as of 4/17/20, this method is only used to format
+               // dynamic orders' CustomState params in the orderbook)
+               return $"({dyn_param.ValueCase})";
+         }
+      }
+
       public void add_dynamic_order_defs(List<CustomOrderDefinitionImmutable> defs, Func<CustomOrderDefinition.CustomOneofCase, CustomOrderDefinition> factory) {
          foreach (var d in _dynamic_types.DynamicOrderDesc) {
             var dorder = factory(CustomOrderDefinition.CustomOneofCase.Dynamic);

src/client/spark/Orders/OrderUtils.cs

@@ -63,6 +63,11 @@ namespace mbts.spark {
       public static string last_msg(this OrderRecordImmutable order) =>
          order.HasState ? order.State.LastMsg : string.Empty;
 
+      public static IReadOnlyDictionary<string, DynamicOrderParam> dynamic_custom_state(this OrderRecordImmutable order) =>
+         order.HasState && order.State.HasDynamic ?
+            (order.State.Dynamic.CustomState ?? DictionaryHelper.empty<string, DynamicOrderParam>()) :
+            DictionaryHelper.empty<string, DynamicOrderParam>();
+
       public static DateTime created(this OrderRecordImmutable order) =>
          order.HasState && order.State.HasCreated ? order.State.Created.ToDateTime().ToLocalTime() : DateTime.MinValue;
 

src/client/spark/view/Controls/DIYGrid/Columns/DIYGridColumn.cs

@@ -0,0 +1,9 @@
+namespace mbts.ui.controls {
+   class DIYGridColumn : DIYGrid.ColumnBase<string> {
+      public DIYGridColumn()
+         : base() {
+         is_visible = true;
+         Width = 60;
+      }
+   }
+}

src/client/spark/view/Controls/DIYGrid/Settings/Animation/SoundAnimationView.xaml.cs

@@ -38,11 +38,7 @@ namespace mbts.ui.controls {
       }
 
       public static ICommand PreviewSoundCommand =>
-         new DelegateCommand<SoundNotification.SystemSound>(s =>  {
-            try {
-               NAudioPlayer.create(s).play();
-            } catch (NAudio.MmException) { }
-         });
+         new DelegateCommand<SoundNotification.SystemSound>(NAudioPlayer.play);
 
       #region SoundTypes
       public static readonly DependencyProperty SoundTypesProperty =

src/client/spark/view/Controls/DataGridEx/Filters/VM/RegexFilterVM.cs

@@ -1,39 +1,43 @@
-using System;
-using System.Text.RegularExpressions;
+using System;
+using System.Text.RegularExpressions;
+
+namespace mbts.spark.filters {
+   static class RegexFilterUtils {
+      public static Regex build_regex(string str) {
+         if (str == null)
+            return null;
+         str = str.Trim();
+         if (str == string.Empty)
+            return null;
+         return new Regex(str, RegexOptions.IgnoreCase | RegexOptions.IgnorePatternWhitespace);
+      }
+   }
+   class RegexFilterVM<TRow> : FilterVMBase<TRow, string, string> {
+      string _text = string.Empty;
+      Regex _regex;
+
+      public RegexFilterVM(Func<TRow, string> get_row_val, string prop_name = null) : base(get_row_val, prop_name) { }
+
+      public string text {
+         get => _text;
+         set => _regex = RegexFilterUtils.build_regex(_text = (value ?? string.Empty));
+      }
+
+      public override string get_state() => _text ?? string.Empty;
+
+      protected override bool _get_active() => _regex != null;
+      protected override bool _filter(string value) =>
+         _regex != null && (
+            _regex.IsMatch(value) ||
+            (!string.IsNullOrEmpty(value) && _regex.IsMatch(value.Replace(" ", ""))));
 
-namespace mbts.spark.filters {
-   static class RegexFilterUtils {
-      public static Regex build_regex(string str) {
-         if (str == null)
-            return null;
-         str = str.Trim();
-         if (str == string.Empty)
-            return null;
-         return new Regex(str, RegexOptions.IgnoreCase | RegexOptions.IgnorePatternWhitespace);
-      }
-   }
-   class RegexFilterVM<TRow> : FilterVMBase<TRow, string, string> {
-      string _text = string.Empty;
-      Regex _regex;
-
-      public RegexFilterVM(Func<TRow, string> get_row_val, string prop_name = null) : base(get_row_val, prop_name) { }
-
-      public string text {
-         get => _text;
-         set => _regex = RegexFilterUtils.build_regex(_text = (value ?? string.Empty));
-      }
-
-      public override string get_state() => _text ?? string.Empty;
-
-      protected override bool _get_active() => _regex != null;
-      protected override bool _filter(string value) => _regex != null && _regex.IsMatch(value);
-      protected override void _load_state_internal(string state) => text = state;
-      protected override void _clear() {
-         _close();
-         _text = string.Empty;
-         _regex = null;
-         RaisePropertyChanged(nameof(text));
-         _raise_filter_changed();
-      }
-   }
+      protected override void _load_state_internal(string state) => text = state;
+      protected override void _clear() {
+         _close();
+         _text = string.Empty;
+         _regex = null;
+         RaisePropertyChanged(nameof(text));
+         _raise_filter_changed();
+      }
+   }
 }

src/client/spark/view/Utility/EnabledValueVM.cs

@@ -30,6 +30,21 @@ namespace mbts.ui {
       public string name { get; }
    }
 
+   class NamedValueVM : ViewModelBase {
+      string _name;
+      object _value;
+
+      public NamedValueVM() { }
+
+      public NamedValueVM(string name, object value) {
+         _name = name;
+         _value = value;
+      }
+
+      public string name { get => _name; set => Set(ref _name, value); }
+      public object value { get => _value; set => Set(ref _value, value); }
+   }
+
    static class EnabledValueVMExtensions {
       public static T? get_nullable<T>(this EnabledValueVM<T> vm) where T : struct =>
          vm.enabled ? vm.value : (T?)null;

src/client/spark_api/security/SecurityHelper.cs

@@ -83,22 +83,24 @@ namespace bts.api {
          }
 
          public static int Compare(SecurityImmutable x, SecurityImmutable y) {
-            if (x.ContractCase == ContractOneofCase.Future) return y.ContractCase == ContractOneofCase.Future ? Compare(x.Symbol, x.Future, y.Symbol, y.Future) : -1;
-            else if (y.ContractCase == ContractOneofCase.Future) return 1;
+            if (x.ContractCase == ContractOneofCase.Future) {
+               return y.ContractCase == ContractOneofCase.Future ? Compare(x.Symbol, x.Future, y.Symbol, y.Future) : -1;
+            } else if (y.ContractCase == ContractOneofCase.Future) {
+               return 1;
+            }
             var xlegs = get_flat_future_legs(x).ToList();
             var ylegs = get_flat_future_legs(y).ToList();
             int max = Math.Min(xlegs.Count, ylegs.Count);
             for (int i = 0; i < max; i++) {
                int rv = Compare(xlegs[i].symbol, xlegs[i].future, ylegs[i].symbol, ylegs[i].future);
-               if (rv != 0) return rv;
-            }
-            for (int i = 0; i < max; i++) {
-               int rv = xlegs[i].quantity.CompareTo(ylegs[i].quantity);
-               if (rv != 0) return rv;
-            }
-            for (int i = 0; i < max; i++) {
-               int rv = xlegs[i].exchange.CompareTo(ylegs[i].exchange);
-               if (rv != 0) return rv;
+               if (rv != 0)
+                  return rv;
+               rv = xlegs[i].quantity.CompareTo(ylegs[i].quantity);
+               if (rv != 0)
+                  return rv;
+               rv = xlegs[i].exchange.CompareTo(ylegs[i].exchange);
+               if (rv != 0)
+                  return rv;
             }
             return xlegs.Count.CompareTo(ylegs.Count);
          }

src/client/spark_core/Collections/Extensions/EnumerableExtensions.cs

@@ -390,6 +390,15 @@ namespace System.Collections.Generic {
       public static ReadOnlyDictionary<TKey, TElement> ToReadOnlyDictionary<T, TKey, TElement>(this IEnumerable<T> enumerable, Func<T, TKey> key_selector, Func<T, TElement> element_selector, IEqualityComparer<TKey> comparer) =>
          new ReadOnlyDictionary<TKey, TElement>(enumerable.ToDictionary(key_selector, element_selector, comparer));
 
+      public static ReadOnlyDictionary<TKey, TElement> ToReadOnlyDictionaryWhere<TKey, TElement>(this IEnumerable<TKey> keys, TryGetValue<TKey, TElement> value_selector) {
+         var mdict = new Dictionary<TKey, TElement>();
+         foreach (var k in keys) {
+            if (value_selector(k, out var val))
+               mdict[k] = val;
+         }
+         return new ReadOnlyDictionary<TKey, TElement>(mdict);
+      }
+
       public static ILookup<TKey, TElement> ToFlatLookup<T, TKey, TElement>(this IEnumerable<T> source, Func<T, TKey> key_selector, Func<T, IEnumerable<TElement>> selector) =>
          source.SelectMany(v => {
             var k = key_selector(v);

src/client/spark_core/utils/Delegates.cs

@@ -1,11 +1,11 @@
 using System;
 using System.Globalization;
 
-namespace mbts.utils {
-   delegate bool TryGetValue<T>(out T value);
-   delegate bool TryGetValue<TKey, TValue>(TKey key, out TValue value);
+namespace bts.utils {
+   public delegate bool TryGetValue<T>(out T value);
+   public delegate bool TryGetValue<TKey, TValue>(TKey key, out TValue value);
 
-   static class LogicalDelegates {
+   public static class LogicalDelegates {
       public static readonly Func<bool, bool, bool>
          And = (a, b) => a && b,
          Or = (a, b) => a || b,
@@ -13,7 +13,7 @@ namespace mbts.utils {
          None = (a, b) => !a && !b;
    }
 
-   static class FormattingDelegates {
+   public static class FormattingDelegates {
       public static class Price {
          public static readonly Func<double, string> Value =
             p => double.IsNaN(p) ? "" : p.ToString("#,0.00");
@@ -26,4 +26,4 @@ namespace mbts.utils {
             p => p.ToString("#,0.00", CultureInfo.CurrentCulture);
       }
    }
-}
+}

src/client/spark_core/utils/FilePathHelper.cs

@@ -0,0 +1,21 @@
+using System;
+using System.IO;
+
+namespace bts.utils {
+   public static class FilePathHelper {
+      public static string GetUniquePathWithDate(string path_root, string ext) =>
+         GetUniquePath($"{path_root}_{DateTime.Now:MM - dd - yyyy_hh - mm}", ext);
+
+      public static string GetUniquePath(string path_root, string ext) {
+         var path = $"{path_root}.{ext}";
+         const int max_copies = 2 ^ 14;
+         int i = 1;
+         while (File.Exists(path)) {
+            path = $"{path_root}({i++}).{ext}";
+            if (i == max_copies)
+               throw new InvalidOperationException("Unable to make path unique by concating a number.");
+         }
+         return path;
+      }
+   }
+}

src/client/spark_core/utils/ObservableExtensions.cs

@@ -209,7 +209,7 @@ namespace System.Reactive.Linq {
       /// <returns>An IDisposable object used to unsubscribe from the source sequence, and dispose the latest value subscription.</returns>
       public static IDisposable SubscribeEach<T>(this IObservable<T> source, Func<T, IDisposable> subscribe_value) {
          IDisposable value_sub = null;
-         IDisposable source_sub = source.Select(v => v).Subscribe(on_source);
+         IDisposable source_sub = source.Subscribe(on_source);
          return Disposable.Create(dispose);
          // local methods
          void on_source(T value) {

src/client/spark_core/utils/UpdateExtensions.cs

@@ -1,208 +1,253 @@
-using bts.api.core;
-using bts.collections;
-using System;
-using System.Collections.Generic;
-using System.Linq;
-using System.Reactive.Linq;
-
-namespace bts.utils {
-   public static class UpdateExtensions {
-      public static TableUpdate<T, TKey> with_added<T, TKey>(this TableUpdate<T, TKey> table, IEnumerable<T> added) where TKey : IEquatable<TKey> =>
-         TableUpdate<T, TKey>.update(table, added, Enumerable.Empty<Change<T>>(), Enumerable.Empty<T>(), table.key_extractor);
-
-      public static TableUpdate<T, TKey> with_removed<T, TKey>(this TableUpdate<T, TKey> table, IEnumerable<T> removed) where TKey : IEquatable<TKey> =>
-         TableUpdate<T, TKey>.update(table, Enumerable.Empty<T>(), Enumerable.Empty<Change<T>>(), removed, table.key_extractor);
-
-      public static TableUpdate<T, TKey> with_changed<T, TKey>(this TableUpdate<T, TKey> table, IEnumerable<T> changed) where TKey : IEquatable<TKey> =>
-         TableUpdate<T, TKey>.update(table, Enumerable.Empty<T>(), changed.Select(v => new Change<T>(table.state[table.key_extractor(v)], v)), Enumerable.Empty<T>(), table.key_extractor);
-
-      public static TableUpdate<T, TKey> with_changes<T, TKey>(this TableUpdate<T, TKey> table, IEnumerable<Change<T>> changes) where TKey : IEquatable<TKey> =>
-         TableUpdate<T, TKey>.update(table, Enumerable.Empty<T>(), changes, Enumerable.Empty<T>(), table.key_extractor);
-
-      public static IObservable<TableUpdate<TResult, TKey>> RootDependent<T, TResult, TKey>(
-         this IObservable<TableUpdate<T, TKey>> source,
-         Func<T, TKey> key,
-         Func<T, TKey> parent_key,
-         Func<T, T, TResult> with_root,
-         Func<T, TResult> without_root,
-         Func<TResult, TKey> result_key)
-         where TKey : IEquatable<TKey> =>
-         RootDependent(source, key, parent_key, with_root, without_root, result_key, default);
-
-      // TODO: unit tests
-      public static IObservable<TableUpdate<TResult, TKey>> RootDependent<T, TKey, TResult>(
-         this IObservable<TableUpdate<T, TKey>> source,
-         Func<T, TKey> key,
-         Func<T, TKey> parent_key,
-         Func<T, T, TResult> get_result,
-         Func<T, TResult> get_result_no_parent,
-         Func<TResult, TKey> result_key,
-         TKey no_parent_key)
-         where TKey : IEquatable<TKey> =>
-         Observable.Create<TableUpdate<TResult, TKey>>(observer => {
-            var state = DictionaryHelper.empty<TKey, T>();
-            var output = TableUpdate<TResult, TKey>.mk_empty(result_key);
-            var parent_to_children = new SetMap<TKey, T>();
-            return source.Subscribe(update);
-            //
-            void update(TableUpdate<T, TKey> input) {
-               state = input.state;
-               var builder = output.to_builder();
-               // execute all parent_to_children changes first, so descendent-propagation will be efficient (otherwise propagation might be needlessly performed on values that are yet to be removed, or performed with the wrong root, if the real root is yet to be added).
-               foreach (var v in input.removed.Concat(input.changes.Select(ch => ch.Old))) {
-                  var k = input.key_extractor(v);
-                  parent_to_children.remove(parent_key(v), v);
-               }
-               foreach (var v in input.added.Concat(input.changed)) {
-                  var pk = parent_key(v);
-                  if (!pk.Equals(no_parent_key))
-                     parent_to_children.add(pk, v);
-               }
-               foreach (var v in input.removed) {
-                  var k = input.key_extractor(v);
-                  builder.remove(k);
-                  if (parent_to_children.try_lookup(k, out ISet<T> children)) {
-                     foreach (var child in children) {
-                        var ck = key(child);
-                        builder.add_or_update(ck, get_result_no_parent(child));
-                        foreach (var d in descendents(ck))
-                           builder.add_or_update(key(d), get_result(d, child));
-                     }
-                  }
-               }
-               foreach (var v in input.changed) {
-                  var k = input.key_extractor(v);
-                  var root = get_root(v);
-                  TResult result;
-                  if (key(root).Equals(k)) {
-                     result = get_result_no_parent(v);
-                     foreach (var d in descendents(k))
-                        builder.add_or_update(key(d), get_result(d, v));
-                  } else {
-                     result = get_result(v, root);
-                  }
-                  builder.add_or_update(k, result);
-               }
-               foreach (var v in input.added) {
-                  var k = input.key_extractor(v);
-                  var pk = parent_key(v);
-                  TResult result;
-                  if (pk.Equals(no_parent_key)) {
-                     result = get_result_no_parent(v);
-                     foreach (var d in descendents(k))
-                        builder.add_or_update(key(d), get_result(d, v));
-                  } else {
-                     parent_to_children.add(pk, v);
-                     var root = get_root(v);
-                     if (key(root).Equals(k)) {
-                        result = get_result_no_parent(v);
-                        foreach (var d in descendents(k))
-                           builder.add_or_update(key(d), get_result(d, v));
-                     } else {
-                        result = get_result(v, root);
-                     }
-                  }
-                  builder.add_or_update(k, result);
-               }
-               observer.OnNext(output = builder.to_table());
-            }
-            T get_root(T value) {
-               var pk = parent_key(value);
-               while (!pk.Equals(no_parent_key) && state.TryGetValue(pk, out var parent)) {
-                  value = parent;
-                  pk = parent_key(value);
-               }
-               return value;
-            }
-            IEnumerable<T> descendents(TKey k) {
-               if (parent_to_children.try_lookup(k, out ISet<T> children)) {
-                  foreach (var child in children) {
-                     yield return child;
-                     foreach (var d in descendents(key(child)))
-                        yield return d;
-                  }
-               }
-            }
-         });
-
-      public static IObservable<TableUpdate<T, TKey>> MergeTable<T, TKey>(this IObservable<TableUpdate<T, TKey>> source, IObservable<TableUpdate<T, TKey>> other) where TKey : IEquatable<TKey> =>
-         source.AggregateLatest(other, _Merge<T, TKey>.source_change_del, _Merge<T, TKey>.other_change_del, () => TableUpdate<T, TKey>.empty, TableUpdate<T, TKey>.empty, TableUpdate<T, TKey>.empty);
-
-      public static IObservable<TableUpdate<KeyValuePair<TKey, TResult>, TKey>> ParentDependent<T, TKey, TResult>(
-         this IObservable<TableUpdate<T, TKey>> source,
-         Func<T, TKey> key,
-         Func<T, TKey> parent_key,
-         Func<T, T, TResult> get_result,
-         Func<T, TResult> get_result_no_parent)
-         where TResult : IEquatable<TResult>
-         where TKey : IEquatable<TKey> =>
-         ParentDependent(source, key, parent_key, get_result, get_result_no_parent, EqualityComparer<TResult>.Default, default);
-
-      public static IObservable<TableUpdate<KeyValuePair<TKey, TResult>, TKey>> ParentDependent<T, TKey, TResult>(
-         this IObservable<TableUpdate<T, TKey>> source,
-         Func<T, TKey> key,
-         Func<T, TKey> parent_key,
-         Func<T, T, TResult> get_result,
-         Func<T, TResult> get_result_no_parent,
-         IEqualityComparer<TResult> result_comparer,
-         TKey no_parent_key)
-         where TKey : IEquatable<TKey> =>
-         Observable.Create<TableUpdate<KeyValuePair<TKey, TResult>, TKey>>(observer => {
-            var value_comparer = EqualityComparer.Create<KeyValuePair<TKey, TResult>, TResult>(kv => kv.Value, result_comparer);
-            var state = DictionaryHelper.empty<TKey, T>();
-            var output = TableUpdate<KeyValuePair<TKey, TResult>, TKey>.mk_empty(kv => kv.Key);
-            var parent_to_children = new SetMap<TKey, T>();
-            return source.Subscribe(update);
-            //
-            void update(TableUpdate<T, TKey> input) {
-               state = input.state;
-               var builder = output.to_builder();
-               foreach (var v in input.removed) {
-                  var k = input.key_extractor(v);
-                  builder.remove(k);
-                  parent_to_children.remove(parent_key(v), v);
-                  if (parent_to_children.try_lookup(k, out ISet<T> children)) {
-                     foreach (var child in children)
-                        builder.add_or_update(new KeyValuePair<TKey, TResult>(key(child), get_result_no_parent(child)));
-                  }
-               }
-               foreach (var v in input.changed) {
-                  var k = key(v);
-                  builder.add_or_update(new KeyValuePair<TKey, TResult>(k, lookup_result(v)));
-                  parent_to_children.set(parent_key(v), v);
-                  if (parent_to_children.try_lookup(k, out ISet<T> children)) {
-                     foreach (var child in children)
-                        builder.add_or_update(new KeyValuePair<TKey, TResult>(key(child), get_result(child, v)));
-                  }
-               }
-               foreach (var v in input.added) {
-                  var k = key(v);
-                  builder.add_or_update(new KeyValuePair<TKey, TResult>(k, lookup_result(v)));
-                  parent_to_children.add(parent_key(v), v);
-                  if (parent_to_children.try_lookup(k, out ISet<T> children)) {
-                     foreach (var child in children)
-                        builder.add_or_update(new KeyValuePair<TKey, TResult>(key(child), get_result(child, v)));
-                  }
-               }
-               var result = builder.to_table_checked(value_comparer);
-               if (result.has_changes())
-                  observer.OnNext(output = result);
-            }
-            TResult lookup_result(T child) {
-               var pk = parent_key(child);
-               return !pk.Equals(no_parent_key) && state.TryGetValue(pk, out T parent) ?  get_result(child, parent) : get_result_no_parent(child);
-            }
-         });
-
-      static class _Merge<T, TKey> where TKey : IEquatable<TKey> {
-         public static readonly Func<TableUpdate<T, TKey>, Change<TableUpdate<T, TKey>>, TableUpdate<T, TKey>, TableUpdate<T, TKey>> source_change_del = _source_changed_impl;
-         public static readonly Func<TableUpdate<T, TKey>, TableUpdate<T, TKey>, Change<TableUpdate<T, TKey>>, TableUpdate<T, TKey>> other_change_del = _other_changed_impl;
-
-         static TableUpdate<T, TKey> _source_changed_impl(TableUpdate<T, TKey> target, Change<TableUpdate<T, TKey>> source_change, TableUpdate<T, TKey> other_table) =>
-            TableUpdate<T, TKey>.update(target, source_change.New.added, source_change.New.changes, source_change.New.removed, source_change.New.key_extractor);
-
-         static TableUpdate<T, TKey> _other_changed_impl(TableUpdate<T, TKey> target, TableUpdate<T, TKey> source_table, Change<TableUpdate<T, TKey>> other_change) =>
-            TableUpdate<T, TKey>.update(target, other_change.New.added, other_change.New.changes, other_change.New.removed, other_change.New.key_extractor);
-      }
-   }
-}
+using bts.api.core;
+using bts.collections;
+using bts.utils.proto;
+using Google.Protobuf;
+using mbts;
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Reactive.Linq;
+
+namespace bts.utils {
+   public static class UpdateExtensions {
+      public static TableUpdate<T, TKey> with_added<T, TKey>(this TableUpdate<T, TKey> table, IEnumerable<T> added) where TKey : IEquatable<TKey> =>
+         TableUpdate<T, TKey>.update(table, added, Enumerable.Empty<Change<T>>(), Enumerable.Empty<T>(), table.key_extractor);
+
+      public static TableUpdate<T, TKey> with_removed<T, TKey>(this TableUpdate<T, TKey> table, IEnumerable<T> removed) where TKey : IEquatable<TKey> =>
+         TableUpdate<T, TKey>.update(table, Enumerable.Empty<T>(), Enumerable.Empty<Change<T>>(), removed, table.key_extractor);
+
+      public static TableUpdate<T, TKey> with_changed<T, TKey>(this TableUpdate<T, TKey> table, IEnumerable<T> changed) where TKey : IEquatable<TKey> =>
+         TableUpdate<T, TKey>.update(table, Enumerable.Empty<T>(), changed.Select(v => new Change<T>(table.state[table.key_extractor(v)], v)), Enumerable.Empty<T>(), table.key_extractor);
+
+      public static TableUpdate<T, TKey> with_changes<T, TKey>(this TableUpdate<T, TKey> table, IEnumerable<Change<T>> changes) where TKey : IEquatable<TKey> =>
+         TableUpdate<T, TKey>.update(table, Enumerable.Empty<T>(), changes, Enumerable.Empty<T>(), table.key_extractor);
+
+      public static IObservable<TableUpdate<TResult, TKey>> RootDependent<T, TResult, TKey>(
+         this IObservable<TableUpdate<T, TKey>> source,
+         Func<T, TKey> key,
+         Func<T, TKey> parent_key,
+         Func<T, T, TResult> with_root,
+         Func<T, TResult> without_root,
+         Func<TResult, TKey> result_key)
+         where TKey : IEquatable<TKey> =>
+         RootDependent(source, key, parent_key, with_root, without_root, result_key, default);
+
+      // TODO: unit tests
+      public static IObservable<TableUpdate<TResult, TKey>> RootDependent<T, TKey, TResult>(
+         this IObservable<TableUpdate<T, TKey>> source,
+         Func<T, TKey> key,
+         Func<T, TKey> parent_key,
+         Func<T, T, TResult> get_result,
+         Func<T, TResult> get_result_no_parent,
+         Func<TResult, TKey> result_key,
+         TKey no_parent_key)
+         where TKey : IEquatable<TKey> =>
+         Observable.Create<TableUpdate<TResult, TKey>>(observer => {
+            var state = DictionaryHelper.empty<TKey, T>();
+            var output = TableUpdate<TResult, TKey>.mk_empty(result_key);
+            var parent_to_children = new SetMap<TKey, T>();
+            return source.Subscribe(update);
+            //
+            void update(TableUpdate<T, TKey> input) {
+               state = input.state;
+               var builder = output.to_builder();
+               // execute all parent_to_children changes first, so descendent-propagation will be efficient (otherwise propagation might be needlessly performed on values that are yet to be removed, or performed with the wrong root, if the real root is yet to be added).
+               foreach (var v in input.removed.Concat(input.changes.Select(ch => ch.Old))) {
+                  var k = input.key_extractor(v);
+                  parent_to_children.remove(parent_key(v), v);
+               }
+               foreach (var v in input.added.Concat(input.changed)) {
+                  var pk = parent_key(v);
+                  if (!pk.Equals(no_parent_key))
+                     parent_to_children.add(pk, v);
+               }
+               foreach (var v in input.removed) {
+                  var k = input.key_extractor(v);
+                  builder.remove(k);
+                  if (parent_to_children.try_lookup(k, out ISet<T> children)) {
+                     foreach (var child in children) {
+                        var ck = key(child);
+                        builder.add_or_update(ck, get_result_no_parent(child));
+                        foreach (var d in descendents(ck))
+                           builder.add_or_update(key(d), get_result(d, child));
+                     }
+                  }
+               }
+               foreach (var v in input.changed) {
+                  var k = input.key_extractor(v);
+                  var root = get_root(v);
+                  TResult result;
+                  if (key(root).Equals(k)) {
+                     result = get_result_no_parent(v);
+                     foreach (var d in descendents(k))
+                        builder.add_or_update(key(d), get_result(d, v));
+                  } else {
+                     result = get_result(v, root);
+                  }
+                  builder.add_or_update(k, result);
+               }
+               foreach (var v in input.added) {
+                  var k = input.key_extractor(v);
+                  var pk = parent_key(v);
+                  TResult result;
+                  if (pk.Equals(no_parent_key)) {
+                     result = get_result_no_parent(v);
+                     foreach (var d in descendents(k))
+                        builder.add_or_update(key(d), get_result(d, v));
+                  } else {
+                     parent_to_children.add(pk, v);
+                     var root = get_root(v);
+                     if (key(root).Equals(k)) {
+                        result = get_result_no_parent(v);
+                        foreach (var d in descendents(k))
+                           builder.add_or_update(key(d), get_result(d, v));
+                     } else {
+                        result = get_result(v, root);
+                     }
+                  }
+                  builder.add_or_update(k, result);
+               }
+               observer.OnNext(output = builder.to_table());
+            }
+            T get_root(T value) {
+               var pk = parent_key(value);
+               while (!pk.Equals(no_parent_key) && state.TryGetValue(pk, out var parent)) {
+                  value = parent;
+                  pk = parent_key(value);
+               }
+               return value;
+            }
+            IEnumerable<T> descendents(TKey k) {
+               if (parent_to_children.try_lookup(k, out ISet<T> children)) {
+                  foreach (var child in children) {
+                     yield return child;
+                     foreach (var d in descendents(key(child)))
+                        yield return d;
+                  }
+               }
+            }
+         });
+
+      public static IObservable<TableUpdate<T, TKey>> MergeTable<T, TKey>(this IObservable<TableUpdate<T, TKey>> source, IObservable<TableUpdate<T, TKey>> other) where TKey : IEquatable<TKey> =>
+         source.AggregateLatest(other, _Merge<T, TKey>.source_change_del, _Merge<T, TKey>.other_change_del, () => TableUpdate<T, TKey>.empty, TableUpdate<T, TKey>.empty, TableUpdate<T, TKey>.empty);
+
+      public static IObservable<TableUpdate<KeyValuePair<TKey, TResult>, TKey>> ParentDependent<T, TKey, TResult>(
+         this IObservable<TableUpdate<T, TKey>> source,
+         Func<T, TKey> key,
+         Func<T, TKey> parent_key,
+         Func<T, T, TResult> get_result,
+         Func<T, TResult> get_result_no_parent)
+         where TResult : IEquatable<TResult>
+         where TKey : IEquatable<TKey> =>
+         ParentDependent(source, key, parent_key, get_result, get_result_no_parent, EqualityComparer<TResult>.Default, default);
+
+      public static IObservable<TableUpdate<KeyValuePair<TKey, TResult>, TKey>> ParentDependent<T, TKey, TResult>(
+         this IObservable<TableUpdate<T, TKey>> source,
+         Func<T, TKey> key,
+         Func<T, TKey> parent_key,
+         Func<T, T, TResult> get_result,
+         Func<T, TResult> get_result_no_parent,
+         IEqualityComparer<TResult> result_comparer,
+         TKey no_parent_key)
+         where TKey : IEquatable<TKey> =>
+         Observable.Create<TableUpdate<KeyValuePair<TKey, TResult>, TKey>>(observer => {
+            var value_comparer = EqualityComparer.Create<KeyValuePair<TKey, TResult>, TResult>(kv => kv.Value, result_comparer);
+            var state = DictionaryHelper.empty<TKey, T>();
+            var output = TableUpdate<KeyValuePair<TKey, TResult>, TKey>.mk_empty(kv => kv.Key);
+            var parent_to_children = new SetMap<TKey, T>();
+            return source.Subscribe(update);
+            //
+            void update(TableUpdate<T, TKey> input) {
+               state = input.state;
+               var builder = output.to_builder();
+               foreach (var v in input.removed) {
+                  var k = input.key_extractor(v);
+                  builder.remove(k);
+                  parent_to_children.remove(parent_key(v), v);
+                  if (parent_to_children.try_lookup(k, out ISet<T> children)) {
+                     foreach (var child in children)
+                        builder.add_or_update(new KeyValuePair<TKey, TResult>(key(child), get_result_no_parent(child)));
+                  }
+               }
+               foreach (var v in input.changed) {
+                  var k = key(v);
+                  builder.add_or_update(new KeyValuePair<TKey, TResult>(k, lookup_result(v)));
+                  parent_to_children.set(parent_key(v), v);
+                  if (parent_to_children.try_lookup(k, out ISet<T> children)) {
+                     foreach (var child in children)
+                        builder.add_or_update(new KeyValuePair<TKey, TResult>(key(child), get_result(child, v)));
+                  }
+               }
+               foreach (var v in input.added) {
+                  var k = key(v);
+                  builder.add_or_update(new KeyValuePair<TKey, TResult>(k, lookup_result(v)));
+                  parent_to_children.add(parent_key(v), v);
+                  if (parent_to_children.try_lookup(k, out ISet<T> children)) {
+                     foreach (var child in children)
+                        builder.add_or_update(new KeyValuePair<TKey, TResult>(key(child), get_result(child, v)));
+                  }
+               }
+               var result = builder.to_table_checked(value_comparer);
+               if (result.has_changes())
+                  observer.OnNext(output = result);
+            }
+            TResult lookup_result(T child) {
+               var pk = parent_key(child);
+               return !pk.Equals(no_parent_key) && state.TryGetValue(pk, out T parent) ?  get_result(child, parent) : get_result_no_parent(child);
+            }
+         });
+
+      static class _Merge<T, TKey> where TKey : IEquatable<TKey> {
+         public static readonly Func<TableUpdate<T, TKey>, Change<TableUpdate<T, TKey>>, TableUpdate<T, TKey>, TableUpdate<T, TKey>> source_change_del = _source_changed_impl;
+         public static readonly Func<TableUpdate<T, TKey>, TableUpdate<T, TKey>, Change<TableUpdate<T, TKey>>, TableUpdate<T, TKey>> other_change_del = _other_changed_impl;
+
+         static TableUpdate<T, TKey> _source_changed_impl(TableUpdate<T, TKey> target, Change<TableUpdate<T, TKey>> source_change, TableUpdate<T, TKey> other_table) =>
+            TableUpdate<T, TKey>.update(target, source_change.New.added, source_change.New.changes, source_change.New.removed, source_change.New.key_extractor);
+
+         static TableUpdate<T, TKey> _other_changed_impl(TableUpdate<T, TKey> target, TableUpdate<T, TKey> source_table, Change<TableUpdate<T, TKey>> other_change) =>
+            TableUpdate<T, TKey>.update(target, other_change.New.added, other_change.New.changes, other_change.New.removed, other_change.New.key_extractor);
+      }
+   }
+}
+namespace bts.utils.diagnostics {
+   public static class DiagnosticUpdateExtensions {
+      // Hard to imagine why this util would be needed outside of diagnosig bugs.
+      // The #if DEBUG directive is an extra precaution to avoid accidentally writing
+      // files like this on a client's machine.
+#if DEBUG
+      static readonly string _proto_capture_ext = "proto_capture";
+
+      public static IObservable<TableUpdate<T, TKey>> ProtoBufCapture<T, TKey, TProto>(this IObservable<TableUpdate<T, TKey>> source, Func<T, TProto> to_message)
+         where TKey : IEquatable<TKey>
+         where TProto : class, IMessage<TProto> =>
+         ProtoBufCapture(source,
+            FilePathHelper.GetUniquePathWithDate(Path.Combine(Directory.GetCurrentDirectory(), typeof(TProto).Name), _proto_capture_ext),
+            to_message);
+
+      public static IObservable<TableUpdate<T, TKey>> ProtoBufCapture<T, TKey, TProto>(this IObservable<TableUpdate<T, TKey>> source, string path, Func<T, TProto> to_message)
+         where TKey : IEquatable<TKey>
+         where TProto : class, IMessage<TProto> =>
+         Observable.Create<TableUpdate<T, TKey>>(o => {
+            // write the state on dispose, and/or on error
+            IReadOnlyDictionary<TKey, T> state = DictionaryHelper.empty<TKey, T>();
+            var sub = source.Subscribe(table => {
+               state = table.state;
+               o.OnNext(table);
+            }, ex => {
+               save(true);
+               o.OnError(ex);
+            },
+            o.OnCompleted);
+            return Disposable.Create(() => {
+               save(false);
+               Disposable.dispose(ref sub);
+            });
+            void save(bool has_error) {
+               if (state.Count > 0)
+                  FileStreamHelper.WriteProtos(has_error ? $"{path}(error)" : path, state.Values.Select(to_message).ExceptNull());
+            }
+         });
+#endif
+   }
+}

src/client/spark_test/Resources/ResourceHelper.cs

@@ -0,0 +1,13 @@
+using System.IO;
+using System.Reflection;
+
+namespace bts.resources {
+   class ResourceHelper {
+      const string _default_namespace = "spark_test.Resources";
+
+      public static Stream LoadResource(string name) {
+         var ass = Assembly.GetExecutingAssembly();
+         return ass.GetManifestResourceStream(name) ?? ass.GetManifestResourceStream($"{_default_namespace}.{name}");
+      }
+   }
+}

src/client/spark_test/Security/SecurityTests.cs

@@ -1,13 +1,11 @@
-using System;
+using bts.api;
+using bts.resources;
+using bts.test;
+using bts.utils.proto;
 using System.Collections.Generic;
 using System.Linq;
-using System.Text;
-using System.Threading.Tasks;
-using bts.api;
-using bts.test;
 
-namespace bts.mcore
-{
+namespace bts.mcore {
    [TestGroup]
    class SecurityTests {
       [Test]
@@ -28,5 +26,23 @@ namespace bts.mcore
          Assert.That(dict[b] == 1);
          Assert.That(dict[c] == 2);
       }
+
+      [Test]
+      public void test_security_compare() {
+         const string securities_resource = "KrandallOrderBookCrash.security_proto";
+         // verify that the sample size doesn't shrink for some reason.
+         // if it grows a bit that's fine; the generic comparison test
+         // asserts against collections that are too large.
+         const int min_sample_size = 91;
+
+         var securities = StreamHelper.ReadProtos<Security>(
+            ResourceHelper.LoadResource(securities_resource))
+            .Select(s => new SecurityImmutable(s))
+            .ToList();
+         Assert.IsTrue(securities.Count >= min_sample_size);
+
+         ComparisonTests.transitive_comparison(securities, SecurityHelper.Comparer.InstanceImmutable);
+         ComparisonTests.identity_comparison(securities, SecurityHelper.Comparer.InstanceImmutable);
+      }
    }
 }

src/client/spark_test/Utils/ComparisonTests.cs

@@ -0,0 +1,34 @@
+using bts.test;
+using System.Collections.Generic;
+using System.Linq;
+
+namespace bts.mcore {
+   static class ComparisonTests {
+      public static void transitive_comparison<T>(IEnumerable<T> items, IComparer<T> comparer) {
+         // Verify that the sample size isn't too big.
+         // This test compares each item to every other item
+         // (technically twice; lhs/rhs is flipped the second time),
+         // so performance is O(n^2).
+         const int max_sample_size = 4092;
+         var sorted_items = items.ToList();
+         Assert.That(sorted_items.Count <= max_sample_size);
+         sorted_items.Sort(comparer);
+         for (int i = 0; i < sorted_items.Count - 1; i++) {
+            for (int j = i + 1; j < sorted_items.Count; j++) {
+               assert_comp(i, j);
+            }
+         }
+         void assert_comp(int lhs, int rhs) {
+            int pos_comp = lhs.CompareTo(rhs);
+            // hit both sides for good measure
+            Assert.That(comparer.Compare(sorted_items[lhs], sorted_items[rhs]) == pos_comp);
+            Assert.That(comparer.Compare(sorted_items[rhs], sorted_items[lhs]) == -pos_comp);
+         }
+      }
+
+      public static void identity_comparison<T>(IEnumerable<T> items, IComparer<T> comparer) {
+         foreach (var v in items)
+            Assert.That(comparer.Compare(v, v) == 0);
+      }
+   }
+}

src/client/spark_test/Utils/MarketDataUtilities.cs

@@ -61,7 +61,7 @@ public static async Task<IEnumerable<SecurityImmutable>> get_multiple_securities
          return Enumerable.Empty<SecurityImmutable>();
       }
 
-      public static async Task<IEnumerable<FutureContract>> get_expirations(MarketDataRelay relay, string symbol, SecurityType security_type, MDRequest.Types.Flags flags = MDRequest.Types.Flags.Bbo, Venue v = Venue.Sim) {
+      public static async Task<(bool, List<FutureContract>)> get_expirations(MarketDataRelay relay, string symbol, SecurityType security_type, MDRequest.Types.Flags flags = MDRequest.Types.Flags.Bbo, Venue v = Venue.Sim) {
          var request = new MDRequest() {
             Host = _MD_HOST,
             Channels = {
@@ -89,7 +89,7 @@ public static async Task<IEnumerable<SecurityImmutable>> get_multiple_securities
             }
          }
 
-         return (IEnumerable<FutureContract>) futures;
+         return (result.success, futures);
       }
 
    }

src/client/spark_test/Utils/SpreaderUtilities.cs

@@ -22,8 +22,9 @@ namespace bts.tests
    public class SpreaderUtilities {
 
       public static async Task<(SecurityImmutable s1, SecurityImmutable s2)> get_securities_for_common_setup_one(MarketDataRelay relay, string symbol) {
-         var exps  = (await MarketDataUtilities.get_expirations(relay, symbol, SecurityType.Future)).ToList();
+         var (success, exps) = await MarketDataUtilities.get_expirations(relay, symbol, SecurityType.Future);
 
+         Assert.IsTrue(success, "Could not reach market data service");
          Assert.IsNotNull(exps);
          Assert.IsTrue(exps.Count() > 2, "Need at least two expirations");
 

src/client/spark_test/spark_test.csproj

@@ -14,5 +14,7 @@
   <ItemGroup>
     <Content Include="../spark_test/system_test_scripts/*.sh" CopyToOutputDirectory="PreserveNewest"></Content>
   </ItemGroup>
-
+  <ItemGroup>
+    <EmbeddedResource Include="Resources\KrandallOrderBookCrash.security_proto" />
+  </ItemGroup>
 </Project>

src/data_files/files/onixs_ice/ConnectivityConfiguration.APITest.xml

@@ -15,7 +15,7 @@
         <server name="default">
             <ip>63.247.113.163</ip>
             <port>3000</port>
-            <userName>bluetrd-ps</userName>
+            <userName>katas2f-ps</userName>
             <password>Starts123</password>
             <getStripInfoMessages>N</getStripInfoMessages>
             <strategyPreference>New</strategyPreference>
@@ -24,8 +24,8 @@
         <server name="endex">
             <ip>63.247.113.77</ip>
             <port>3000</port>
-            <userName>Value from ICE Support</userName>
-            <password>Value from ICE Support</password>
+            <userName>katas2f-ps</userName>
+            <password>Starts123</password>
             <getStripInfoMessages>N</getStripInfoMessages>
             <strategyPreference>New</strategyPreference>
             <localNetworkInterface></localNetworkInterface>

src/lib/CMakeDeps.cmake

@@ -1 +1 @@
-set(sources api/apicore.proto;api/apigateway.proto;api/apioptions.proto;api/core/ApiClientInterop.cpp;api/core/ApiServerInterop.cpp;api/public/ApiServer.cpp;api/public/ClientChannel.cpp;api/public/QuickbusClientChannel.cpp;api/public/ServerChannel.cpp;api/public/ServiceContext.cpp;api/service_helpers/AdminServiceUtils.cpp;api/service_helpers/AlgoServiceUtils.cpp;api/service_helpers/PositionUtilities.cpp;api/services/AdminService.proto;api/services/AlgoDefinitionService.proto;api/services/AlgoService.proto;api/services/ApiUnitTestService.proto;api/services/AuthService.proto;api/services/BasePositionService.proto;api/services/ErrorReportService.proto;api/services/HelloService.proto;api/services/MBOService.proto;api/services/MarginService.proto;api/services/MarketDataAuxiliaryService.proto;api/services/MarketDataRelay.proto;api/services/MarketDataService.proto;api/services/MoneyFileService.proto;api/services/OrderService.proto;api/services/OutOfProcessURLListener.proto;api/services/PersistenceService.proto;api/services/PositionService.proto;api/services/ShellDiagnosticService.proto;api/services/ShellLatencyService.proto;api/services/SyntheticSecurityService.proto;api/test/ApiTests.cpp;api/test/HelloServiceServer.cpp;config/ConfigCheck.cpp;config/ConfigDownloader.cpp;config/ConfigErrorCollector.cpp;config/ConfigLog.cpp;config/ConfigPropertyEvaluator.cpp;config/ConfigValueQueue.cpp;config/ConfigurationException.cpp;config/ConfigurationState.cpp;config/PreprocessInputStream.cpp;config/config.cpp;config/defaults/MarketData.conf;config/defaults/Ports.conf;config/defaults/Symbols.conf;config/defaults/cme_reference_data.conf;config/defaults/edge.conf;config/defaults/jetfix.conf;config/defaults/keycode.conf;config/defaults/logging.conf;config/defaults/orderman_cfg.conf;config/defaults/orders_fills_cleanup.conf;config/defaults/position_cfg.conf;config/defaults/span.conf;config/defaults/spreader.conf;config/defaults/testkeys.conf;config/defaults/threads.conf;config/defaults/toggles.conf;config/extensions/extensions.proto;config/invalid/invalid.proto;config/protos/ExchangeSimCfg.proto;config/protos/MarketData.proto;config/protos/Ports.proto;config/protos/Symbols.proto;config/protos/UI.proto;config/protos/cme_reference_data.proto;config/protos/jetfix.proto;config/protos/keycode.proto;config/protos/logging.proto;config/protos/mercury_cfg.proto;config/protos/orderman_cfg.proto;config/protos/orders_fills_cleanup.proto;config/protos/perflog_cfg.proto;config/protos/position_cfg.proto;config/protos/quickbus_cfg.proto;config/protos/span.proto;config/protos/spreader.proto;config/protos/subprocess.proto;config/protos/system.proto;config/protos/testkeys.proto;config/protos/threads.proto;config/protos/toggles.proto;config/tests/HTTPServer.cpp;config/tests/config_ref_tests.cpp;config/tests/config_tests.cpp;config/text_format/text_format.cpp;domain_model/Security.cpp;domain_model/Security.proto;domain_model/SecurityTests.cpp;domain_model/sod.cpp;domain_model/tests/domain_tests.cpp;expression/BooleanGrammar.cpp;expression/ExpressionEvaluator.cpp;expression/ParserExceptions.cpp;expression/Value.cpp;expression/tests/expression_parser_tests.cpp;jetfix/codec/FIXArray.cpp;jetfix/common/OutFIX.cpp;jetfix/connection/AsyncReader.cpp;jetfix/connection/IConnection.cpp;jetfix/connection/SocketConnection.cpp;jetfix/gateway/CoreGateway.cpp;jetfix/gateway/JetFIXGateway.cpp;jetfix/gateway/Session.cpp;jetfix/store/FIXFileStore.cpp;jetfix/store/session_name.cpp;jetfix/tests/async_reader_tests.cpp;jetfix/tests/decoding_tests.cpp;jetfix/tests/fix_encoding_stream_tests.cpp;jetfix/tests/fix_fragment_tests.cpp;jetfix/tests/jetfix_performance_tests.cpp;jetfix/tests/sample_fix.cpp;jetfix/tests/store_tests.cpp;logging/LoggerHub.cpp;logging/Sink.cpp;logging/logging.cpp;logging/tests/logging_tests.cpp;md/Diag/MDPeriodicDumper.cpp;md/Diag/ShmDumper.cpp;md/MBOProcessor.cpp;md/MBOServiceServer.cpp;md/MDClient.cpp;md/MDConsumer.cpp;md/MDConsumerHandle.cpp;md/MDConsumerInterop.cpp;md/MDProducer.cpp;md/MDProducerHandle.cpp;md/MemorySegmentUtils.cpp;md/PlatformLoader.cpp;md/PriceSize.cpp;md/Product.cpp;md/SecurityCache.cpp;md/SecurityDB.cpp;md/SecurityIDRecorder.cpp;md/SecurityMapper.cpp;md/SecurityOracle.cpp;md/SettlementRecorder.cpp;md/Shm/FixedSecurity.cpp;md/SyntheticSecurityService.cpp;md/cme/MDP3Utils.cpp;md/common.cpp;md/ice/ICEUtils.cpp;md/protos/co2.proto;md/relay/QueryServer.cpp;md/relay/RelayClient.cpp;md/relay/RelayRecorder.cpp;md/service/MarketDataServiceServer.cpp;md/test/Co2ManagedTestHelper.cpp;md/test/Co2TestInstrument.cpp;md/test/mbo_tests.cpp;md/test/md2_tests.cpp;md/test/md_test_utils.cpp;md/test/synthetic_book_merge_tests.cpp;md/vendor/OnixsUtils.cpp;mercury/actors/IPollableActor.cpp;mercury/broker/Broker.cpp;mercury/broker/BrokerProxy.cpp;mercury/broker/ILocalBroker.cpp;mercury/broker/IRemoteBroker.cpp;mercury/common/mercury_common.cpp;mercury/events/TimerEvent.cpp;mercury/executor/Executor.cpp;mercury/topics/Topic.cpp;metrics/metrics.cpp;metricsLine truncated
+set(sources api/apicore.proto;api/apigateway.proto;api/apioptions.proto;api/core/ApiClientInterop.cpp;api/core/ApiServerInterop.cpp;api/public/ApiServer.cpp;api/public/ClientChannel.cpp;api/public/QuickbusClientChannel.cpp;api/public/ServerChannel.cpp;api/public/ServiceContext.cpp;api/service_helpers/AdminServiceUtils.cpp;api/service_helpers/AlgoServiceUtils.cpp;api/service_helpers/PositionUtilities.cpp;api/services/AdminService.proto;api/services/AlgoDefinitionService.proto;api/services/AlgoService.proto;api/services/ApiUnitTestService.proto;api/services/AuthService.proto;api/services/BasePositionService.proto;api/services/ErrorReportService.proto;api/services/HelloService.proto;api/services/MBOService.proto;api/services/MarginService.proto;api/services/MarketDataAuxiliaryService.proto;api/services/MarketDataRelay.proto;api/services/MarketDataService.proto;api/services/MoneyFileService.proto;api/services/OrderService.proto;api/services/OutOfProcessURLListener.proto;api/services/PersistenceService.proto;api/services/PositionService.proto;api/services/ShellDiagnosticService.proto;api/services/ShellLatencyService.proto;api/services/SyntheticSecurityService.proto;api/test/ApiTests.cpp;api/test/HelloServiceServer.cpp;config/ConfigCheck.cpp;config/ConfigDownloader.cpp;config/ConfigErrorCollector.cpp;config/ConfigLog.cpp;config/ConfigPropertyEvaluator.cpp;config/ConfigValueQueue.cpp;config/ConfigurationException.cpp;config/ConfigurationState.cpp;config/PreprocessInputStream.cpp;config/config.cpp;config/defaults/MarketData.conf;config/defaults/Ports.conf;config/defaults/Symbols.conf;config/defaults/cme_reference_data.conf;config/defaults/edge.conf;config/defaults/jetfix.conf;config/defaults/keycode.conf;config/defaults/logging.conf;config/defaults/orderman_cfg.conf;config/defaults/orders_fills_cleanup.conf;config/defaults/position_cfg.conf;config/defaults/span.conf;config/defaults/spreader.conf;config/defaults/testkeys.conf;config/defaults/threads.conf;config/defaults/toggles.conf;config/extensions/extensions.proto;config/invalid/invalid.proto;config/protos/ExchangeSimCfg.proto;config/protos/MarketData.proto;config/protos/Ports.proto;config/protos/Symbols.proto;config/protos/UI.proto;config/protos/cme_reference_data.proto;config/protos/jetfix.proto;config/protos/keycode.proto;config/protos/logging.proto;config/protos/mercury_cfg.proto;config/protos/orderman_cfg.proto;config/protos/orders_fills_cleanup.proto;config/protos/perflog_cfg.proto;config/protos/position_cfg.proto;config/protos/quickbus_cfg.proto;config/protos/span.proto;config/protos/spreader.proto;config/protos/subprocess.proto;config/protos/system.proto;config/protos/testkeys.proto;config/protos/threads.proto;config/protos/toggles.proto;config/tests/HTTPServer.cpp;config/tests/config_ref_tests.cpp;config/tests/config_tests.cpp;config/text_format/text_format.cpp;domain_model/Security.cpp;domain_model/Security.proto;domain_model/SecurityTests.cpp;domain_model/sod.cpp;domain_model/tests/domain_tests.cpp;expression/BooleanGrammar.cpp;expression/ExpressionEvaluator.cpp;expression/ParserExceptions.cpp;expression/Value.cpp;expression/tests/expression_parser_tests.cpp;jetfix/codec/FIXArray.cpp;jetfix/common/OutFIX.cpp;jetfix/connection/AsyncReader.cpp;jetfix/connection/IConnection.cpp;jetfix/connection/SocketConnection.cpp;jetfix/gateway/CoreGateway.cpp;jetfix/gateway/JetFIXGateway.cpp;jetfix/gateway/Session.cpp;jetfix/store/FIXFileStore.cpp;jetfix/store/session_name.cpp;jetfix/tests/async_reader_tests.cpp;jetfix/tests/decoding_tests.cpp;jetfix/tests/fix_encoding_stream_tests.cpp;jetfix/tests/fix_fragment_tests.cpp;jetfix/tests/jetfix_performance_tests.cpp;jetfix/tests/sample_fix.cpp;jetfix/tests/store_tests.cpp;logging/LoggerHub.cpp;logging/Sink.cpp;logging/logging.cpp;logging/tests/logging_tests.cpp;md/Diag/MDPeriodicDumper.cpp;md/Diag/ShmDumper.cpp;md/MBOProcessor.cpp;md/MBOServiceServer.cpp;md/MDClient.cpp;md/MDConsumer.cpp;md/MDConsumerHandle.cpp;md/MDConsumerInterop.cpp;md/MDProducer.cpp;md/MDProducerHandle.cpp;md/MemorySegmentUtils.cpp;md/PlatformLoader.cpp;md/PriceSize.cpp;md/Product.cpp;md/SecurityCache.cpp;md/SecurityDB.cpp;md/SecurityIDRecorder.cpp;md/SecurityMapper.cpp;md/SecurityOracle.cpp;md/SettlementRecorder.cpp;md/Shm/FixedSecurity.cpp;md/SyntheticSecurityService.cpp;md/cme/MDP3Utils.cpp;md/common.cpp;md/ice/ICEUtils.cpp;md/protos/co2.proto;md/relay/QueryServer.cpp;md/relay/RelayClient.cpp;md/relay/RelayRecorder.cpp;md/service/MarketDataServiceServer.cpp;md/test/Co2ManagedTestHelper.cpp;md/test/Co2TestInstrument.cpp;md/test/mbo_tests.cpp;md/test/md2_tests.cpp;md/test/md_test_utils.cpp;md/test/synthetic_book_merge_tests.cpp;md/vendor/OnixsUtils.cpp;mercury/actors/IPollableActor.cpp;mercury/broker/Broker.cpp;mercury/broker/BrokerProxy.cpp;mercury/broker/ILocalBroker.cpp;mercury/broker/IRemoteBroker.cpp;mercury/common/mercury_common.cpp;mercury/events/TimerEvent.cpp;mercury/executor/Executor.cpp;mercury/topics/Topic.cpp;metrics/metrics.cpp;metricsLine truncated

src/lib/api/service_helpers/AdminServiceUtils.cpp

@@ -685,6 +685,7 @@ bts::api::admin::BTSAccount* AdminServiceUtils::create_account(bts::api::admin::
    new_account_ptr->set_name(account_name);
    new_account_ptr->set_description(description);
    new_account_ptr->set_is_active(is_active);
+   new_account_ptr->mutable_risk_limit_enforcement()->set_ignore_spread_position_limits(true);
    return new_account_ptr;
 }
 void AdminServiceUtils::update_account(bts::api::admin::AdminData& admin_data,

src/lib/api/services/OrderService.proto

@@ -383,7 +383,9 @@ message OCOOrderState {}
 message PricerOrderState {}
 message ScratchOrderState {}
 message ProRataOrderState {}
-message DynamicOrderState {}
+message DynamicOrderState {
+  map<string,DynamicOrderParam> custom_state = 1;
+}
 message QueueHolderState {}
 message SplitterOrderState {}
 
@@ -644,7 +646,7 @@ message RegisteredDynamicOrderTypes {
 
 message DynamicOrderParam {
   oneof value {
-    uint32 as_int32 = 1;
+    int32 as_int32 = 1;
     uint32 as_uint32 = 2;
     double as_double = 3;
     string as_string = 4;

src/lib/config/defaults/MarketData.conf

@@ -1,136 +1,130 @@
-#
-# Common Market data options
-#
-
-MarketDataConfig {
-   rule: "DeploymentType.type == 'PROD'"
-   value {
-      use_sim: false
-   }
-}
-
-PublisherOptions {
-    rule: "DeploymentType.type == 'dev'"
-    value {
-        disable_heartbeat: true
-    }
-}
-
-PlatformMap {
-    value {
-        glbx_platform: "PLATFORM_MDP3ONIXS"
-        ice_platform: "PLATFORM_ICEONIXS"
-    }
-}
-
-PublisherNamespace {
-	value {
-		shared_memory_namespace: @System.os_user
-	}
-}
-
-CMEOptions {
-    value {
-        feed_engine {
-            name: "mdp3feed"
-            # udp_socket_buffer_size: 425984
-            # channels_per_thread: 1
-            spin_time: 0
-        }
-        channel {
-            id: "*"
-            feed_engine: "mdp3feed"
-            use_tcp_replay: false
-        }
-    }
-}
-
-ICEPublisherOptions {
-   value { 
-      network_interface: "eth1"
-   }
-}
-
-#to make this work in dev clone the operations repo
-#go here: c:\\code\\bts\\operations\\ice\\ICE_iMpactMulticastFeed_GettingStarted-1_1_36\\bin\\easyTunnel>
-#modify the easyTunnelProxy.properties file to replace 
-#tunnel.multicast.interface=10.96.18.37 with your local ip
-#run easyTunnelProxy.bat BRENT-FI for Brent Full implied
-#you should see ice data
-
-ICEOptions {
-   value {
-      channel {
-         name: "oil" #these names are arbitrary and are for debugging
-         market_type: 3 
-         uds_market_type: 3
-         market_type: 4 #ICE IPE Gas Oil Futures
-         uds_market_type: 4         
-         market_type: 5 #ICE Futures Brent, IPE Brent Futures
-         uds_market_type: 5
-         market_type: 9 #ICE Futures WTI, ICE WTI Crude Futures                  
-         uds_market_type: 9
-         market_type: 13 #ICE Heating Oil Futures         
-         uds_market_type: 13
-         market_type: 15 #ICE NYH (RBOB) Gasoline Futures
-         uds_market_type: 15
-         ##market_type: 21 #ICE Futures US Agricultures, Coffee C
-         ##uds_market_type: 21
-         ##market_type: 23 #ICE Futures US Agricultures, FCOJ A
-         ##uds_market_type: 23               
-         
-         ##market_type: 26 #ICE Futures US Financials DX
-         ##uds_market_type: 26
-         
-         uds_market_type: 11 #ICE WTI-Brent spreads
-         uds_market_type: 14 #ICE Heating Oil/WTI Futures Crack
-         uds_market_type: 16 #ICE NYH (RBOB) Gasoline/WTI Futures Crack         
-         uds_market_type: 58 #ICE Futures UI Heating Oil / Brent
-         uds_market_type: 59 #ICE NYH (RBOB) Gasoline/Brent Futures Crack 
-         uds_market_type: 66 #ICE NYH (RBOB) Gasoline/Heating Oil Spread
-         uds_market_type: 70 #ICE Heating Oil/LS Gasoil Futures Spread
-         uds_market_type: 71 #ICE LS Gasoil/Brent Futures Crack
-         uds_market_type: 72 #ICE NYH (RBOB) Gasoline/LS Gasoil Futures Spread         
-         
-         #onixs_log_mode: Debug
-      }
-      # channel {
-      #    name: "juice"
-      #    market_type: 26 #ICE Futures US Agricultures, FCOJ A
-      # }
-      ice_config_file: "ConnectivityConfiguration.APITest.xml"      
-      #ice_config_file: "ConnectivityConfiguration.PerfTest.xml"      
-   }
-}
-
-MDBootDelayWindow {
-    rule: "DeploymentType.type == 'dev'"
-    value {
-        delay_seconds: 0
-    }
-}
-
-MDBootDelayWindow {
-    value {
-        delay_seconds: 120
-        standard {
-            start: "16:30"
-            end: "16:40"
-        }
-        sunday {
-            start: "14:00"
-            end: "14:10"
-        }
-    }
-}
-
-DisplayFactorOverride {
-    value {
-        override {
-            symbol: "SOM"
-            exchange: "GLBX"
-            type: "SPREAD"
-            price_multiplier: 1
-        }
-    }
-}
+#
+# Common Market data options
+#
+
+MarketDataConfig {
+   rule: "DeploymentType.type == 'PROD'"
+   value {
+      use_sim: false
+   }
+}
+
+PublisherOptions {
+    rule: "DeploymentType.type == 'dev'"
+    value {
+        disable_heartbeat: true
+    }
+}
+
+PlatformMap {
+    value {
+        glbx_platform: "PLATFORM_MDP3ONIXS"
+        ice_platform: "PLATFORM_ICEONIXS"
+    }
+}
+
+PublisherNamespace {
+	value {
+		shared_memory_namespace: @System.os_user
+	}
+}
+
+CMEOptions {
+    value {
+        feed_engine {
+            name: "mdp3feed"
+            # udp_socket_buffer_size: 425984
+            # channels_per_thread: 1
+            spin_time: 0
+        }
+        channel {
+            id: "*"
+            feed_engine: "mdp3feed"
+            use_tcp_replay: false
+        }
+    }
+}
+
+#to make this work in dev clone the operations repo
+#go here: c:\\code\\bts\\operations\\ice\\ICE_iMpactMulticastFeed_GettingStarted-1_1_36\\bin\\easyTunnel>
+#modify the easyTunnelProxy.properties file to replace 
+#tunnel.multicast.interface=10.96.18.37 with your local ip
+#run easyTunnelProxy.bat BRENT-FI for Brent Full implied
+#you should see ice data
+
+ICEOptions {
+   value {
+      channel {
+         name: "oil" #these names are arbitrary and are for debugging
+         market_type: 3 
+         uds_market_type: 3
+         market_type: 4 #ICE IPE Gas Oil Futures
+         uds_market_type: 4         
+         market_type: 5 #ICE Futures Brent, IPE Brent Futures
+         uds_market_type: 5
+         market_type: 9 #ICE Futures WTI, ICE WTI Crude Futures                  
+         uds_market_type: 9
+         market_type: 13 #ICE Heating Oil Futures         
+         uds_market_type: 13
+         market_type: 15 #ICE NYH (RBOB) Gasoline Futures
+         uds_market_type: 15
+         ##market_type: 21 #ICE Futures US Agricultures, Coffee C
+         ##uds_market_type: 21
+         ##market_type: 23 #ICE Futures US Agricultures, FCOJ A
+         ##uds_market_type: 23               
+         
+         ##market_type: 26 #ICE Futures US Financials DX
+         ##uds_market_type: 26
+         
+         uds_market_type: 11 #ICE WTI-Brent spreads
+         uds_market_type: 14 #ICE Heating Oil/WTI Futures Crack
+         uds_market_type: 16 #ICE NYH (RBOB) Gasoline/WTI Futures Crack         
+         uds_market_type: 58 #ICE Futures UI Heating Oil / Brent
+         uds_market_type: 59 #ICE NYH (RBOB) Gasoline/Brent Futures Crack 
+         uds_market_type: 66 #ICE NYH (RBOB) Gasoline/Heating Oil Spread
+         uds_market_type: 70 #ICE Heating Oil/LS Gasoil Futures Spread
+         uds_market_type: 71 #ICE LS Gasoil/Brent Futures Crack
+         uds_market_type: 72 #ICE NYH (RBOB) Gasoline/LS Gasoil Futures Spread         
+         
+         #onixs_log_mode: Debug
+      }
+      # channel {
+      #    name: "juice"
+      #    market_type: 26 #ICE Futures US Agricultures, FCOJ A
+      # }
+      ice_config_file: "ConnectivityConfiguration.APITest.xml"      
+      #ice_config_file: "ConnectivityConfiguration.PerfTest.xml"      
+   }
+}
+
+MDBootDelayWindow {
+    rule: "DeploymentType.type == 'dev'"
+    value {
+        delay_seconds: 0
+    }
+}
+
+MDBootDelayWindow {
+    value {
+        delay_seconds: 120
+        standard {
+            start: "16:30"
+            end: "16:40"
+        }
+        sunday {
+            start: "14:00"
+            end: "14:10"
+        }
+    }
+}
+
+DisplayFactorOverride {
+    value {
+        override {
+            symbol: "SOM"
+            exchange: "GLBX"
+            type: "SPREAD"
+            price_multiplier: 1
+        }
+    }
+}

src/lib/config/protos/MarketData.proto

@@ -52,6 +52,11 @@ message PublisherOptions
    optional uint32 heartbeat_boot_seconds = 12 [default = 60]; // 1 minute grace period from shm creation
 
    optional bool skip_options = 13 [default = true];
+
+   optional uint32 hot_initial = 20 [default = 1024];
+   optional uint32 hot_growth = 21 [default = 1024];
+   optional uint32 cold_initial = 22 [default = 8192];
+   optional uint32 cold_growth = 23 [default = 2048];
 }
 
 message ICEPublisherOptions
@@ -193,6 +198,11 @@ message RecordOptions {
    optional bool record_sim = 5 [default=false];
 }
 
+// If starting on Sunday pre-load the selected symbols from disk
+message SecurityPreload {
+   repeated string symbol = 1;
+}
+
 //
 // Replay options
 // 

src/lib/config/protos/toggles.proto

@@ -67,4 +67,5 @@ message DynamicAlgos{
    optional bool enable_dorman = 1 [default = false];
    optional bool enable_pyramid = 2 [default = false];
    optional bool enable_tas = 3 [default = false];
+   optional bool enable_tascounter = 4 [default = false];
 }

src/lib/md/MDConsumer.cpp

@@ -792,7 +792,7 @@ public:
             // We can safely send a security key to every _instance because by definition
             // a security cannot appear in 2 instances
             if (unresolved_securities.size() > 250) {
-               LOG(md,Warn,"Failed to resolve legs for {}"
+               LOG(md,Info,"Failed to resolve legs for {}"
                    " spreads. This can happen when a spread is subscribed without "
                    "it's underliers and may lead to higher than expected CPU use.",
                    unresolved_securities.size() );
@@ -901,6 +901,10 @@ public:
       return _cache.get_security(key);
    }
 
+   std::vector<api::Security const *> get_securities() const {
+      return _cache.get_securities();
+   }
+
    MDConsumerChannel get_channel(const std::string& symbol, SecurityType type, Venue exchange) const {
       for (auto& inst : _instances) {
          if ((inst->symbol() == symbol && inst->type() == type && inst->exchange() == exchange) ||
@@ -976,6 +980,7 @@ size_t MDConsumer::walk_security_list(SecurityCallback cb, const string& symbol,
 size_t MDConsumer::scan_securities(bool resolve_legs) { return _pimpl->scan_securities(resolve_legs); }
 void MDConsumer::on_security(SecurityCallback cb, bool resolve_legs) { _pimpl->on_security(cb, resolve_legs); }
 const api::Security* MDConsumer::get_security(SecurityKey key) const { return _pimpl->get_security(key).get(); }
+std::vector<api::Security const*> MDConsumer::get_securities() const {return _pimpl->get_securities();}
 
 MDConsumerChannel MDConsumer::get_channel(const std::string& symbol, SecurityType type, Venue exchange) const {
    return _pimpl->get_channel(symbol, type, exchange);

src/lib/md/MDConsumer.h

@@ -59,6 +59,9 @@ public:
    // You still need to call scan_securities() periodically to discover new securities
    const api::Security* get_security(SecurityKey key) const;
 
+   // Retrieve all cached securities
+   std::vector<api::Security const*> get_securities() const;
+
    MDConsumerChannel get_channel(const std::string& symbol, SecurityType type, Venue exchange) const;
    MDConsumerChannel get_channel(const UserSymbol&) const;
 

src/lib/md/MDProducer.cpp

@@ -20,8 +20,14 @@ public:
       _symbol(symbol),
       _type(type),
       _exchange(exchange),
-      _hot(hot_segment_name(symbol, type, exchange, platform), 1024, 1024, flags & MDProducerFlags::Shared),
-      _cold(cold_segment_name(symbol, type, exchange, platform), 8192, 2048, flags & MDProducerFlags::Shared),
+      _hot(hot_segment_name(symbol, type, exchange, platform), 
+         config::get<PublisherOptions>().hot_initial(), 
+         config::get<PublisherOptions>().hot_growth(),
+         flags & MDProducerFlags::Shared),
+      _cold(cold_segment_name(symbol, type, exchange, platform), 
+         config::get<PublisherOptions>().cold_initial(), 
+         config::get<PublisherOptions>().cold_growth(), 
+         flags & MDProducerFlags::Shared),
       _trades(_cold.header().trade_buffer, BTS_ARRAYSIZE(_cold.header().trade_buffer), true),
       _rfqs(_cold.header().rfq_buffer, BTS_ARRAYSIZE(_cold.header().rfq_buffer), true)
       {

src/lib/md/SecurityCache.cpp

@@ -69,6 +69,17 @@ SecurityPtr SecurityCache::get_security(SecurityKey key) {
    }
 }
 
+std::vector<api::Security const*> SecurityCache::get_securities()
+{
+   std::vector<api::Security const*> result;
+   result.reserve(100000);
+
+   std::unique_lock<std::mutex> lock(_me->cache_mutex);
+   for (auto const&p : _me->cache)
+      result.push_back(p.second.get());
+   return result;
+}
+
 SecurityPtr SecurityCache::add_db_security(const SecurityPtr& security) {
    bts_assert(security);
    std::lock_guard<std::mutex> lock(_me->cache_mutex);
@@ -90,4 +101,4 @@ bool SecurityCache::add_live_security(const SecurityPtr& security) {
 
 void SecurityCache::flush_to_disk() { _me->flush(); }
 
-}}//ns
+}}//ns

src/lib/md/SecurityCache.h

@@ -22,6 +22,9 @@ public:
    //Load security from cache with fallback to database
    SecurityPtr get_security(SecurityKey key); 
 
+   // Load all securities from cache, no fallback to database
+   std::vector<api::Security const*> get_securities();
+   
    // Get the security if it's in the cache otherwise return nullptr. Does not query security database.
    SecurityPtr try_get_security(SecurityKey key) const;
 
@@ -33,4 +36,4 @@ public:
 
 };
 
-}}//ns
+}}//ns

src/lib/md/SecurityOracle.cpp

@@ -10,4 +10,8 @@ const api::Security* SecurityOracle::get_security(SecurityKey key) const {
    return _client.current_consumer().get_security(key);
 }
 
-}}//namespace
+std::vector<api::Security const*> SecurityOracle::get_securities() const {
+   return _client.current_consumer().get_securities();
+}
+
+}}//namespace

src/lib/md/SecurityOracle.h

@@ -1,6 +1,8 @@
 #pragma once
+
 #include "lib/domain_model/SecurityKey.h"
 #include <memory>
+#include <vector>
 
 namespace bts {
 namespace api {class Security;}
@@ -12,6 +14,7 @@ public:
    SecurityOracle(const MDClient&);
    ~SecurityOracle();
    const api::Security* get_security(SecurityKey key) const;
+   std::vector<api::Security const*> get_securities() const;
 };
 
-}}//namespace
+}}//namespace

src/lib/md/vendor/OnixsUtils.cpp

@@ -18,16 +18,16 @@ const char * onixs_license[] = {
 "    <name>Blue Trading Systems OF 9420-2</name>",
 "  </customer>",
 "  <period>",
-"    <since>2019-05-02</since>",
-"    <till>2020-05-13</till>",
+"    <since>2020-05-01</since>",
+"    <till>2021-05-12</till>",
 "  </period>",
 "  <products>",
 "    <product id=\"4\" name=\"CME Market Data Handler for C++\" />",
 "    <product id=\"7\" name=\"ICE iMpact Multicast Price Feed Handler for C++\" />",
 "    <product id=\"67\" name=\"Eurex Market and Reference Data Interfaces (EMDI, RDI) Handler for C++\" />",
 "  </products>",
-"  <checksum>5E50A1DACD94D7C1BECD316EEAE5AB02</checksum>",
-"</license>",
+"  <checksum>1C9210495AA3C9C884B7A37A4C7AD8F9</checksum>",
+"</license>"
 };
 
 OnixsTemplLicense onixs_create_temporary_license() {

src/lib/orderman/actors/OrderStore.cpp

@@ -154,10 +154,13 @@ void OrderStore::store_and_publish_if_complete(
 
 bool OrderStore::_order_cutoff(const api::orderman::OrderRecord &rec) {
    // TODO: move this to orderutils
-   bool working = (rec.state().status() ==
-                   api::orderman::OrderStatus::ORDER_STATUS_WORKING) ||
-                  (rec.state().status() ==
-                   api::orderman::OrderStatus::ORDER_STATUS_PENDING_WORKING);
+   bool working =
+     (rec.state().status() ==
+      api::orderman::OrderStatus::ORDER_STATUS_WORKING) ||
+     (rec.state().status() ==
+      api::orderman::OrderStatus::ORDER_STATUS_DISCONNECTED) ||
+     (rec.state().status() ==
+      api::orderman::OrderStatus::ORDER_STATUS_PENDING_WORKING);
    return me->soft_cleanup_enabled && !working &&
           rec.state().created().seconds() < me->cutoff_time.tv_sec;
 }

src/lib/orderman/actors/SODPosition.cpp

@@ -7,6 +7,7 @@
 #include "lib/utils/PBUtils.h"
 #include "lib/utils/bts_assert.h"
 #include <google/protobuf/timestamp.pb.h>
+#include <google/protobuf/util/time_util.h>
 #include <sstream>
 #include <sys/time.h>
 
@@ -53,7 +54,7 @@ timeval SODPosition::sod_time() {
    // if the server started before the sod for today
    if (rv.tv_sec >= now_epoch)
       rv = subtract_days (rv,1);
-   return rv;   
+   return rv;
 }
 
 timeval SODPosition::get_order_cutoff_time(
@@ -98,6 +99,48 @@ timeval SODPosition::now(){
    return tv;
 }
 
+google::protobuf::Timestamp SODPosition::today_local_time_to_utc_timestamp (
+                                                   uint16_t hours,
+                                                   uint16_t mins,
+                                                   uint16_t secs,
+                                                   bool add_day_if_less,
+                                                   bool subtract_day_if_more){
+   time_t now_epoch;
+   time(&now_epoch); // seconds since epoch (utc) (independent of machine timezone)
+   auto now_ts = google::protobuf::util::TimeUtil::TimeTToTimestamp(now_epoch);
+   struct tm *ptm_local = localtime(&now_epoch);  // in local timezone
+   ptm_local->tm_hour = hours;
+   ptm_local->tm_min = mins;
+   ptm_local->tm_sec = secs;
+   ptm_local->tm_isdst = -1;
+   time_t ovrd_epoch = mktime(ptm_local);  // converts local time to UTC epoch time
+   if(ovrd_epoch < now_epoch){
+      if (add_day_if_less)
+         ovrd_epoch += 24*60*60;
+   }
+   else{
+      if(subtract_day_if_more)
+         ovrd_epoch -= 24*60*60;
+   }
+   auto ovrd_ts = google::protobuf::util::TimeUtil::TimeTToTimestamp(ovrd_epoch);
+   return ovrd_ts;
+}
+
+google::protobuf::Timestamp SODPosition::add_seconds (
+                                       const google::protobuf::Timestamp& ts,
+                                       int secs_diff){
+   auto secs = google::protobuf::util::TimeUtil::TimestampToSeconds(ts);
+   secs += secs_diff;
+   return google::protobuf::util::TimeUtil::SecondsToTimestamp(secs);
+}
+
+google::protobuf::Timestamp SODPosition::add_nanoseconds (
+                                       const google::protobuf::Timestamp& ts,
+                                       long diff){
+   return google::protobuf::util::TimeUtil::NanosecondsToTimestamp(
+      google::protobuf::util::TimeUtil::TimestampToNanoseconds(ts) + diff);
+}
+
 google::protobuf::Timestamp SODPosition::to_timestamp (timeval tv){
    google::protobuf::Timestamp rv;
    rv.set_seconds(tv.tv_sec);

src/lib/orderman/actors/SODPosition.h

@@ -31,6 +31,18 @@ public:
                      const std::map<std::string,std::string>& ordermap);
    static timeval now();
    static google::protobuf::Timestamp to_timestamp (timeval);
+   static google::protobuf::Timestamp today_local_time_to_utc_timestamp (
+            uint16_t hours,
+            uint16_t mins,
+            uint16_t secs,
+            bool add_day_if_less,
+            bool subtract_day_if_more);
+   static google::protobuf::Timestamp add_seconds (
+                                 const google::protobuf::Timestamp& ts,
+                                 int secs);
+   static google::protobuf::Timestamp add_nanoseconds (
+                                 const google::protobuf::Timestamp& ts,
+                                 long diff);
    bool is_sod (const google::protobuf::Timestamp& fill_time) const;
 };
 

src/lib/orderman/actors/order_manager/OrderManager.cpp

@@ -157,6 +157,7 @@ void OrderManager::on_filled_position(
       }
 
       for (auto const &ev : *(update.events)) {
+         me->services_data.position.update(ev);
          me->services_data.limiter.set_filled_position(
              *ev.security, ev.account_id, ev.long_quantity, ev.short_quantity);
       }
@@ -550,6 +551,8 @@ void OrderManager::init_order_services() {
        std::make_unique<ContentSubscription<TimerEvent, NoKey>>(*this, log);
    me->services_data.order_creation_subs =
        std::make_unique<ContentSubscription<CreateOrderReplyEvent>>(*this, log);
+   me->services_data.position_subs =
+      std::make_unique<ContentSubscription<FilledPositionEvent,NoKey>>(*this, log);
 }
 
 void OrderManager::after_event() {

src/lib/orderman/actors/order_manager/OrderServices.cpp

@@ -32,6 +32,7 @@ void OrderServices::init(OrderServicesData &data) {
    _status_updates.init(*data.status_subs);
    _settlement_updates.init(*data.settlement_subs);
    _time_change.init(*data.time_subs);
+   _position_update.init(*data.position_subs);
    if (_sessions.empty())
       for (auto &sd : data.sessions)
          _sessions.emplace_back(*sd);

src/lib/orderman/actors/order_manager/OrderServices.h

@@ -8,6 +8,7 @@
 #include "lib/orderman/actors/order_manager/SessionServices.h"
 #include "lib/orderman/events/TickerUpdateEvent.h"
 #include "lib/orderman/events/StatusUpdateEvent.h"
+#include "lib/orderman/events/FilledPositionEvent.h"
 #include "lib/orderman/governor/Governor.h"
 #include "lib/orderman/orderman_utils/EventStream.h"
 #include "lib/orderman/orders/Order.h"
@@ -62,6 +63,10 @@ class OrderServices {
       _data->log.log(corid(), entry_kind, uint64_1, txt_1, uint32_1);
    }
 
+   std::vector<PositionUpdate> const& current_positions() {
+      return _data->position.current_positions();
+   }
+
    trace::ScopedPerfLogEntry log_scope(const char *entry_kind,
                                        uint64_t uint64_1 = 0,
                                        const char *txt_1 = "",
@@ -123,6 +128,15 @@ class OrderServices {
    void unsubscribe_from_time() { _time_change.unsub(_time_change_sub); }
    // Time
 
+
+   // Position Updates
+   template <class CB> void subscribe_to_position_updates(CB &&cb) {
+      _position_update_sub = _position_update.sub(
+          [this, cb](auto const &ev) { this->on_external_event(ev, cb); });
+   }
+   void unsubscribe_from_position_updates() { _position_update.unsub(_position_update_sub); }
+   // Position Updates
+   
    // Execution Reports
    using ExeReportSubID = EventStream<NewExecutionReportEvent>::SubID;
    template <class CB>
@@ -298,8 +312,10 @@ class OrderServices {
    EventStream<StatusUpdateEvent> _status_updates;
    EventStream<SettlementUpdateEvent> _settlement_updates;
    EventStream<mercury::TimerEvent, mercury::NoKey> _time_change;
+   EventStream<orderman::FilledPositionEvent, mercury::NoKey> _position_update;
    boost::container::small_vector<SessionServices, 20> _sessions;
    EventStream<mercury::TimerEvent, mercury::NoKey>::SubID _time_change_sub;
+   EventStream<orderman::FilledPositionEvent, mercury::NoKey>::SubID _position_update_sub;
    EventStream<CreateOrderReplyEvent> _create_order_reply;
 }; // namespace orderman
 } // namespace orderman

src/lib/orderman/actors/order_manager/OrderServicesData.h

@@ -35,6 +35,7 @@
 #include "lib/orderman/order_router/OrderRouter.h"
 #include "lib/utils/datetime.h"
 #include "lib/utils/Puid.h"
+#include "lib/orderman/actors/order_manager/PositionIndex.h"
 
 #include <boost/date_time/posix_time/posix_time.hpp>
 
@@ -87,6 +88,7 @@ class OrderServicesData {
    OrderManager &owner;
    trace::PerfLog &log;
    securities::MapSecurityDB sec_db;
+   PositionIndex position;
    OrderRouter order_router;
    Limiter limiter;
    OrderFactory order_factory;
@@ -101,6 +103,8 @@ class OrderServicesData {
        time_subs;
    std::unique_ptr<mercury::ContentSubscription<CreateOrderReplyEvent>>
        order_creation_subs;
+   std::unique_ptr<mercury::ContentSubscription<FilledPositionEvent,mercury::NoKey>>
+       position_subs;
 
    mercury::Stream<CreateOrderRequestEvent> create_order_req_pub;
    mercury::Stream<CreateFIXBufferRequestEvent> create_fix_req_pub;

src/lib/orderman/actors/order_manager/PositionIndex.cpp

@@ -0,0 +1,31 @@
+#include "PositionIndex.h"
+
+#include "lib/orderman/events/FilledPositionEvent.h"
+#include "lib/orderman/orders/algo/PositionUpdate.h"
+
+namespace bts {
+namespace orderman {
+
+struct PositionIndex::PImpl {
+   PImpl() : position_idx(), current_positions() {}
+   std::map<std::pair<SecurityKey,AccountID>,size_t> position_idx;
+   std::vector<PositionUpdate> current_positions;
+};
+
+void PositionIndex::update(FilledPositionEvent const&ev) {
+   auto &idx = me->position_idx[{SecurityKey(ev.security->id()),ev.account_id}];
+   if (idx == 0) {
+      me->current_positions.push_back({});
+      idx = me->current_positions.size();
+   }
+   me->current_positions[idx-1] = {ev.security,ev.account_id,ev.long_quantity, ev.short_quantity};
+}
+
+std::vector<PositionUpdate> const& PositionIndex::current_positions() const {
+   return me->current_positions;
+}
+
+PositionIndex::PositionIndex() : me (new PImpl()) {}
+PositionIndex::~PositionIndex() {}
+
+}}

src/lib/orderman/actors/order_manager/PositionIndex.h

@@ -0,0 +1,23 @@
+#pragma once
+
+#include <memory>
+#include <vector>
+
+namespace bts {
+namespace orderman {
+
+class FilledPositionEvent;
+struct PositionUpdate;
+
+class PositionIndex {
+public:
+   PositionIndex();
+   ~PositionIndex(); 
+   void update(FilledPositionEvent const&);
+   std::vector<PositionUpdate> const &current_positions() const;
+private:
+   struct PImpl;
+   std::unique_ptr<PImpl> me;
+};
+
+}}

src/lib/orderman/events/OrderStateUpdateEvent.h

@@ -11,16 +11,16 @@ namespace bts {
 namespace orderman {
 
 class OrderStateUpdateEvent : public mercury::Event<OrderStateUpdateEvent> {
- public:
+public:
    using SynopsisPolicy = mercury::CacheAll<OrderStateUpdateEvent>;
 
    OrderStateUpdateEvent()
       : key(null_order_key()), parent_key(null_order_key()) {}
 
    OrderStateUpdateEvent(
-       OrderKey k, OrderKey pk,
-       std::shared_ptr<const api::orderman::OrderDefinition> const &def,
-       api::orderman::OrderState const &state) {
+      OrderKey k, OrderKey pk,
+      std::shared_ptr<const api::orderman::OrderDefinition> const &def,
+      api::orderman::OrderState const &state) {
       init(k, pk, def, state);
    }
 
@@ -41,18 +41,21 @@ class OrderStateUpdateEvent : public mercury::Event<OrderStateUpdateEvent> {
       display_status = state.display_status();
 
       custom = state.custom_case();
-
-      auto native_state = get_native_state(state);
-      if (native_state) {
-         exchange_order_id = native_state->exchange_order_id();
-         last_msg_seq_num = native_state->last_msg_seq_num();
-         sub_error_code = native_state->sub_error_code();
-         native_status = native_state->native_status();
-         routing_record = native_state->routing_record();
-         native_type = native_state->native_type();
-         orderman::str_to(native_state->last_transact_time(),
-                          &last_transact_time);
-         stop_px = native_state->stop_px();
+      if (custom == api::orderman::OrderState::kDynamic) {
+         dynamic_order_state = state.dynamic();
+      } else {
+         auto native_state = get_native_state(state);
+         if (native_state) {
+            exchange_order_id = native_state->exchange_order_id();
+            last_msg_seq_num = native_state->last_msg_seq_num();
+            sub_error_code = native_state->sub_error_code();
+            native_status = native_state->native_status();
+            routing_record = native_state->routing_record();
+            native_type = native_state->native_type();
+            orderman::str_to(native_state->last_transact_time(),
+                             &last_transact_time);
+            stop_px = native_state->stop_px();
+         }
       }
    }
 
@@ -70,7 +73,6 @@ class OrderStateUpdateEvent : public mercury::Event<OrderStateUpdateEvent> {
    uint32_t filled_at_attach_time_quantity;
    std::string last_msg;
    api::orderman::OrderState::DisplayStatus display_status;
-   
    api::orderman::OrderState::CustomCase custom;
 
    // native order state
@@ -82,7 +84,10 @@ class OrderStateUpdateEvent : public mercury::Event<OrderStateUpdateEvent> {
    api::orderman::NativeOrderType native_type;
    mercury::TimeStampStr last_transact_time;
    double stop_px;
-   
+
+   // dynamic order state
+   std::optional<api::orderman::DynamicOrderState> dynamic_order_state;
+
    api::orderman::OrderState quick_rebuild_state() const {
       api::orderman::OrderState result;
       result.set_status(status);
@@ -111,11 +116,11 @@ class OrderStateUpdateEvent : public mercury::Event<OrderStateUpdateEvent> {
          break;
       case api::orderman::OrderState::kStopLimit:
          rebuild_native_state(
-             result.mutable_stop_limit()->mutable_native_state());
+            result.mutable_stop_limit()->mutable_native_state());
          break;
       case api::orderman::OrderState::kGlbxIceberg:
          rebuild_native_state(
-             result.mutable_glbx_iceberg()->mutable_native_state());
+            result.mutable_glbx_iceberg()->mutable_native_state());
          break;
       case api::orderman::OrderState::kIceberg:
          result.mutable_iceberg();
@@ -163,7 +168,7 @@ class OrderStateUpdateEvent : public mercury::Event<OrderStateUpdateEvent> {
          result.mutable_prorata();
          break;
       case api::orderman::OrderState::kDynamic:
-         result.mutable_dynamic();
+         *result.mutable_dynamic() = dynamic_order_state.value();
          break;
       case api::orderman::OrderState::kSplitter:
          result.mutable_splitter();
@@ -175,7 +180,7 @@ class OrderStateUpdateEvent : public mercury::Event<OrderStateUpdateEvent> {
       return result;
    }
 
- private:
+private:
    void rebuild_native_state(api::orderman::NativeOrderState *st) const {
       st->set_stop_px(stop_px);
       st->set_exchange_order_id(exchange_order_id);

src/lib/orderman/orders/GLBXNativeOrder.cpp

@@ -190,21 +190,18 @@ void NativeOrder::on_init_recovered(OrderRecord const &rec,
          return;
       }
 
-      // bail if the session we used for the order is no longer valid
-      auto const &session = native_state.routing_record().session();
-      if (!serv.order_router.is_valid_session(session)) {
-         LOG(OrderMan, Error,
-             "Unknown JetFIX session {} referenced by recovered "
-             "order {}. This is a serious "
-             "problem which has to be corrected manually. ",
-             session, rec.key());
+      auto rq = serv.order_router.get_routing_record(
+         UserID(def.user_id()), AccountID(def.account_id()), *me->sec);
+      if (!rq.has_route) {
+         auto msg = fmt::format("Could not route order: {}", rq.error_msg);
+         LOG(OrderMan, Error, "Unable to route order {} recovered from disk. Message: {}",
+             rec.key(), msg);
          transition_to_error(NativeOrderState::SUB_ERROR_UNKNOWN_FIX_SESSION,
-                             fmt::format("Unknown FIX session: {}", session));
+                             "Could not route order");
          return;
       }
-
       // if we made it this far we are good.
-      me->routing_record = native_state.routing_record();
+      me->routing_record = rq.routing_record; 
       if (me->state.status() != ORDER_STATUS_PENDING_WORKING)
          me->exchange_order_id = native_state.exchange_order_id();
       init_fix();

src/lib/orderman/orders/ICENativeOrder.cpp

@@ -215,13 +215,6 @@ api::orderman::OrderDefinition
 ICENativeOrder::on_init_from_exe_report(const ExecutionReport& er, const PreActivationOrderServices& os) 
 {
    auto k = er.order_key;
-   auto nextid = me->services->gen_id();
-   if(k.value() < nextid)
-   {
-      k = OrderKey(nextid);
-      LOG(OrderMan, Error, "Updating recovered order id with newly generated id {}", k.value());
-   }
-
    OrderRecord orecord;
    orecord.set_key(k.value());
    orecord.set_parent_key(null_order_key().value());

src/lib/orderman/orders/algo/AlgoWrapperOrder.cpp

@@ -56,11 +56,12 @@ struct AlgoWrapperOrder::PImpl {
    OrderState const &state() const { return _state; }
 
    Map<SecurityKey, OrderServices::BookSubID> book_subs;
+   Map<SecurityKey, OrderServices::SettlementSubID> settlement_subs;
    Map<SecurityKey, OrderServices::TickerSubID> ticker_subs;
    Map<SecurityKey, OrderServices::StatusSubID> status_subs;
    Map<OrderKey, ChildRecord> children;
    bool has_timer_sub = false;
-
+   bool has_position_sub = false;
    bool is_state_dirty = false;
 
    Map<Edge, Action> transition_actions;
@@ -118,6 +119,13 @@ struct AlgoWrapperOrder::PImpl {
                                            : OrderState::HIDDEN);
    }
 
+   void update_custom_state(api::orderman::DynamicOrderState const&cstate) {
+      is_state_dirty = true;
+      bts_assert(definition.has_custom_def());
+      bts_assert(definition.custom_def().has_dynamic());
+      *_state.mutable_dynamic() = cstate;
+   }
+   
    void set_key(OrderKey k) { _my_key = k; }
 
  private:
@@ -382,6 +390,10 @@ api::Security const *AlgoWrapperOrder::lookup_security(SecurityKey k) const {
    return me->security_db->resolve(k);
 }
 
+std::vector<api::Security const *> AlgoWrapperOrder::known_securities() const {
+   return me->security_db->get_securities();
+}
+
 Price AlgoWrapperOrder::price() const { return Price(me->state().price()); }
 
 Quantity AlgoWrapperOrder::quantity() const {
@@ -396,6 +408,10 @@ bool AlgoWrapperOrder::is_shutting_down() const {
    return me->status == Status::SHUTTING_DOWN;
 }
 
+std::vector<PositionUpdate> const&AlgoWrapperOrder::current_positions() const {
+   return me->services->current_positions();
+}
+
 void AlgoWrapperOrder::subscribe_book(SecurityKey k) {
    if (!is_working())
       return;
@@ -416,6 +432,26 @@ void AlgoWrapperOrder::unsubscribe_book(SecurityKey k) {
       me->services->unsubscribe_from_book(it->second);
 }
 
+void AlgoWrapperOrder::subscribe_settlement(SecurityKey k) {
+   if (!is_working())
+      return;
+   bts_assert(me->settlement_subs.find(k) == me->settlement_subs.end());
+
+   me->settlement_subs[k] =
+       me->services->subscribe_to_settlements(k, [this, k](auto const &ev) {
+          ReportStateGuard _(*this);
+          me->algo->on_settlement(k, Price(ev.settlement_price.price()));
+       });
+}
+
+void AlgoWrapperOrder::unsubscribe_settlement(SecurityKey k) {
+   if (!is_working())
+      return;
+   bts_assert(me->settlement_subs.find(k) != me->settlement_subs.end());
+   if (auto it = me->settlement_subs.find(k); it != me->settlement_subs.end())
+      me->services->unsubscribe_from_settlements(it->second);
+}
+
 void AlgoWrapperOrder::subscribe_ticker(SecurityKey k) {
    if (!is_working())
       return;
@@ -497,6 +533,25 @@ void AlgoWrapperOrder::unsubscribe_timer() {
    me->has_timer_sub = false;
 }
 
+void AlgoWrapperOrder::subscribe_position() {
+   if (!is_working())
+      return;
+   bts_assert(!me->has_position_sub);
+   me->has_position_sub = true;
+   me->services->subscribe_to_position_updates([this](auto const &ev) {
+      ReportStateGuard _(*this);
+      me->algo->on_position({ev.security,ev.account_id,ev.long_quantity, ev.short_quantity});
+   });
+}
+
+void AlgoWrapperOrder::unsubscribe_position() {
+   if (!is_working())
+      return;
+   bts_assert(me->has_position_sub);
+   me->services->unsubscribe_from_position_updates();
+   me->has_position_sub = false;
+}
+
 bool AlgoWrapperOrder::is_attaching() const {
    return me->attach_target != null_order_key();
 }
@@ -610,6 +665,12 @@ void AlgoWrapperOrder::set_ladder_visibility(bool is_visible) {
    me->set_ladder_visibility(is_visible);
 }
 
+void AlgoWrapperOrder::update_custom_state(api::orderman::DynamicOrderState const&cstate) {
+   if (!is_working())
+      return;
+   me->update_custom_state(cstate);
+}
+
 void AlgoWrapperOrder::cancel(std::string const &msg) {
    if (!is_working())
       return;
@@ -841,6 +902,8 @@ void AlgoWrapperOrder::check_invariants() {
       bts_assert(is_attaching());
       bts_assert(me->external_status() == ORDER_STATUS_WORKING);
       bts_assert(me->book_subs.empty());
+      bts_assert(me->settlement_subs.empty());
+      bts_assert(me->ticker_subs.empty());
       bts_assert(me->children.empty());
       bts_assert(!me->has_timer_sub);
       bts_assert(!me->pending_terminate.has_value());
@@ -860,6 +923,8 @@ void AlgoWrapperOrder::check_invariants() {
       bts_assert(me->security);
       bts_assert(me->external_status() == ORDER_STATUS_WORKING);
       bts_assert(me->book_subs.empty());
+      bts_assert(me->settlement_subs.empty());
+      bts_assert(me->ticker_subs.empty());
       bts_assert(!me->has_timer_sub);
       bts_assert(!me->pending_update.has_value());
       bts_assert(me->pending_terminate.has_value());
@@ -873,6 +938,8 @@ void AlgoWrapperOrder::check_invariants() {
                  me->external_status() == ORDER_STATUS_ERROR ||
                  me->external_status() == ORDER_STATUS_REJECTED_BY_SPARK);
       bts_assert(me->book_subs.empty());
+      bts_assert(me->settlement_subs.empty());
+      bts_assert(me->ticker_subs.empty());
       bts_assert(!me->has_timer_sub);
       bts_assert(!me->pending_update.has_value());
       bts_assert(!me->pending_terminate.has_value());
@@ -919,10 +986,18 @@ void AlgoWrapperOrder::shutdown(api::orderman::OrderStatus st,
 }
 
 void AlgoWrapperOrder::cleanup() {
-   for (auto const &s : me->book_subs) {
+   for (auto const &s : me->book_subs) 
       me->services->unsubscribe_from_book(s.second);
-      me->book_subs.clear();
-   }
+   me->book_subs.clear();
+
+   for (auto const &s : me->settlement_subs) 
+      me->services->unsubscribe_from_settlements(s.second);
+   me->settlement_subs.clear();
+
+   for (auto const &s : me->ticker_subs) 
+      me->services->unsubscribe_from_ticker(s.second);
+   me->ticker_subs.clear();
+
    for (auto &c : me->children)
       c.second.order.cancel();
    if (me->has_timer_sub)

src/lib/orderman/orders/algo/AlgoWrapperOrder.h

@@ -33,6 +33,8 @@ class AlgoWrapperOrder : public Order,
    OrderKey order_key() const override;
    OrderKey parent_order_key() const override;
    api::Security const *lookup_security(SecurityKey k) const override;
+   std::vector<api::Security const *> known_securities() const override;
+
    bool is_attaching() const override;
    ChildTemplate &make_child_template(api::orderman::OrderDefinition const &,
                                       size_t) override;
@@ -41,8 +43,12 @@ class AlgoWrapperOrder : public Order,
    Quantity quantity() const override;
    Quantity total_fills() const override;
    bool is_shutting_down() const override;
+   std::vector<PositionUpdate> const&current_positions() const override;
+
    void subscribe_book(SecurityKey) override;
    void unsubscribe_book(SecurityKey) override;
+   void subscribe_settlement(SecurityKey) override;
+   void unsubscribe_settlement(SecurityKey) override;
    void subscribe_child_change(OrderKey) override;
    void unsubscribe_child_change(OrderKey) override;
    void subscribe_timer() override;
@@ -51,6 +57,8 @@ class AlgoWrapperOrder : public Order,
    void unsubscribe_ticker(SecurityKey) override;
    void subscribe_status(SecurityKey) override;
    void unsubscribe_status(SecurityKey) override;
+   void subscribe_position() override;
+   void unsubscribe_position() override;
 
    OrderKey launch_child(ChildTemplate &, Price, Quantity,
                          bool subscribe = true) override;
@@ -64,6 +72,7 @@ class AlgoWrapperOrder : public Order,
    void terminate_error(std::string const &msg) override;
    void complete() override;
    void update_fill_state(Quantity total_fills, Price last_price) override;
+   void update_custom_state(api::orderman::DynamicOrderState const&) override;
    void set_ladder_visibility (bool is_visibile) override;
    
    // Private

src/lib/orderman/orders/algo/IAlgoOrder.cpp

@@ -3,6 +3,7 @@
 
 namespace bts {
 namespace orderman {
+#include "lib/orderman/orders/algo/IAlgoOrder.h"
 IAlgoOrder::~IAlgoOrder() {}
 
 OrderOpResult IAlgoOrder::do_before_launch(IBeforeLaunchAlgoServices &) {

src/lib/orderman/orders/algo/IAlgoOrder.h

@@ -2,6 +2,8 @@
 
 #include "lib/orderman/orders/Order.h"
 #include "lib/orderman/events/TickerUpdateEvent.h"
+#include "lib/orderman/orders/algo/PositionUpdate.h"
+
 #include <memory>
 
 namespace bts {
@@ -44,6 +46,12 @@ class IAlgoOrder {
    // The trading status for the specified security has changed
    virtual void on_status(SecurityKey, co2::TradingStatus) {};
 
+   // The settlement price for the specified security has changed
+   virtual void on_settlement(SecurityKey, Price) {};
+   
+   // The current position for the security, account combination
+   virtual void on_position(PositionUpdate const&) {};
+
    // A periodic timer has fired
    virtual void on_timer() {}
 

src/lib/orderman/orders/algo/IAlgoServices.h

@@ -1,10 +1,15 @@
 #pragma once
 
 #include "lib/orderman/orders/algo/IBeforeLaunchAlgoServices.h"
+#include "lib/orderman/orders/algo/PositionUpdate.h"
 
 namespace bts {
 namespace orderman {
 
+
+// Services to an algo *after* it has been launched. See
+// IBeforeLaunchAlgoServices for services available before launch,
+// that is outside of the latency sensitive path.
 class IAlgoServices : public IBeforeLaunchAlgoServices {
  public:
    // --------------------------------------------------------------
@@ -26,6 +31,13 @@ class IAlgoServices : public IBeforeLaunchAlgoServices {
    // cancel child orders will be ignored.
    virtual bool is_shutting_down() const = 0;
 
+   // Returns the current position per security and account. The
+   // returned reference is only valid immediately after the
+   // call. Callers should make a copy if they wish to use it beyond
+   // that. Callers can keep track of changes to the position by
+   // subscribing to the position update events below.
+   virtual std::vector<PositionUpdate> const&current_positions() const = 0;
+
    // --------------------------------------------------------------
    // Event Subscriptions
    // --------------------------------------------------------------
@@ -34,6 +46,10 @@ class IAlgoServices : public IBeforeLaunchAlgoServices {
    virtual void subscribe_book(SecurityKey) = 0;
    virtual void unsubscribe_book(SecurityKey) = 0;
 
+   // Subscribe to settlement prices for the specified security
+   virtual void subscribe_settlement(SecurityKey) = 0;
+   virtual void unsubscribe_settlement(SecurityKey) = 0;
+
    // Subscribe to child order updates for the specified child
    // order. This is rarely needed since launch_child() by default
    // automatically subscribes to new orders.
@@ -52,6 +68,10 @@ class IAlgoServices : public IBeforeLaunchAlgoServices {
    virtual void subscribe_status(SecurityKey) = 0;
    virtual void unsubscribe_status(SecurityKey) = 0;
 
+   // Subscribe to position updates
+   virtual void subscribe_position() = 0;
+   virtual void unsubscribe_position() = 0;
+
    // --------------------------------------------------------------
    // Child management
    // --------------------------------------------------------------
@@ -102,6 +122,9 @@ class IAlgoServices : public IBeforeLaunchAlgoServices {
    // Set wheter this algo instance should be displayed in the Spark UI ladder
    virtual void set_ladder_visibility(bool is_visible) = 0;
 
+   // Report custom algo state for display in the Spark UI
+   virtual void update_custom_state(api::orderman::DynamicOrderState const&) = 0;
+   
    // Update this algos's filled quantity and last price
    virtual void update_fill_state(Quantity total_fills, Price last_price) = 0;
 };

src/lib/orderman/orders/algo/IBeforeLaunchAlgoServices.h

@@ -31,6 +31,9 @@ class IBeforeLaunchAlgoServices {
    // The security identified by k or nullptr if no such security exists
    virtual api::Security const *lookup_security(SecurityKey k) const = 0;
 
+   // Known active securities
+   virtual std::vector<api::Security const*> known_securities() const = 0;
+   
    // Whether this algo is being attached to an existing order
    // If an algo does not support attaching it can return an error 
    // if is_attaching() == true from do_before_launch.

src/lib/orderman/orders/algo/PositionUpdate.cpp

@@ -0,0 +1,15 @@
+#include "PositionUpdate.h"
+#include "lib/orderman/orderman_utils/orderman_common.h"
+
+namespace bts {
+namespace orderman {
+
+std::ostream& operator<<(std::ostream&out,const PositionUpdate &update) {
+   out << "Security:  " << update.security->id()
+       << " Account: " << update.account_id.value()
+       << " Long Qty: " << update.long_quantity.value()
+       << " Short Qty: " << update.short_quantity.value();
+   return out;
+}
+
+}}

src/lib/orderman/orders/algo/PositionUpdate.h

@@ -0,0 +1,21 @@
+#pragma once
+
+#include "lib/domain_model/common.h"
+#include "lib/orderman/orderman_utils/orderman_common.h"
+
+namespace bts {
+namespace api { class Security;}
+
+namespace orderman {
+
+struct PositionUpdate {
+   api::Security const *security;
+   AccountID account_id;
+   Quantity long_quantity;
+   Quantity short_quantity;
+};
+
+std::ostream& operator<<(std::ostream&,const PositionUpdate &);
+
+}}
+

src/lib/orderman/orders/algo/algo_orders/RegisterDorman.cpp

@@ -8,6 +8,7 @@
 #include "DormanAlgo.h"
 #include "PyramidAlgo.h"
 #include "TASAlgo.h"
+#include "TASCounterAlgo.h"
 
 using namespace bts;
 using namespace api::orderman;
@@ -148,6 +149,45 @@ extern "C" void init_pyramid_plugin(bts::SparkServices& srv) {
    srv.register_dynamic_algo_order(join_desc, []() { return std::make_unique<orderman::PyramidAlgo>(); });
 }
 
+extern "C" void init_tascounter_plugin(bts::SparkServices& srv) {
+   DynamicOrderDescriptor join_desc;
+
+   join_desc.set_dynamic_order_type("TASCounter");
+   // all 6 contracts. They can theoretically be inferred from the front month
+   // since T1 are calendars 1 month apart starting from the front month
+   // ex Mar-May, May-Jul, Jul-Sep being T1 front, second and third respectively
+   // (assuming Mar is the front month)
+   *join_desc.add_param_desc() = param_security_key("T1 Front");
+   *join_desc.add_param_desc() = param_security_key("T1 Second");
+   *join_desc.add_param_desc() = param_security_key("T1 Third");
+   *join_desc.add_param_desc() = param_security_key("T2 Front");
+   *join_desc.add_param_desc() = param_security_key("T2 Second");
+   *join_desc.add_param_desc() = param_security_key("T3 Front");
+   // tas contracts (could have been derived)
+   *join_desc.add_param_desc() = param_security_key("T1 Front TAS");
+   *join_desc.add_param_desc() = param_security_key("T1 Second TAS");
+   *join_desc.add_param_desc() = param_security_key("T1 Third TAS");
+   // automatic orders
+   *join_desc.add_param_desc() = param_uint32("T1 Front Time");
+   *join_desc.add_param_desc() = param_uint32("T1 Front Qty");
+   *join_desc.add_param_desc() = param_bool("T1 Front Order Enabled");
+
+   *join_desc.add_param_desc() = param_uint32("T1 Second Time");
+   *join_desc.add_param_desc() = param_uint32("T1 Second Qty");
+   *join_desc.add_param_desc() = param_bool("T1 Second Order Enabled");
+
+   *join_desc.add_param_desc() = param_uint32("T1 Third Time");
+   *join_desc.add_param_desc() = param_uint32("T1 Third Qty");
+   *join_desc.add_param_desc() = param_bool("T1 Third Order Enabled");
+   // settlement time
+   *join_desc.add_param_desc() = param_uint32("Settlement Start Hour");
+   *join_desc.add_param_desc() = param_uint32("Settlement Start Minutes");
+   *join_desc.add_param_desc() = param_uint32("Settlement Start Seconds");
+   *join_desc.add_param_desc() = param_uint32("Settlement Period (seconds)");
+   
+   srv.register_dynamic_algo_order(join_desc, []() { return std::make_unique<orderman::TASCounterAlgo>(); });
+}
+
 extern "C" void init_tas_plugin(bts::SparkServices& srv) {
    DynamicOrderDescriptor tas_desc;
 
@@ -171,6 +211,8 @@ extern "C" void init_spark_plugin(bts::SparkServices &srv) {
       init_pyramid_plugin(srv);
    if(dyn_algo_config.enable_tas())
       init_tas_plugin(srv);
+   if(dyn_algo_config.enable_tascounter())
+      init_tascounter_plugin(srv);
 }
 
 }}}

src/lib/orderman/orders/algo/algo_orders/TASCounterAlgo.cpp

@@ -0,0 +1,502 @@
+// TODO
+// put settle time data in the input (X)
+// better names (X)
+// at end of settle, order should dettach and complete (X)
+// order going out logic (X)
+// settlement compute logic (X)
+// actual publish (X)
+// no PTime
+// check settle calc (X)
+// check order shooting logic
+
+#include "lib/orderman/orders/algo/algo_orders/TASCounterAlgo.h"
+
+#include "lib/domain_model/Security.h"
+#include "lib/orderman/order_factory/IOrderFactory.h"
+#include "lib/orderman/orderman_utils/Book.h"
+#include "lib/orderman/orderman_utils/orderman_utils.h"
+#include "lib/orderman/orders/algo/IAlgoServices.h"
+#include "lib/orderman/orders/spreader_orders/spreader/SpreaderUtils.h"
+#include "lib/os/os.h"
+#include "lib/domain_model/Security.h"
+#include "lib/utils/PBUtils.h"
+#include "lib/utils/datetime.h"
+#include <google/protobuf/timestamp.pb.h>
+#include "lib/orderman/actors/SODPosition.h"
+#include <google/protobuf/util/time_util.h>
+#include <stdint.h>
+
+namespace bts {
+namespace orderman {
+
+using namespace api::orderman;
+using namespace google::protobuf;
+
+enum class TASCounterAlgo::Status { WORKING, DONE };
+
+// book and trade data for each of the 6 contracts as well as the TAS contracts
+// (the vwap of the TAS contracts is not used, but for uniformity is kept in the
+// the same structure)
+struct ContractMarketData{
+   uint32_t tot_vol = 0;
+   double running_sum = 0;
+   Book book;
+   double estimated_settle = 0.0;
+   void on_trade (const TickerUpdateEvent::Trade& t){
+      tot_vol += t.quantity.value();
+      running_sum += t.price.value()*t.quantity.value();
+   }
+   const double vwap(){
+      return tot_vol == 0 ? os::nan() : running_sum/tot_vol;
+   }
+};
+// the input for the t1 front, second and third orders, as well as the template
+// and working order key are kept here
+struct ContractAutoOrder{
+   // inputs
+   uint32_t n_secs = 0;
+   uint32_t qty = 0;
+   bool enabled = false;
+   // dont know yet which side will be sent
+   ChildTemplate* buy_ord = nullptr;
+   ChildTemplate* sell_ord = nullptr;
+   // computed input
+   // the is AlgoData::settle_stop - n_secs
+   Timestamp algo_start;
+   // output
+   std::optional<OrderKey> tas_order = std::nullopt;
+};
+
+struct AlgoData{
+   double tick_size;
+   Timestamp settle_start;
+   Timestamp settle_stop;
+   ContractMarketData t1_front,
+                      t1_second,
+                      t1_third,
+                      t2_front,
+                      t2_second,
+                      t3_front;
+   ContractMarketData t1_front_tas,
+                      t1_second_tas,
+                      t1_third_tas;
+   ContractAutoOrder t1_front_order,
+                     t1_second_order,
+                     t1_third_order;
+   DynamicOrderState out_state;
+};
+
+struct TASCounterAlgo::PImpl {
+   // reporting
+   uint16_t reporting_period_msecs = 500;
+   bts::datetime::PTime last_report_time = bts::datetime::now();
+   // immutable state
+   IAlgoServices* algo_services = nullptr;
+   // mutable state
+   AlgoData data;
+   // to avoid lookups on md and ticker callbacks
+   std::map<SecurityKey,ContractMarketData*> md_map;
+   Status state = Status::WORKING;   
+};
+
+//fwd decls
+void _copy_output(AlgoData&);
+bool _copy_settle_times (AlgoData&,const Map<std::string,DynamicOrderParam>&);
+void _copy_order_data (AlgoData&,const Map<std::string,DynamicOrderParam>&);
+bool _setup_data(IBeforeLaunchAlgoServices&, AlgoData&,
+                 std::map<SecurityKey,ContractMarketData*>&);
+double round_to_nearest(double price, double tick_size);                 
+long long t1_minus_t2_usecs (const Timestamp&, const Timestamp&);
+long long t_minus_now_usecs (const google::protobuf::Timestamp&);
+bool trade_within_time_win (const Timestamp&, const Timestamp&,
+                            const TickerUpdateEvent::Trade& t);
+bool now_within_time_win (const Timestamp&, const Timestamp&);
+
+
+TASCounterAlgo::TASCounterAlgo() : me(new PImpl()) {}
+TASCounterAlgo::~TASCounterAlgo() {}
+// initialization /////////////////////////////////////////////////////////////
+///////////////////////////////////////////////////////////////////////////////
+OrderOpResult TASCounterAlgo::do_before_launch(IBeforeLaunchAlgoServices& serv) {
+   if(_setup_data(serv,me->data,me->md_map))
+      return {};
+   return {OrderError::ORDER_ERROR_INVALID_PARAMETER,
+           std::string("Parameters are not well formed")};
+}
+void TASCounterAlgo::do_launch(IAlgoServices& algo_services) {
+   me->algo_services = &algo_services;
+   algo_services.subscribe_timer();
+   for (const auto& kp : me->md_map){
+      algo_services.subscribe_book(kp.first);
+      algo_services.subscribe_ticker(kp.first);      
+   }
+}
+void TASCounterAlgo::do_launch_attached(
+                                 IAlgoServices& algo_services, 
+                                 OrderKey,
+                                 ChildState) {
+   // not allowed
+   me->algo_services = &algo_services;
+}
+// updates ////////////////////////////////////////////////////////////////////
+///////////////////////////////////////////////////////////////////////////////
+void TASCounterAlgo::do_update(const api::orderman::Update& u) {
+   if(me->state == Status::DONE)
+      return;
+   const auto& p = u.custom().dynamic().order_params();
+   // only order data change is allowed
+   _copy_order_data(me->data,p);
+   _check_and_send_order();
+}
+// callbacks //////////////////////////////////////////////////////////////////
+///////////////////////////////////////////////////////////////////////////////
+void TASCounterAlgo::on_book(SecurityKey k, const Book& b) {
+   if(me->state == Status::DONE)
+      return;
+   const auto& it = me->md_map.find(k);
+   if(it != me->md_map.end()){
+      it->second->book = b;
+      _check_and_send_order();
+   }
+}
+void TASCounterAlgo::on_ticker(SecurityKey k, const TradeVector& v){
+   if(me->state == Status::DONE)
+      return;
+   const auto& it = me->md_map.find(k);   
+   if(it != me->md_map.end()){
+      auto contract_md = it->second;
+      for (const auto& t : v)
+         if (_within_settle()){
+            contract_md->on_trade(t);
+         }
+      _check_and_send_order();      
+   }
+}
+
+// this only gets called on an explicit algo cancel
+// from the outside
+TASCounterAlgo::OnCancelResponse TASCounterAlgo::on_cancel(){
+   return IAlgoOrder::OnCancelResponse::HONOR;
+}
+
+void TASCounterAlgo::on_timer(){
+   if(me->state == Status::DONE)
+      return;      
+   _check_and_send_order();
+   if(bts::datetime::msecs_since(me->last_report_time) >= 
+                                 me->reporting_period_msecs){
+      me->last_report_time = bts::datetime::now();
+      _report_state();
+   }
+   // check for end and exit
+   if (t1_minus_t2_usecs(util::TimeUtil::GetCurrentTime(),
+                         me->data.settle_stop) > 0){
+      std::string msg = "session over";
+      _cxl_and_end(false,msg);
+   }
+}
+
+IAlgoServices& TASCounterAlgo::_algo() const { return *me->algo_services; }
+
+void TASCounterAlgo::_switch_state(TASCounterAlgo::Status new_state){
+   switch (new_state){
+      case (TASCounterAlgo::Status::WORKING):
+         bts_assert(false);
+         break;
+      case (TASCounterAlgo::Status::DONE):
+         bts_assert(me->state == TASCounterAlgo::Status::WORKING);
+         break;
+   }
+   LOG(OrderMan, Info,"TASCounterAlgo::_switch_state key:{} {}->{} ",
+                      _algo().order_key(),
+                      static_cast<int>(me->state), 
+                      static_cast<int>(new_state));
+   me->state = new_state;
+}
+
+void TASCounterAlgo::_cxl_and_end (bool is_error, const std::string& err_msg){
+   if(me->state == Status::DONE)
+      return;
+   if(is_error)
+      _algo().terminate_error(err_msg);
+   else
+      _algo().complete();
+   _switch_state(TASCounterAlgo::Status::DONE);      
+}
+
+void TASCounterAlgo::_compute_settles(){
+   AlgoData& d = me->data;
+   d.t1_front.estimated_settle = d.t1_front.vwap();
+   d.t1_second.estimated_settle = d.t2_front.vwap() - d.t1_front.vwap();
+   d.t1_third.estimated_settle = d.t3_front.vwap() - d.t1_front.vwap() - 
+                                 d.t1_second.vwap();
+   // round to closest value
+   d.t1_front.estimated_settle = round_to_nearest(
+                        d.t1_front.estimated_settle,me->data.tick_size);
+   d.t1_second.estimated_settle = round_to_nearest(
+                        d.t1_second.estimated_settle,me->data.tick_size);
+   d.t1_third.estimated_settle = round_to_nearest(
+                        d.t1_third.estimated_settle,me->data.tick_size);
+}
+
+// the decision to send an order can be caused by
+// 1. change in settings (=> time window, enabling orders etc)
+// 2. book update (=> TAS book can result in order)
+// 3. ticker (=> new estimated settlement)
+// 4. time passing (=> in the window now)
+void TASCounterAlgo::_check_and_send_order(){
+   _compute_settles();
+   auto should_send_tas_order = 
+                     [](const ContractMarketData& c,
+                         const ContractAutoOrder& o,
+                         const ContractMarketData& tas_c,
+                         bool is_buy,
+                         double& out_price,
+                         uint32_t& out_qty) -> bool {
+      // buying the tas at the offer or selling it at the bid                      
+      const auto& tas_opp_top = is_buy ? tas_c.book.ask()[0] : tas_c.book.bid()[0];
+      const auto& c_same_top = is_buy ? c.book.bid()[0] : c.book.ask()[0];
+      if(tas_opp_top.size() > 0 && c_same_top.size() > 0){
+         // delivery price as defined by where the TAS order is traded
+         double tas_delv_price = c.estimated_settle + tas_opp_top.price();
+         // if the delv price for the buy order is <= contract bid or
+         // or the delv price for the sell order is >= contract offer
+         // then we are buying at the bid or selling at the offer 
+         double c_price = c_same_top.price();
+         if((is_buy && tas_delv_price <= c_price) ||
+            (!is_buy && tas_delv_price >= c_price)){
+            out_qty = std::min(o.qty,tas_opp_top.size());
+            out_price = tas_opp_top.price();
+            return true;
+         }
+      }
+      return false;
+   };
+
+   auto _send_order_tn = [&](
+                            const ContractMarketData& c,
+                            ContractAutoOrder& o,
+                            const ContractMarketData& tas_c) -> void {
+      if(o.enabled && o.n_secs > 0 && o.qty > 0 &&
+         !o.tas_order.has_value() &&
+         now_within_time_win(o.algo_start,me->data.settle_stop)){
+            double price;
+            uint32_t qty;
+            if(should_send_tas_order(c,o,tas_c,true,price,qty)){
+               o.tas_order = _algo().launch_child(*o.buy_ord,
+                                                   Price(price),
+                                                   Quantity(qty));
+            }
+            else if(should_send_tas_order(c,o,tas_c,false,price,qty)){
+               o.tas_order = _algo().launch_child(*o.sell_ord,
+                                                   Price(price),
+                                                   Quantity(qty));
+            }
+      }
+   };
+   _send_order_tn(me->data.t1_front,me->data.t1_front_order,me->data.t1_front_tas);
+   _send_order_tn(me->data.t1_second,me->data.t1_second_order,me->data.t1_second_tas);
+   _send_order_tn(me->data.t1_third,me->data.t1_third_order,me->data.t1_third_tas);
+}
+
+void TASCounterAlgo::_report_state(){
+   _compute_settles();
+   _copy_output(me->data);
+   _algo().update_custom_state(me->data.out_state);
+}
+
+bool TASCounterAlgo::_within_settle() const{
+   return now_within_time_win(me->data.settle_start,me->data.settle_stop);
+}
+///////////////////////////////////////////////////////////////////////////////
+// stateless conversion ///////////////////////////////////////////////////////
+
+// stateless helpers
+ChildTemplate* _create_order_template (IBeforeLaunchAlgoServices& serv,
+                                       SecurityKey key,
+                                       api::Side side){
+   OrderDefinition order_def = serv.definition();
+   order_def.set_security_key (key.value());
+   order_def.set_side(side);   
+   order_def.mutable_custom_def()->mutable_limit()->MergeFrom (
+                                       api::orderman::LimitOrderDefinition());
+   return &(serv.make_child_template(order_def));
+}
+// copies output into the dynamic state for publishing
+void _copy_output(AlgoData& d){
+   auto& m = *(d.out_state.mutable_custom_state());
+   // we can safely assume its within int32
+   auto secs_to_settle = static_cast<int>(t_minus_now_usecs(d.settle_start)/1000000);
+   // if its in settle (-ve value), its set to 0
+   secs_to_settle = std::max(secs_to_settle,0);
+   m["Seconds To Settle"].set_as_int32(secs_to_settle);
+   m["VWAP T1-Front"].set_as_double(d.t1_front.vwap());
+   m["VWAP T1-Second"].set_as_double(d.t1_second.vwap());
+   m["VWAP T1-Third"].set_as_double(d.t1_third.vwap());
+   m["VWAP T2-Front"].set_as_double(d.t2_front.vwap());
+   m["VWAP T2-Second"].set_as_double(d.t2_second.vwap());
+   m["VWAP T3-Front"].set_as_double(d.t3_front.vwap());
+   m["T1-Front Projected Settle"].set_as_double(d.t1_front.estimated_settle);
+   m["T1-Second Projected Settle"].set_as_double(d.t1_second.estimated_settle);
+   m["T1-Third Projected Settle"].set_as_double(d.t1_third.estimated_settle);
+}
+// copies the settle time/period part of the input data
+bool _copy_settle_times (AlgoData& d,
+                         const Map<std::string,DynamicOrderParam>& params){
+   auto hh = get_if<uint32_t>(params, "Settlement Start Hour");
+   auto mm = get_if<uint32_t>(params, "Settlement Start Minutes");
+   auto ss = get_if<uint32_t>(params, "Settlement Start Seconds");
+   auto settle_period_secs = get_if<uint32_t>(params, "Settlement Period (seconds)");
+   if (hh.has_value() && mm.has_value() && 
+       ss.has_value() && settle_period_secs.has_value() &&
+       hh.value() <= 23 && mm.value() <= 59 && ss.value() <= 59){
+      d.settle_start = SODPosition::today_local_time_to_utc_timestamp(
+            hh.value(),
+            mm.value(),
+            ss.value(),
+            true,
+            false);
+      d.settle_stop = SODPosition::add_seconds(d.settle_start,
+                                               settle_period_secs.value());
+      auto settle_start_txt = util::TimeUtil::ToString(d.settle_start);
+      auto settle_stop_txt = util::TimeUtil::ToString(d.settle_stop);
+      LOG(OrderMan, Info, "TASCounterAlgo::_copy_settle_times "
+          "settle_start:{} settle_stop:{} ",
+          settle_start_txt,
+          settle_stop_txt);      
+      return true;
+   }
+   return false;
+}
+// copies the order execution part of the input data
+void _copy_order_data (AlgoData& out_d,
+                       const Map<std::string,DynamicOrderParam>& params){   
+   auto copy_o = [&](ContractAutoOrder& out_order,
+                     std::string time_name,
+                     std::string qty_name,
+                     std::string enabled_name
+                     ) -> bool {
+      out_order.enabled = false;
+      auto time = get_if<uint32_t>(params, time_name);
+      auto qty = get_if<uint32_t>(params, qty_name);
+      auto enabled = get_if<bool>(params, enabled_name);
+      if(time.has_value() && qty.has_value() && enabled.has_value()){
+         out_order.n_secs = time.value();
+         out_order.qty = qty.value();
+         out_order.enabled = enabled.value();
+         out_order.algo_start = SODPosition::add_seconds(out_d.settle_stop,
+                                                         -1 * out_order.n_secs);
+         return true;
+      }
+      return false;
+   };
+   copy_o(out_d.t1_front_order,                                 
+          "T1 Front Time",
+          "T1 Front Qty",
+          "T1 Front Order Enabled");
+   copy_o(out_d.t1_second_order,
+          "T1 Second Time",
+          "T1 Second Qty",
+          "T1 Second Order Enabled");
+   copy_o(out_d.t1_third_order,
+          "T1 Third Time",
+          "T1 Third Qty",
+          "T1 Third Order Enabled");
+}
+// copies all of the input params (in the algo) into proper structures
+bool _setup_data(IBeforeLaunchAlgoServices& serv,
+                 AlgoData& out_d,
+                 std::map<SecurityKey,ContractMarketData*>& out_md_map){
+   auto& def = serv.definition();
+   // [ captures ] (parameters) -> returnTypesDeclaration { lambdaStatements; }   
+   auto init_contract_bk = [&out_md_map](auto& out_contract_md, 
+                            const auto& params,
+                            auto param_name) -> void {
+      auto k = get_if<uint32_t>(params, param_name);
+      if(k.has_value()){
+         out_contract_md.book = Book(SecurityKey(k.value()));
+         out_md_map[out_contract_md.book.security_key()] = &out_contract_md;   
+      }
+   };
+   if (def.has_custom_def() && def.custom_def().has_dynamic()){
+      out_d.tick_size = serv.security().contract_spec().low_tick();
+      const auto& p = def.custom_def().dynamic().order_params();
+      // settlement time
+      if(!_copy_settle_times(out_d,p))
+         return false;
+      // main contracts
+      init_contract_bk(out_d.t1_front, p,"T1 Front");
+      init_contract_bk(out_d.t1_second, p,"T1 Second");
+      init_contract_bk(out_d.t1_third, p,"T1 Third");
+      init_contract_bk(out_d.t2_front, p,"T2 Front");
+      init_contract_bk(out_d.t2_second, p,"T2 Second");
+      init_contract_bk(out_d.t3_front, p,"T3 Front");
+      // TAS contracts
+      init_contract_bk(out_d.t1_front_tas, p,"T1 Front TAS");
+      init_contract_bk(out_d.t1_second_tas, p,"T1 Second TAS");
+      init_contract_bk(out_d.t1_third_tas, p,"T1 Third TAS");
+      // auto orders
+      _copy_order_data(out_d,p);      
+      // order template
+      out_d.t1_front_order.buy_ord = _create_order_template(serv,
+                        out_d.t1_front_tas.book.security_key(),api::Side::SIDE_BUY);
+      out_d.t1_front_order.sell_ord = _create_order_template(serv,
+                        out_d.t1_front_tas.book.security_key(),api::Side::SIDE_SELL);
+      out_d.t1_second_order.buy_ord = _create_order_template(serv,
+                        out_d.t1_second_tas.book.security_key(),api::Side::SIDE_BUY);
+      out_d.t1_second_order.sell_ord = _create_order_template(serv,
+                        out_d.t1_second_tas.book.security_key(),api::Side::SIDE_SELL);
+      out_d.t1_third_order.buy_ord = _create_order_template(serv,
+                        out_d.t1_third_tas.book.security_key(),api::Side::SIDE_BUY);
+      out_d.t1_third_order.sell_ord = _create_order_template(serv,
+                        out_d.t1_third_tas.book.security_key(),api::Side::SIDE_SELL);
+      return true;
+   }
+   return false;
+}
+double round_to_nearest(double price, double tick_size){
+   if(os::is_nan(price))
+      return price;
+   auto buy_price = bts::securities::raw_round_to_quote_price(
+                                                         price,
+                                                         tick_size,
+                                                         api::Side::SIDE_BUY);
+   auto sell_price = buy_price + tick_size;
+   return std::abs(price-buy_price) < std::abs(sell_price-price) ? buy_price :
+                                                                   sell_price;
+}
+// time helpers
+// converts std::chrono::system_clock::time_point to google timestamp
+google::protobuf::Timestamp to_pf_ts (const bts::TimeStamp& ts){   
+   auto ts_us = std::chrono::time_point_cast<std::chrono::microseconds>(ts);
+   auto ts_us_val = ts_us.time_since_epoch();
+   auto usecs = ts_us_val.count();
+   return google::protobuf::util::TimeUtil::MicrosecondsToTimestamp(usecs);
+}
+long long t1_minus_t2_usecs (const Timestamp& t1,const Timestamp& t2){
+   return util::TimeUtil::TimestampToMicroseconds(t1) -
+          util::TimeUtil::TimestampToMicroseconds(t2);
+}
+long long t_minus_now_usecs (const Timestamp& t){
+   auto now = util::TimeUtil::GetCurrentTime();
+   return t1_minus_t2_usecs(t,now);
+}
+bool within_time_win (const Timestamp& beg, const Timestamp& end,
+                      const Timestamp& ts){
+   return t1_minus_t2_usecs(ts,beg)>=0 &&
+          t1_minus_t2_usecs(ts,end)<=0;
+}
+// if current time is within time window
+bool now_within_time_win (const Timestamp& beg,
+                      const Timestamp& end){
+   return within_time_win(beg,end,util::TimeUtil::GetCurrentTime());
+}
+bool trade_within_time_win (const Timestamp& beg, const Timestamp& end,
+                            const TickerUpdateEvent::Trade& t){
+   auto ts = to_pf_ts(t.ts);
+   return within_time_win(beg,end,ts);
+}
+
+} // namespace orderman
+
+} // namespace bts

src/lib/orderman/orders/algo/algo_orders/TASCounterAlgo.h

@@ -0,0 +1,41 @@
+#pragma once
+
+#include "lib/orderman/orders/algo/IAlgoOrder.h"
+
+namespace bts {
+namespace orderman {
+
+class ChildTemplate;
+
+class TASCounterAlgo : public IAlgoOrder {
+public:
+   TASCounterAlgo();
+   ~TASCounterAlgo();
+   struct TASCounterAlgoParams{
+   };
+public:
+   OrderOpResult do_before_launch(IBeforeLaunchAlgoServices &) override;
+   void do_launch(IAlgoServices &) override;
+   void do_launch_attached(IAlgoServices &, OrderKey, ChildState) override;
+   void do_update(api::orderman::Update const &) override;
+   void on_book(SecurityKey, Book const &) override;
+   void on_ticker(SecurityKey, TradeVector const&) override;
+   IAlgoOrder::OnCancelResponse on_cancel() override;
+   void on_timer() override;
+private:
+   enum class Status;
+
+   struct PImpl;
+   std::unique_ptr<PImpl> me;
+private:
+   IAlgoServices& _algo() const ;
+   void _switch_state (Status new_status);
+   void _cxl_and_end (bool is_error, const std::string& err_msg);
+   void _check_and_send_order();
+   void _compute_settles ();
+   void _report_state();
+   bool _within_settle() const;
+};
+
+} // namespace orderman
+} // namespace bts

src/lib/orderman/orders/algo/algo_orders/TopAlgo.h

@@ -19,7 +19,7 @@ class TopAlgo : public IAlgoOrder {
    void do_update(api::orderman::Update const &) override;
    void on_book(SecurityKey, Book const &) override; 
    void on_child_change(OrderKey, ChildState const &) override;
-
+   
  private:
    struct PImpl;
    std::unique_ptr<PImpl> me;

src/lib/orderman/orders/algo/algo_orders/tests/AlgoTestDriver.cpp

@@ -56,16 +56,19 @@ struct AlgoTestDriverBase::Impl : public RecursionCounter {
    api::Security security;
    OrderKey order_key, parent_key;
    std::tuple<Quantity, Price> fill_state;
-
+   std::vector<PositionUpdate> positions;
    AlgoState state;
    std::vector<ChildOrder> orders;
    std::vector<ChildOrder> detached_orders;
 
    std::map<SecurityKey, Book> books;
+   std::map<SecurityKey, Price> settlements;
    std::map<SecurityKey, bool> book_subs;
+   std::map<SecurityKey, bool> settlement_subs;
    std::map<SecurityKey, bool> ticker_subs;
    std::map<SecurityKey, bool> status_subs;
    bool sub_timer = false;
+   bool sub_position = false;
 
    // Callbacks
    std::vector<std::function<void(OrderKey)>> on_child_launching;
@@ -158,6 +161,15 @@ void AlgoTestDriverBase::update_book(const Book& book) {
       me->algo.on_book(key, book);
    }
 }
+
+void AlgoTestDriverBase::update_settlement(SecurityKey key, Price price) {
+   me->settlements[key] = price;
+   auto iter = me->settlement_subs.find(key);
+   if (iter != me->settlement_subs.end()) {
+      me->algo.on_settlement(key, price);
+   }
+}
+
 void AlgoTestDriverBase::update_ticker(SecurityKey key, const TradeVector& data) {
    auto iter = me->ticker_subs.find(key);
    if (iter != me->ticker_subs.end() && iter->second) {
@@ -301,6 +313,15 @@ const api::Security* AlgoTestDriverBase::lookup_security(SecurityKey key) const
    }
    return nullptr;
 }
+std::vector<api::Security const *> AlgoTestDriverBase::known_securities() const {
+   std::vector<api::Security const *> result;
+   auto s = get_some_securities();
+   for (auto const &p : s) {
+      result.push_back(&p.second);
+   }
+   return result;
+}
+
 bool AlgoTestDriverBase::is_attaching() const { return has_flag(me->flags, ATDFlags::IS_ATTACHING); }
 ChildTemplate& AlgoTestDriverBase::make_child_template(const api::orderman::OrderDefinition& def, size_t) {
    REQUIRE(def.custom_def().custom_case() != CustomOrderDefinition::CUSTOM_NOT_SET);
@@ -321,6 +342,10 @@ bool AlgoTestDriverBase::is_shutting_down() const {
    return false;
 }
 
+std::vector<PositionUpdate> const & AlgoTestDriverBase::current_positions() const {
+   return me->positions;
+}
+
 void AlgoTestDriverBase::subscribe_book(SecurityKey key){
    me->book_subs[key] = true;
    auto iter = me->books.find(key);
@@ -332,6 +357,18 @@ void AlgoTestDriverBase::unsubscribe_book(SecurityKey key) {
    me->book_subs.erase(key);
 }
 
+void AlgoTestDriverBase::subscribe_settlement(SecurityKey key){
+   me->settlement_subs[key] = true;
+   auto iter = me->settlements.find(key);
+   if (iter != me->settlements.end()) {
+      me->algo.on_settlement(key, iter->second);
+   }
+}
+
+void AlgoTestDriverBase::unsubscribe_settlement(SecurityKey key) {
+   me->settlement_subs.erase(key);
+}
+
 void AlgoTestDriverBase::subscribe_ticker(SecurityKey key)  {
    me->ticker_subs[key] = true;
 }
@@ -363,6 +400,8 @@ void AlgoTestDriverBase::unsubscribe_child_change(OrderKey k) {
 void AlgoTestDriverBase::subscribe_timer() { me->sub_timer = true; }
 void AlgoTestDriverBase::unsubscribe_timer() { me->sub_timer = false; }
 
+void AlgoTestDriverBase::subscribe_position() { me->sub_position= true; }
+void AlgoTestDriverBase::unsubscribe_position() { me->sub_position = false; }
 
 OrderKey AlgoTestDriverBase::launch_child(ChildTemplate&, Price price, Quantity qty, bool sub) {
    OrderKey key = me->next_order();
@@ -451,5 +490,6 @@ void AlgoTestDriverBase::complete() {
 }
 void AlgoTestDriverBase::set_ladder_visibility(bool) { /* whatever, ok */ }
 void AlgoTestDriverBase::update_fill_state(Quantity q, Price p) { me->fill_state = { q, p }; }
+void AlgoTestDriverBase::update_custom_state(api::orderman::DynamicOrderState const&) {}
 
-}}}//namespace
+}}}//namespace

src/lib/orderman/orders/algo/algo_orders/tests/AlgoTestDriver.h

@@ -60,7 +60,7 @@ class AlgoTestDriverBase : private IAlgoServices {
    Quantity quantity() const override;
    Quantity total_fills() const override;
    bool is_shutting_down() const override;
-   
+   std::vector<PositionUpdate> const & current_positions() const override;
    //
    // Change state
 
@@ -70,6 +70,7 @@ class AlgoTestDriverBase : private IAlgoServices {
    void do_update(const api::orderman::Update&);
 
    void update_book(const Book&);
+   void update_settlement(SecurityKey , Price ) ;
    void update_ticker(SecurityKey, const TradeVector&);
    void update_status(SecurityKey, co2::TradingStatus);
    void advance_time(Duration, uint32_t steps=1);
@@ -104,6 +105,7 @@ private:
    OrderKey order_key() const override;
    OrderKey parent_order_key() const override;
    const api::Security* lookup_security(SecurityKey) const override;
+   std::vector<api::Security const *> known_securities() const override;
    bool is_attaching() const override;
    ChildTemplate& make_child_template(const api::orderman::OrderDefinition& child_def, size_t reserve = 1) override;
    TimeStamp get_time() const override;
@@ -112,6 +114,8 @@ private:
    // IAlgoServices
    void subscribe_book(SecurityKey)override;
    void unsubscribe_book(SecurityKey) override;
+   void subscribe_settlement(SecurityKey)override;
+   void unsubscribe_settlement(SecurityKey) override;
    void subscribe_child_change(OrderKey) override;
    void unsubscribe_child_change(OrderKey) override;
    void subscribe_timer() override;
@@ -120,6 +124,8 @@ private:
    void unsubscribe_ticker(SecurityKey) override;
    void subscribe_status(SecurityKey) override;
    void unsubscribe_status(SecurityKey) override;
+   void subscribe_position() override;
+   void unsubscribe_position() override;
 
    OrderKey launch_child(ChildTemplate &, Price, Quantity, bool) override;
    OrderKey launch_child(api::orderman::OrderDefinition const &, Price,
@@ -134,7 +140,7 @@ private:
    void complete() override;
    void set_ladder_visibility(bool) override;
    void update_fill_state(Quantity , Price) override;
-
+   void update_custom_state(api::orderman::DynamicOrderState const&) override;
 };
 
 template <class Algo>
@@ -151,4 +157,4 @@ struct AlgoTestDriver : public AlgoTestDriverBase
    const Algo& algo() const { return _algo; }
 };
 
-}}}//namespace
+}}}//namespace

src/lib/orderman/securities/ISecurityDB.h

@@ -13,6 +13,7 @@ class ISecurityDB
 public:
    virtual ~ISecurityDB() {};
    virtual api::Security const *resolve(SecurityKey const&k) const = 0;
+   virtual std::vector<api::Security const*> get_securities() const = 0;
    using SecurityMap = std::unordered_map<SecurityKey,api::Security const*>  ;
 };
 

src/lib/orderman/securities/MapSecurityDB.h

@@ -13,6 +13,11 @@ struct MapSecurityDB : public securities::ISecurityDB
    api::Security const *resolve(SecurityKey const&k) const override {
       return _oracle.get_security(k);
    }
+
+   std::vector<api::Security const*> get_securities() const override {
+      return _oracle.get_securities();
+   }
+
    private:
    md::SecurityOracle _oracle;
 };

src/sdk/examples/client_sdk/.vscode/launch.json

@@ -8,7 +8,7 @@
             "name": "(gdb) Launch",
             "type": "cppdbg",
             "request": "launch",
-            "program": "/sparkbin/sparkd",
+            "program": "${workspaceFolder}/client_sdk_example",
             "args": ["run"],
             "stopAtEntry": false,
             "cwd": "/sparkbin",

src/sdk/examples/client_sdk/client_sdk_example.cpp

@@ -2,6 +2,7 @@
 #include "spark/lib/domain_model/Security.h"
 #include "lib/api/services/AdminService.client.pb.h"
 #include "lib/api/services/PositionService.client.pb.h"
+#include "lib/api/services/MarketDataService.client.pb.h"
 
 #include <string>
 #include <vector>
@@ -30,6 +31,17 @@ int main(int argc, char** argv) {
       }
    });
 
+
+   auto md_service = ctx.locate<bts::api::MarketDataService>();
+   auto sub2 = 
+      md_service
+      .subscribe_securities()
+      .subscribe([](const auto& update) {
+      for (const auto& s : update.added()) {
+         std::cout << "Added security: " << s->id() << ": " << s->symbol() << std::endl;
+      }
+   });
+
    std::this_thread::sleep_for(std::chrono::minutes(5));
    return 0;
 }

src/sdk/examples/dynamic_join_order/order_plugin_example.cpp

@@ -36,6 +36,8 @@ class DynamicJoin : public IAlgoOrder {
    struct PImpl;
    std::unique_ptr<PImpl> me;
 
+   std::string status_name (DynamicJoin::Status);
+   void transition_status(DynamicJoin::Status new_status);
    IAlgoServices &algo_services() const;
    void try_to_join();
 };
@@ -60,6 +62,28 @@ struct DynamicJoin::PImpl {
    Book last_book;
 };
 
+std::string DynamicJoin::status_name (DynamicJoin::Status st) {
+   switch (st) {
+   case DynamicJoin::Status::WAITING_FOR_MD: return "WAITING_FOR_MD";
+   case DynamicJoin::Status::WAITING_TO_JOIN: return "WAITING_TO_JOIN";
+   case DynamicJoin::Status::JOINED: return "JOINED";
+   default:
+      bts_assert(false);
+      return "";
+   }
+}
+void DynamicJoin::transition_status(Status new_status) {
+   me->status = new_status;
+   // Every time we change our internal state we report
+   // a description of the new state so that the user
+   // can monitor the algo progression in the UI
+   api::orderman::DynamicOrderState report;
+   api::orderman::DynamicOrderParam status;
+   status.set_as_string(status_name(new_status));
+   (*report.mutable_custom_state())[std::string("Status:")] = status;
+   me->algo_services->update_custom_state(report);
+}
+   
 DynamicJoin::DynamicJoin() : me(new PImpl()) {}
 DynamicJoin::~DynamicJoin() {}
 IAlgoServices &DynamicJoin::algo_services() const { return *me->algo_services; }
@@ -226,7 +250,7 @@ void DynamicJoin::on_book(SecurityKey, Book const &b) {
       if (other_side_top.size() > 0) {
          // First time we get the opposite side market, we will use
          // this price as the target price for the join
-         me->status = Status::WAITING_TO_JOIN;
+         transition_status(Status::WAITING_TO_JOIN);
          me->book_at_inception = b;
          LOG(OrderMan, Diag,
              "JoinOrder order {} receiving first book: {} Target: {}",
@@ -266,7 +290,7 @@ void DynamicJoin::try_to_join() {
       child_update.set_price(my_side_psz.price());
       child_update.set_quantity(algo_services().quantity().value());
       algo_services().update_child(me->child_key, child_update);
-      me->status = Status::JOINED;
+      transition_status(Status::JOINED);
 
       LOG(OrderMan, Diag, "DynamicJoin order {} joined: {}@{}",
           algo_services().order_key().value(),

src/sdk/sdk_spark.conf

@@ -23,6 +23,8 @@ Ports {
    value {
       config_port: 9000
       qbus_port: 9001
+      glbx_sim_fix_port: 9002
+      ice_sim_fix_port: 9003
    }
 }
 

src/server/sparkd/md/platforms/ice_onixs/ICEListener.cpp

@@ -1,421 +1,451 @@
-#include "ICEListener.h"
-#include "lib/utils/datetime.h"
-#include <vector>
-#include "lib/utils/helper_macros.h"
-#include "lib/config/protos/MarketData.pb.h"
-#include "lib/md/protos/co2.pb.h"
-
-using namespace std;
-using namespace bts::co2;
-using namespace bts::md::ice;
-
-namespace bts {
-namespace md {
-namespace ice {
-   
-template <>
-double double_of_price<onixs_ice::Rational>(const onixs_ice::Rational& price, int8_t denominator) 
-{
-   UNUSED(denominator);
-   double dbl = price.convertTo<double>();
-   return dbl;
-}   
-
-template<>
-void ICEListener::set_security_specific<onixs_ice::FuturesProductDefinition>(const onixs_ice::FuturesProductDefinition& def, 
-                                                                             FixedSecurity& fsec)
-{
-   //fsec.cash_per_point = double_of_price(def.tickValue, fsec.ice_price_denominator);
-   //fsec.cash_per_point = def.tickValue;   
-   fsec.cash_per_point = def.unitQuantity;
-}
-
-ICEListener::ICEListener(ICESecurityIndex& index, MDProducer& producer, const std::vector<MDProducerChannel> channels) :
-   _index(index),
-   _producer(producer),
-   _symbol_filter(),
-   _channels(channels),
-   _concurrency_monitor(0)
-{
-}
-
-ICEListener::~ICEListener() {}
-
-std::string ICEListener::parse_contract_symbol(const std::string& sym)
-{
-   auto end = sym.find(' ');
-   if(end == std::string::npos)
-      return sym;
-   else
-      return sym.substr(0, end);
-}
-
-//   
-//State change
-void ICEListener::onHandlerStateChange(const onixs_ice::HandlerStateChange&) 
-{
-}
-   
-void ICEListener::onMarketStateChange(const onixs_ice::MarketStateChange& , const onixs_ice::MessageInfo&)
-{
-}
-
-//
-// Warnings/Errors
-void ICEListener::onError(const onixs_ice::Error& error) 
-{
-   std::string error_desc(error.description());
-   LOG(md,Warn, "OnixS ICE Error: {} - {}", error.toString(), error_desc );
-}
-void ICEListener::onFeedError(onixs_ice::FeedId feedid, onixs_ice::ErrorCode code, const std::string& desc)
-{
-   LOG(md,Warn, "OnixS ICE FeedError: {} - {}", feedid, desc, code);   
-}
-void ICEListener::onWarning(const onixs_ice::Warning& warning) 
-{
-   LOG(md,Warn, "OnixS ICE Warn: {} - {}", warning.toString(), std::string(warning.description()) );
-}
-
-//
-// Security discovery
-void ICEListener::onFuturesProductDefinition(const onixs_ice::FuturesProductDefinition& msg, bool& is_interested) 
-{
-   if(msg.isSpread)
-      on_future_strip(msg, is_interested);
-   else
-      on_future(msg, is_interested);
-}
-
-void ICEListener::onNewExpiry(const onixs_ice::NewExpiry& msg, bool& is_interested, const onixs_ice::MessageInfo& ) 
-{   
-   on_future(msg, is_interested);
-}
-
-void ICEListener::onFuturesStrategyDefinition(const onixs_ice::FuturesStrategyDefinition& def, bool& is_interested) 
-{
-    on_uds_future_spread(def, is_interested);
-}
-//This is only for UDS
-void ICEListener::onNewFuturesStrategyDefinition(const onixs_ice::NewFuturesStrategyDefinition& msg, bool& is_interested, const onixs_ice::MessageInfo& )
-{
-   on_uds_future_spread(msg, is_interested);
-}
-   
-void ICEListener::onOptionsProductDefinition(const onixs_ice::OptionsProductDefinition& msg, bool& is_interested) 
-{ 
-   UNUSED(msg);
-   UNUSED(is_interested);
-}
-void ICEListener::onNewOptionsMarketDefinition(const onixs_ice::NewOptionsMarketDefinition& msg, bool& is_interested, const onixs_ice::MessageInfo& msg_info)  
-{ 
-   UNUSED(msg);
-   UNUSED(is_interested);
-   UNUSED(msg_info);
-}
-void ICEListener::onOptionsStrategyDefinition(const onixs_ice::OptionsStrategyDefinition& msg, bool& is_interested) 
-{
-   UNUSED(msg);
-   UNUSED(is_interested);
-}
-void ICEListener::onNewOptionsStrategyDefinition(const onixs_ice::NewOptionsStrategyDefinition& msg, bool& is_interested, const onixs_ice::MessageInfo& msg_info) 
-{
-   UNUSED(msg);
-   UNUSED(is_interested);
-   UNUSED(msg_info);
-}
-
-//
-
-void ICEListener::onMarketSnapshot(const onixs_ice::MarketSnapshot& msg, const onixs_ice::MessageInfo& msg_info) 
-{
-   UNUSED(msg_info);
-   auto* data = _index.get_security(msg.marketId);
-   if (data != nullptr) 
-   {
-      bts_assert(data->id == msg.marketId);
-      auto ts = utils::TimeStamp::now();
-      auto settle_denominator = data->handle.security().ice_settle_denominator;
-      if (msg.settlePriceDateTime > 0) {
-         data->handle.update_settlement( ts,  convert_price(msg.settlementPrice, settle_denominator), date_of_timestamp(msg.settlePriceDateTime) );
-      }
-      
-      LOG(md,Diag,"OnixSICE Got Market Snap : {} - {}", std::to_string(data->id), std::to_string(msg.settlementPrice));
-      
-      auto deal_denominator = data->handle.security().ice_deal_denominator;
-      data->handle.update_opening( ts,  double_of_price(msg.openingPrice, deal_denominator), datetime::today() );
-      data->handle.update_session_high( ts,  double_of_price(msg.high, deal_denominator) );
-      data->handle.update_session_low( ts,  double_of_price(msg.low, deal_denominator) );      
-      
-      data->handle.set_last_trade( ts, pricesize_of_price_size(msg.lastTradePrice, deal_denominator, msg.lastTradeQuantity), 
-                                   ll_of_timestamp(msg.lastTradeDateTime), msg.volume );            
-   }        
-}
-
-void ICEListener::onMarketStatistics(const onixs_ice::MarketStatistics& msg, const onixs_ice::MessageInfo& msg_info)
-{
-   UNUSED(msg_info);
-   auto* data = _index.get_security(msg.marketId);
-   if (data != nullptr) 
-   {
-      bts_assert(data->id == msg.marketId);
-      data->handle.set_total_volume(msg.volume);
-   }   
-}
-
-// Gap Recovery -- This is not necessary if we subscribe to the onOrderBookUpdated callback, they handle the breaks transparently 
-void ICEListener::onMarketSnapshotOrder(const onixs_ice::MarketSnapshotOrder& msg, const onixs_ice::MessageInfo& msg_info) 
-{
-   UNUSED(msg);
-   UNUSED(msg_info);
-}
-
-// Gap Recovery -- This is not necessary if we subscribe to the onOrderBookUpdated callback, they handle the breaks transparently 
-void ICEListener::onMarketSnapshotPriceLevel(const onixs_ice::MarketSnapshotPriceLevel& msg, const onixs_ice::MessageInfo& msg_info) 
-{
-   UNUSED(msg);
-   UNUSED(msg_info);
-}
-
-//
-// Book
-void ICEListener::onOrderBookUpdated(const onixs_ice::OrderBook& onixs_book, const onixs_ice::MessageInfo& msg_info) 
-{
-   //tracepoint (bts_trace,trace_order,(char*) "onBookUpdate", (uint32_t)msg_info.sequenceNumber, (uint16_t) msg_info.sequenceNumber, 0);
-      
-   auto data = _index.get_security(onixs_book.marketId());
-   if (data == nullptr) 
-      return;
-   bts_assert(data->id == onixs_book.marketId());
-   auto denominator = data->handle.security().ice_price_denominator;
-
-   // Overwrite all entries every time
-   size_t bids_size = onixs_book.bids().size();
-   size_t asks_size = onixs_book.offers().size();
-   if (!data->handle.has_depth()) 
-   {
-      bids_size = std::min(bids_size, size_t { 1 });
-      asks_size = std::min(asks_size, size_t { 1 });
-   }
-   std::array<PriceSize,COCONUT_MAX_DEPTH> dbids; dbids.fill(NO_MARKET);
-   std::array<PriceSize,COCONUT_MAX_IMPLIED_DEPTH> ibids; ibids.fill(NO_MARKET);
-   std::array<PriceSize,COCONUT_MAX_DEPTH> dasks; dasks.fill(NO_MARKET);
-   std::array<PriceSize,COCONUT_MAX_IMPLIED_DEPTH> iasks;  iasks.fill(NO_MARKET);
-   for (size_t x = 0; x < bids_size; ++x) 
-   {
-      auto& bid = onixs_book.bids()[x];
-      if(x < COCONUT_MAX_DEPTH)
-         dbids[x] = pricesize_of_price_size(bid.price, denominator, bid.quantity - bid.impliedQuantity);   
-      if(x < COCONUT_MAX_IMPLIED_DEPTH)
-         ibids[x] = pricesize_of_price_size(bid.price, denominator, bid.impliedQuantity);   
-   }      
-   for (size_t x = 0; x < asks_size; ++x) 
-   {
-      auto& ask = onixs_book.offers()[x];
-      if(x < COCONUT_MAX_DEPTH)
-         dasks[x] = pricesize_of_price_size(ask.price, denominator, ask.quantity - ask.impliedQuantity);
-      if(x < COCONUT_MAX_IMPLIED_DEPTH)
-         iasks[x] = pricesize_of_price_size(ask.price, denominator, ask.impliedQuantity);
-   } 
-   data->handle.update_books(TimeStamp::now(), dbids, dasks, ibids, iasks, (uint32_t) msg_info.sequenceNumber);
-} 
-
-
-//
-// Trades
-AggressorSide map_aggressor(onixs_ice::AggressorSide::Enum agside) 
-{
-   switch(agside) {
-      default:
-      case onixs_ice::AggressorSide::NoAggressor:
-         return AGGRESSOR_SIDE_NONE;
-      case onixs_ice::AggressorSide::Buy:
-         return AGGRESSOR_SIDE_BUY;
-      case onixs_ice::AggressorSide::Sell:
-         return AGGRESSOR_SIDE_SELL;
-   }
-}
-
-void on_trade(ICESecurityData* data, const onixs_ice::Price& price, int size, 
-              onixs_ice::AggressorSide::Enum agside, const onixs_ice::DateTime& onixs_date, bool from_spread) 
-{
-   auto ts = utils::TimeStamp::now();
-   PriceSize ps = pricesize_of_price_size(price, data->handle.security().ice_deal_denominator, size);
-   auto date = ll_of_timestamp(onixs_date);
-   data->handle.add_trade(ts, ps, date, map_aggressor(agside), 0u, from_spread);
-}
-
-void ICEListener::onTrade(const onixs_ice::Trade& msg, const onixs_ice::MessageInfo& ) 
-{
-   auto* data = _index.get_security(msg.marketId);
-   if (data != NULL) 
-   {
-      bts_assert(data->id == msg.marketId);
-      auto size = msg.quantity;
-      auto price = msg.price;
-      onixs_ice::AggressorSide::Enum agside = msg.aggressorSide;
-      on_trade(data, price, size, agside, msg.transactDateTime, msg.isSystemPricedLeg);
-   }   
-}
-
-//
-// RFQ
-void ICEListener::onRfq(const onixs_ice::Rfq& msg, const onixs_ice::MessageInfo& ) 
-{
-   auto* data = _index.get_security(msg.marketId);
-   if (data != nullptr) 
-   {
-      bts_assert(data->id == msg.marketId);
-      auto ts = utils::TimeStamp::now();
-      co2::QuoteRequestMessage message;
-      message.set_key(data->handle.key());
-
-      auto time = ll_of_timestamp(msg.messageTimestamp);
-      message.set_time(time);
-
-      long long rfqid = msg.rfqSystemId;
-      message.set_request_id(std::to_string(rfqid));
-
-      //https://ref.onixs.biz/cpp-ice-impact-handler-guide/struct_onix_s_1_1_i_c_e_1_1i_mpact_1_1_market_data_1_1_side.html#a8150b7776c2a1749101acf22e868d091
-      switch(msg.side) 
-      {
-         case ' ':
-            message.set_side(co2::QuoteRequestMessage::UNDEFINED);
-            break;
-         case '1':
-            message.set_side(co2::QuoteRequestMessage::BUY);
-            break;
-         case '2':
-            message.set_side(co2::QuoteRequestMessage::SELL);
-            break;
-      }
-      message.set_size(msg.quantity);
-      data->handle.add_rfq(ts, message);
-   }   
-}
-
-//
-// Misc
-void ICEListener::onSettlementPrice(const onixs_ice::SettlementPrice& msg, const onixs_ice::MessageInfo& ) 
-{   
-   auto* data = _index.get_security(msg.marketId);
-   if (data != nullptr) 
-   {
-      bts_assert(data->id == msg.marketId);
-      auto ts = utils::TimeStamp::now();
-      auto price = msg.settlementPrice;
-      auto time = msg.dateTime;
-      auto denominator = data->handle.security().ice_settle_denominator;
-      LOG(md,Diag,"OnixSICE Got Settlement: {} - {}", std::to_string(data->id), std::to_string(price));
-      data->handle.update_settlement(ts, convert_price(price, denominator), date_of_timestamp(time));
-   }   
-}
-
-void ICEListener::onOpenInterest(const onixs_ice::OpenInterest &msg, const onixs_ice::MessageInfo &msg_info) 
-{
-   UNUSED(msg);
-   UNUSED(msg_info);
-}
-
-void ICEListener::onOpenPrice(const onixs_ice::OpenPrice& msg, const onixs_ice::MessageInfo& ) 
-{
-   auto* data = _index.get_security(msg.marketId);
-   if (data != nullptr) 
-   {
-      bts_assert(data->id == msg.marketId);
-      onixs_ice::Price price = msg.openPrice;
-      onixs_ice::DateTime time = msg.dateTime;
-      auto denominator = data->handle.security().ice_deal_denominator;
-      data->handle.update_opening( utils::TimeStamp::now(),  double_of_price(price, denominator), date_of_timestamp(time));
-   }   
-}
-
-void ICEListener::onClosePrice(const onixs_ice::ClosePrice& msg, const onixs_ice::MessageInfo& ) 
-{
-   auto* data = _index.get_security(msg.marketId);
-   if (data != nullptr) 
-   {
-      bts_assert(data->id == msg.marketId);
-      onixs_ice::Price price = msg.closePrice;
-      onixs_ice::DateTime time = msg.dateTime;
-      auto denominator = data->handle.security().ice_deal_denominator;
-      data->handle.update_closing( utils::TimeStamp::now(),  double_of_price(price, denominator), date_of_timestamp(time));
-   }     
-}
-
-void ICEListener::onEndOfDayMarketSummary(const onixs_ice::EndOfDayMarketSummary& msg, const onixs_ice::MessageInfo& ) 
-{
-   auto* data = _index.get_security(msg.marketId);
-   if (data != nullptr) 
-   {
-      bts_assert(data->id == msg.marketId);
-      onixs_ice::DateTime time = msg.dateTime;
-      auto ts = utils::TimeStamp::now();
-      auto settle_denominator = data->handle.security().ice_settle_denominator;
-      data->handle.update_settlement( ts,  convert_price(msg.settlementPrice, settle_denominator), date_of_timestamp(time) );
-      
-      LOG(md,Warn,"OnixSICE Got EOD Market : {} - {}", std::to_string(data->id), std::to_string(msg.settlementPrice));
-      
-      auto deal_denominator = data->handle.security().ice_deal_denominator;
-      data->handle.update_opening( ts,  double_of_price(msg.openingPrice, deal_denominator), date_of_timestamp(time) );
-      data->handle.update_session_high( ts,  double_of_price(msg.high, deal_denominator) );
-      data->handle.update_session_low( ts,  double_of_price(msg.low, deal_denominator) );      
-   }        
-}
-
-//
-// Feed
-void ICEListener::onMulticastMessageBlockBegin(onixs_ice::FeedId feedId, const onixs_ice::MessageInfo& ) 
-{
-   UNUSED(feedId);
-}
-
-void ICEListener::onMulticastMessageBlockEnd(onixs_ice::FeedId feedId, const onixs_ice::MarketIds& marketIds) 
-{
-   UNUSED(feedId);
-   UNUSED(marketIds);
-}
-
-void ICEListener::onFeedStarted(onixs_ice::FeedId, const onixs_ice::MarketIds&) 
-{}
-
-void ICEListener::onFeedStopped(onixs_ice::FeedId) 
-{}
-
-void ICEListener::onSessionNumberChange(onixs_ice::FeedId , short , short )
-{}
-
-
-//
-// Non-onixs calls
-void ICEListener::add_filter_symbol(const std::string& symbol) 
-{
-   _symbol_filter.insert(symbol);
-}
-
-void ICEListener::heartbeat() 
-{
-   for (auto& c : _channels) { c.heartbeat(); }
-}
-
-//
-// Private functions
-void ICEListener::add_security(const FixedSecurity& security) 
-{
-   // Set up the shared memory for this security
-   auto handle = _producer.add_security(security);
-   bts_assert(security.numeric_id != 0);
-   // Add it to the index so it can receive messages
-   _index.add_security(security.numeric_id, handle);
-}
-
-void ICEListener::update_all_status(CoconutSecurityStatus status) 
-{
-   _index.walk_security_list([=](ICESecurityData& data) 
-   {
-      data.handle.set_status(status);
-   });
-}
-
-}}} //namespace
-
-
+#include "ICEListener.h"
+#include "lib/utils/datetime.h"
+#include <vector>
+#include "lib/utils/helper_macros.h"
+#include "lib/config/protos/MarketData.pb.h"
+#include "lib/md/protos/co2.pb.h"
+
+using namespace std;
+using namespace bts::co2;
+using namespace bts::md::ice;
+
+namespace bts {
+namespace md {
+namespace ice {
+   
+template <>
+double double_of_price<onixs_ice::Rational>(const onixs_ice::Rational& price, int8_t denominator) 
+{
+   UNUSED(denominator);
+   double dbl = price.convertTo<double>();
+   return dbl;
+}   
+
+template<>
+void ICEListener::set_security_specific<onixs_ice::FuturesProductDefinition>(const onixs_ice::FuturesProductDefinition& def, 
+                                                                             FixedSecurity& fsec)
+{
+   //fsec.cash_per_point = double_of_price(def.tickValue, fsec.ice_price_denominator);
+   //fsec.cash_per_point = def.tickValue;   
+   fsec.cash_per_point = def.unitQuantity;
+}
+
+ICEListener::ICEListener(ICESecurityIndex& index, MDProducer& producer, const std::vector<MDProducerChannel> channels) :
+   _index(index),
+   _producer(producer),
+   _symbol_filter(),
+   _channels(channels),
+   _concurrency_monitor(0)
+{
+}
+
+ICEListener::~ICEListener() {}
+
+std::string ICEListener::parse_contract_symbol(const std::string& sym)
+{
+   auto end = sym.find(' ');
+   if(end == std::string::npos)
+      return sym;
+   else
+      return sym.substr(0, end);
+}
+
+//   
+//State change
+void ICEListener::onHandlerStateChange(const onixs_ice::HandlerStateChange&) 
+{
+}
+   
+void ICEListener::onMarketStateChange(const onixs_ice::MarketStateChange& , const onixs_ice::MessageInfo&)
+{
+}
+
+//
+// Warnings/Errors
+void ICEListener::onError(const onixs_ice::Error& error) 
+{
+   std::string error_desc(error.description());
+   LOG(md,Warn, "OnixS ICE Error: {} - {}", error.toString(), error_desc );
+}
+void ICEListener::onFeedError(onixs_ice::FeedId feedid, onixs_ice::ErrorCode code, const std::string& desc)
+{
+   LOG(md,Warn, "OnixS ICE FeedError: {} - {}", feedid, desc, code);   
+}
+void ICEListener::onWarning(const onixs_ice::Warning& warning) 
+{
+   LOG(md,Warn, "OnixS ICE Warn: {} - {}", warning.toString(), std::string(warning.description()) );
+}
+
+//
+// Security discovery
+void ICEListener::onFuturesProductDefinition(const onixs_ice::FuturesProductDefinition& msg, bool& is_interested) 
+{
+   if(msg.isSpread)
+      on_future_strip(msg, is_interested);
+   else
+      on_future(msg, is_interested);
+   LOG(md,Info,"ICE_Future,{},{},{},{},{},{},{},{},{},{},{},{},{}",
+       is_interested,
+       msg.productName,       // NG Swing GDD Futures,
+       msg.hubAlias,          // Henry       
+       msg.marketDesc,        // NG Swing GDD Futures - Henry - 10 Jul 20,
+       msg.stripName,         // 10 Jul 20
+       msg.contractSymbol,    // CGS SDN1020.N1020,
+       msg.productId,
+       msg.marketId,
+       msg.requestMarketType,
+       msg.securitySubType,
+       msg.contractSymbolExtra,       
+       msg.clearedAlias,
+       msg.isSpread);
+}
+
+void ICEListener::onNewExpiry(const onixs_ice::NewExpiry& msg, bool& is_interested, const onixs_ice::MessageInfo& ) 
+{   
+   on_future(msg, is_interested);
+}
+
+void ICEListener::onFuturesStrategyDefinition(const onixs_ice::FuturesStrategyDefinition& msg, bool& is_interested) 
+{
+   on_uds_future_spread(msg, is_interested);
+   LOG(md,Info,"ICE_Strategy,{},{},{},{},{},{},{},{},{},{},{},{},{},{}",
+       is_interested,
+       msg.productName,
+       msg.hubAlias,
+       msg.marketDesc,
+       msg.stripName,
+       msg.contractSymbol,
+       msg.productId,
+       msg.marketId,
+       msg.requestMarketType,
+       msg.securitySubType,
+       log::CharPtrWrapper {""},       
+       msg.clearedAlias,
+       msg.productName,       
+       0);  
+}
+
+void ICEListener::onNewFuturesStrategyDefinition(const onixs_ice::NewFuturesStrategyDefinition& msg, bool& is_interested, const onixs_ice::MessageInfo& )
+{
+   on_uds_future_spread(msg, is_interested);
+}
+   
+void ICEListener::onOptionsProductDefinition(const onixs_ice::OptionsProductDefinition& msg, bool& is_interested) 
+{ 
+   UNUSED(msg);
+   UNUSED(is_interested);
+}
+void ICEListener::onNewOptionsMarketDefinition(const onixs_ice::NewOptionsMarketDefinition& msg, bool& is_interested, const onixs_ice::MessageInfo& msg_info)  
+{ 
+   UNUSED(msg);
+   UNUSED(is_interested);
+   UNUSED(msg_info);
+}
+void ICEListener::onOptionsStrategyDefinition(const onixs_ice::OptionsStrategyDefinition& msg, bool& is_interested) 
+{
+   UNUSED(msg);
+   UNUSED(is_interested);
+}
+void ICEListener::onNewOptionsStrategyDefinition(const onixs_ice::NewOptionsStrategyDefinition& msg, bool& is_interested, const onixs_ice::MessageInfo& msg_info) 
+{
+   UNUSED(msg);
+   UNUSED(is_interested);
+   UNUSED(msg_info);
+}
+
+//
+
+void ICEListener::onMarketSnapshot(const onixs_ice::MarketSnapshot& msg, const onixs_ice::MessageInfo& msg_info) 
+{
+   UNUSED(msg_info);
+   auto* data = _index.get_security(msg.marketId);
+   if (data != nullptr) 
+   {
+      bts_assert(data->id == msg.marketId);
+      auto ts = utils::TimeStamp::now();
+      auto settle_denominator = data->handle.security().ice_settle_denominator;
+      if (msg.settlePriceDateTime > 0) {
+         data->handle.update_settlement( ts,  convert_price(msg.settlementPrice, settle_denominator), date_of_timestamp(msg.settlePriceDateTime) );
+      }
+      
+      LOG(md,Diag,"OnixSICE Got Market Snap : {} - {}", std::to_string(data->id), std::to_string(msg.settlementPrice));
+      
+      auto deal_denominator = data->handle.security().ice_deal_denominator;
+      data->handle.update_opening( ts,  double_of_price(msg.openingPrice, deal_denominator), datetime::today() );
+      data->handle.update_session_high( ts,  double_of_price(msg.high, deal_denominator) );
+      data->handle.update_session_low( ts,  double_of_price(msg.low, deal_denominator) );      
+      
+      data->handle.set_last_trade( ts, pricesize_of_price_size(msg.lastTradePrice, deal_denominator, msg.lastTradeQuantity), 
+                                   ll_of_timestamp(msg.lastTradeDateTime), msg.volume );            
+   }        
+}
+
+void ICEListener::onMarketStatistics(const onixs_ice::MarketStatistics& msg, const onixs_ice::MessageInfo& msg_info)
+{
+   UNUSED(msg_info);
+   auto* data = _index.get_security(msg.marketId);
+   if (data != nullptr) 
+   {
+      bts_assert(data->id == msg.marketId);
+      data->handle.set_total_volume(msg.volume);
+   }   
+}
+
+// Gap Recovery -- This is not necessary if we subscribe to the onOrderBookUpdated callback, they handle the breaks transparently 
+void ICEListener::onMarketSnapshotOrder(const onixs_ice::MarketSnapshotOrder& msg, const onixs_ice::MessageInfo& msg_info) 
+{
+   UNUSED(msg);
+   UNUSED(msg_info);
+}
+
+// Gap Recovery -- This is not necessary if we subscribe to the onOrderBookUpdated callback, they handle the breaks transparently 
+void ICEListener::onMarketSnapshotPriceLevel(const onixs_ice::MarketSnapshotPriceLevel& msg, const onixs_ice::MessageInfo& msg_info) 
+{
+   UNUSED(msg);
+   UNUSED(msg_info);
+}
+
+//
+// Book
+void ICEListener::onOrderBookUpdated(const onixs_ice::OrderBook& onixs_book, const onixs_ice::MessageInfo& msg_info) 
+{
+   //tracepoint (bts_trace,trace_order,(char*) "onBookUpdate", (uint32_t)msg_info.sequenceNumber, (uint16_t) msg_info.sequenceNumber, 0);
+      
+   auto data = _index.get_security(onixs_book.marketId());
+   if (data == nullptr) 
+      return;
+   bts_assert(data->id == onixs_book.marketId());
+   auto denominator = data->handle.security().ice_price_denominator;
+
+   // Overwrite all entries every time
+   size_t bids_size = onixs_book.bids().size();
+   size_t asks_size = onixs_book.offers().size();
+   if (!data->handle.has_depth()) 
+   {
+      bids_size = std::min(bids_size, size_t { 1 });
+      asks_size = std::min(asks_size, size_t { 1 });
+   }
+   std::array<PriceSize,COCONUT_MAX_DEPTH> dbids; dbids.fill(NO_MARKET);
+   std::array<PriceSize,COCONUT_MAX_IMPLIED_DEPTH> ibids; ibids.fill(NO_MARKET);
+   std::array<PriceSize,COCONUT_MAX_DEPTH> dasks; dasks.fill(NO_MARKET);
+   std::array<PriceSize,COCONUT_MAX_IMPLIED_DEPTH> iasks;  iasks.fill(NO_MARKET);
+   for (size_t x = 0; x < bids_size; ++x) 
+   {
+      auto& bid = onixs_book.bids()[x];
+      if(x < COCONUT_MAX_DEPTH)
+         dbids[x] = pricesize_of_price_size(bid.price, denominator, bid.quantity - bid.impliedQuantity);   
+      if(x < COCONUT_MAX_IMPLIED_DEPTH)
+         ibids[x] = pricesize_of_price_size(bid.price, denominator, bid.impliedQuantity);   
+   }      
+   for (size_t x = 0; x < asks_size; ++x) 
+   {
+      auto& ask = onixs_book.offers()[x];
+      if(x < COCONUT_MAX_DEPTH)
+         dasks[x] = pricesize_of_price_size(ask.price, denominator, ask.quantity - ask.impliedQuantity);
+      if(x < COCONUT_MAX_IMPLIED_DEPTH)
+         iasks[x] = pricesize_of_price_size(ask.price, denominator, ask.impliedQuantity);
+   } 
+   data->handle.update_books(TimeStamp::now(), dbids, dasks, ibids, iasks, (uint32_t) msg_info.sequenceNumber);
+} 
+
+
+//
+// Trades
+AggressorSide map_aggressor(onixs_ice::AggressorSide::Enum agside) 
+{
+   switch(agside) {
+      default:
+      case onixs_ice::AggressorSide::NoAggressor:
+         return AGGRESSOR_SIDE_NONE;
+      case onixs_ice::AggressorSide::Buy:
+         return AGGRESSOR_SIDE_BUY;
+      case onixs_ice::AggressorSide::Sell:
+         return AGGRESSOR_SIDE_SELL;
+   }
+}
+
+void on_trade(ICESecurityData* data, const onixs_ice::Price& price, int size, 
+              onixs_ice::AggressorSide::Enum agside, const onixs_ice::DateTime& onixs_date, bool from_spread) 
+{
+   auto ts = utils::TimeStamp::now();
+   PriceSize ps = pricesize_of_price_size(price, data->handle.security().ice_deal_denominator, size);
+   auto date = ll_of_timestamp(onixs_date);
+   data->handle.add_trade(ts, ps, date, map_aggressor(agside), 0u, from_spread);
+}
+
+void ICEListener::onTrade(const onixs_ice::Trade& msg, const onixs_ice::MessageInfo& ) 
+{
+   auto* data = _index.get_security(msg.marketId);
+   if (data != NULL) 
+   {
+      bts_assert(data->id == msg.marketId);
+      auto size = msg.quantity;
+      auto price = msg.price;
+      onixs_ice::AggressorSide::Enum agside = msg.aggressorSide;
+      on_trade(data, price, size, agside, msg.transactDateTime, msg.isSystemPricedLeg);
+   }   
+}
+
+//
+// RFQ
+void ICEListener::onRfq(const onixs_ice::Rfq& msg, const onixs_ice::MessageInfo& ) 
+{
+   auto* data = _index.get_security(msg.marketId);
+   if (data != nullptr) 
+   {
+      bts_assert(data->id == msg.marketId);
+      auto ts = utils::TimeStamp::now();
+      co2::QuoteRequestMessage message;
+      message.set_key(data->handle.key());
+
+      auto time = ll_of_timestamp(msg.messageTimestamp);
+      message.set_time(time);
+
+      long long rfqid = msg.rfqSystemId;
+      message.set_request_id(std::to_string(rfqid));
+
+      //https://ref.onixs.biz/cpp-ice-impact-handler-guide/struct_onix_s_1_1_i_c_e_1_1i_mpact_1_1_market_data_1_1_side.html#a8150b7776c2a1749101acf22e868d091
+      switch(msg.side) 
+      {
+         case ' ':
+            message.set_side(co2::QuoteRequestMessage::UNDEFINED);
+            break;
+         case '1':
+            message.set_side(co2::QuoteRequestMessage::BUY);
+            break;
+         case '2':
+            message.set_side(co2::QuoteRequestMessage::SELL);
+            break;
+      }
+      message.set_size(msg.quantity);
+      data->handle.add_rfq(ts, message);
+   }   
+}
+
+//
+// Misc
+void ICEListener::onSettlementPrice(const onixs_ice::SettlementPrice& msg, const onixs_ice::MessageInfo& ) 
+{   
+   auto* data = _index.get_security(msg.marketId);
+   if (data != nullptr) 
+   {
+      bts_assert(data->id == msg.marketId);
+      auto ts = utils::TimeStamp::now();
+      auto price = msg.settlementPrice;
+      auto time = msg.dateTime;
+      auto denominator = data->handle.security().ice_settle_denominator;
+      LOG(md,Diag,"OnixSICE Got Settlement: {} - {}", std::to_string(data->id), std::to_string(price));
+      data->handle.update_settlement(ts, convert_price(price, denominator), date_of_timestamp(time));
+   }   
+}
+
+void ICEListener::onOpenInterest(const onixs_ice::OpenInterest &msg, const onixs_ice::MessageInfo &msg_info) 
+{
+   UNUSED(msg);
+   UNUSED(msg_info);
+}
+
+void ICEListener::onOpenPrice(const onixs_ice::OpenPrice& msg, const onixs_ice::MessageInfo& ) 
+{
+   auto* data = _index.get_security(msg.marketId);
+   if (data != nullptr) 
+   {
+      bts_assert(data->id == msg.marketId);
+      onixs_ice::Price price = msg.openPrice;
+      onixs_ice::DateTime time = msg.dateTime;
+      auto denominator = data->handle.security().ice_deal_denominator;
+      data->handle.update_opening( utils::TimeStamp::now(),  double_of_price(price, denominator), date_of_timestamp(time));
+   }   
+}
+
+void ICEListener::onClosePrice(const onixs_ice::ClosePrice& msg, const onixs_ice::MessageInfo& ) 
+{
+   auto* data = _index.get_security(msg.marketId);
+   if (data != nullptr) 
+   {
+      bts_assert(data->id == msg.marketId);
+      onixs_ice::Price price = msg.closePrice;
+      onixs_ice::DateTime time = msg.dateTime;
+      auto denominator = data->handle.security().ice_deal_denominator;
+      data->handle.update_closing( utils::TimeStamp::now(),  double_of_price(price, denominator), date_of_timestamp(time));
+   }     
+}
+
+void ICEListener::onEndOfDayMarketSummary(const onixs_ice::EndOfDayMarketSummary& msg, const onixs_ice::MessageInfo& ) 
+{
+   auto* data = _index.get_security(msg.marketId);
+   if (data != nullptr) 
+   {
+      bts_assert(data->id == msg.marketId);
+      onixs_ice::DateTime time = msg.dateTime;
+      auto ts = utils::TimeStamp::now();
+      auto settle_denominator = data->handle.security().ice_settle_denominator;
+      data->handle.update_settlement( ts,  convert_price(msg.settlementPrice, settle_denominator), date_of_timestamp(time) );
+      
+      LOG(md,Warn,"OnixSICE Got EOD Market : {} - {}", std::to_string(data->id), std::to_string(msg.settlementPrice));
+      
+      auto deal_denominator = data->handle.security().ice_deal_denominator;
+      data->handle.update_opening( ts,  double_of_price(msg.openingPrice, deal_denominator), date_of_timestamp(time) );
+      data->handle.update_session_high( ts,  double_of_price(msg.high, deal_denominator) );
+      data->handle.update_session_low( ts,  double_of_price(msg.low, deal_denominator) );      
+   }        
+}
+
+//
+// Feed
+void ICEListener::onMulticastMessageBlockBegin(onixs_ice::FeedId feedId, const onixs_ice::MessageInfo& ) 
+{
+   UNUSED(feedId);
+}
+
+void ICEListener::onMulticastMessageBlockEnd(onixs_ice::FeedId feedId, const onixs_ice::MarketIds& marketIds) 
+{
+   UNUSED(feedId);
+   UNUSED(marketIds);
+}
+
+void ICEListener::onFeedStarted(onixs_ice::FeedId, const onixs_ice::MarketIds&) 
+{}
+
+void ICEListener::onFeedStopped(onixs_ice::FeedId) 
+{}
+
+void ICEListener::onSessionNumberChange(onixs_ice::FeedId , short , short )
+{}
+
+
+//
+// Non-onixs calls
+void ICEListener::add_filter_symbol(const std::string& symbol) 
+{
+   LOG(md, Info, "ICEListener::add_filter_symbol: adding {}", symbol);
+   _symbol_filter.insert(symbol);
+}
+
+void ICEListener::heartbeat() 
+{
+   for (auto& c : _channels) { c.heartbeat(); }
+}
+
+//
+// Private functions
+void ICEListener::add_security(const FixedSecurity& security) 
+{
+   // Set up the shared memory for this security
+   auto handle = _producer.add_security(security);
+   bts_assert(security.numeric_id != 0);
+   // Add it to the index so it can receive messages
+   _index.add_security(security.numeric_id, handle);
+}
+
+void ICEListener::update_all_status(CoconutSecurityStatus status) 
+{
+   _index.walk_security_list([=](ICESecurityData& data) 
+   {
+      data.handle.set_status(status);
+   });
+}
+
+}}} //namespace
+
+

src/server/sparkd/md/platforms/ice_onixs/ICEListener.h

@@ -119,7 +119,20 @@ private:
    enum SecurityFilterAction { ADD, SKIP, DELETE };
    
    std::string parse_contract_symbol(const std::string& sym);
-      
+   template<typename Definition>
+   std::string get_contract_symbol(const Definition& def){
+      const auto& sym = def.contractSymbol;
+      // TODO : this is only exception right now but for HHD sstrips
+      // def.contractSymbol is empty ! 
+      if (sym.size() == 0 && def.marketDesc.rfind("NG Swing GDD Futures - Henry", 0) == 0)
+         return "HHD";
+      auto end = sym.find(' ');
+      if(end == std::string::npos)
+         return sym;
+      else
+         return sym.substr(0, end);
+   }
+
    template<typename Definition>
    SecurityFilterAction filter_security(int exchange_id, const Definition& def) 
    {
@@ -130,7 +143,7 @@ private:
       }
       if(_symbol_filter.count("*") > 0)
          return ADD;
-      return _symbol_filter.count( parse_contract_symbol(def.contractSymbol) ) > 0 ? ADD : SKIP;
+      return _symbol_filter.count(get_contract_symbol(def)) > 0 ? ADD : SKIP;
    }
    void flush_books();
    void update_all_status(co2::CoconutSecurityStatus status);
@@ -146,7 +159,7 @@ template<typename Definition>
 void ICEListener::set_security_common(const Definition& def, FixedSecurity& fsec) 
 {
    fsec.numeric_id = def.marketId;
-   std::string sym = parse_contract_symbol(def.contractSymbol);
+   std::string sym = get_contract_symbol(def);
    if(!sym.empty())
    {
       fsec.symbol( sym );
@@ -165,7 +178,7 @@ void ICEListener::set_security_common(const Definition& def, FixedSecurity& fsec
    fsec.ice_settle_denominator = (int8_t)(def.settlePriceDenominator - '0');
    double min_tick = double_of_price(def.incrementPrice, fsec.ice_price_denominator);
    fsec.low_tick = fsec.high_tick = min_tick;
-   fsec.price_break = 0.0;    
+   fsec.price_break = 0.0;
    
    fsec.ice_market_description(def.marketDesc);
    fsec.ice_cleared_alias(def.clearedAlias);
@@ -207,12 +220,12 @@ void ICEListener::on_uds_future_spread(const Definition& def, bool& is_intereste
    // auto expiration = date_of_maturity(sec);
    // if (expiration < datetime::today()) return;
 
-   FixedSpreadSecurityL fsec;
+   FixedSpreadSecurityXL fsec;
    set_security_common(def, fsec);
    fsec.alt_symbol(def.strategySymbol);      
    
    const auto& legs = def.legs;
-   if (legs.size() > 0) {
+   if (legs.size() > 0/* && legs.size() <= 20*/) {
       fsec.set_leg_count(legs.size());
       for(size_t x = 0; x < fsec.leg_count(); ++x) 
       {

src/server/sparkd/md/platforms/ice_onixs/ICEMain.cpp

@@ -73,31 +73,17 @@ onixs_ice::MarketSubscriptions market_subcription_for_channel(const ICEChannel&
    onixs_ice::MarketSubscriptions subs;
    for(const auto& x : chan.market_types)
    {
-      //onixs_ice::MarketSubType::NonImplied
-      if(x == 26)
-      {
-         onixs_ice::MarketSubscription futures(x, onixs_ice::MarketSubType::NonImplied, onixs_ice::SecurityType::Futures, onixs_ice::BookDepth::FOD);
-         subs.push_back(futures);
-      }
-      else
-      {
-         onixs_ice::MarketSubscription futures(x, onixs_ice::MarketSubType::FullImplied, onixs_ice::SecurityType::Futures, onixs_ice::BookDepth::FOD);      
-         subs.push_back(futures);
-      }
+      onixs_ice::MarketSubscription futures(x, onixs_ice::MarketSubType::NonImplied, onixs_ice::SecurityType::Futures, onixs_ice::BookDepth::FOD);
+      subs.push_back(futures);
+
+      onixs_ice::MarketSubscription spreads(x, onixs_ice::MarketSubType::NonImplied, onixs_ice::SecurityType::UdsFuturesMarkets, onixs_ice::BookDepth::FOD);
+      subs.push_back(spreads);
    }
    for(const auto& x : chan.uds_market_types)
    {
-      if(x == 26)
-      {
-         onixs_ice::MarketSubscription spreads(x, onixs_ice::MarketSubType::NonImplied, onixs_ice::SecurityType::UdsFuturesMarkets, onixs_ice::BookDepth::FOD);
-         subs.push_back(spreads);
-      }
-      else
-      {
-         onixs_ice::MarketSubscription spreads(x, onixs_ice::MarketSubType::FullImplied, onixs_ice::SecurityType::UdsFuturesMarkets, onixs_ice::BookDepth::FOD);
-         subs.push_back(spreads);
-      }
-   }   
+      onixs_ice::MarketSubscription spreads(x, onixs_ice::MarketSubType::NonImplied, onixs_ice::SecurityType::UdsFuturesMarkets, onixs_ice::BookDepth::FOD);
+      subs.push_back(spreads);      
+   }
    return subs;
 }
 

src/server/sparkd/md/platforms/mdp3_onixs/MDP3Main.cpp

@@ -9,6 +9,7 @@
 #include "lib/utils/Dispatcher.h"
 #include "lib/utils/proto_helper.h"
 #include "lib/md/protos/co2.pb.h"
+#include "lib/md/SecurityDB.h"
 
 #include "lib/config/protos/MarketData.pb.h"
 #include "lib/config/protos/system.pb.h"
@@ -337,6 +338,23 @@ unique_ptr<MDP3Listener> MDP3Main::prepare_listener(onixs::Handler& handler, con
       listener->add_filter_symbol(symbol.symbol());
    }
 
+   // preload
+   if (datetime::is_local_time_sunday()) {
+      auto& preload_symbols = config::get<md::SecurityPreload>();
+      std::set<std::string> preload(preload_symbols.symbol().begin(), preload_symbols.symbol().end());
+      for (auto& symbol : channel.symbols) {
+         if (preload.find(symbol.symbol()) != preload.end()) {
+            std::set<UserSymbol> unit_set { symbol };
+            db::load_securities(unit_set, [&](const api::Security& security) {
+               with_security_like(security, [&](FixedSecurity& fs) {
+                  convert_security(security, fs, true);
+                  listener->add_security(fs, bts::co2::TradingStatus::TRADINGSTATUS_UNKNOWN);
+               });
+            });
+         }
+      }
+   }
+
    handler.registerListener(*listener);
    handler.registerMarketDataListener(*listener);
    handler.registerSecurityListener(*listener);

src/web/CMakeDeps.cmake

@@ -1 +1 @@
-set(sources ../../externals/linux/include/google/protobuf/descriptor.proto;../../externals/linux/include/google/protobuf/timestamp.proto;../lib/api/apicore.proto;../lib/api/apigateway.proto;../lib/api/apioptions.proto;../lib/api/services/AdminService.proto;../lib/api/services/AlgoDefinitionService.proto;../lib/api/services/AlgoService.proto;../lib/api/services/ApiUnitTestService.proto;../lib/api/services/AuthService.proto;../lib/api/services/BasePositionService.proto;../lib/api/services/ErrorReportService.proto;../lib/api/services/HelloService.proto;../lib/api/services/MBOService.proto;../lib/api/services/MarginService.proto;../lib/api/services/MarketDataAuxiliaryService.proto;../lib/api/services/MarketDataRelay.proto;../lib/api/services/MarketDataService.proto;../lib/api/services/MoneyFileService.proto;../lib/api/services/OrderService.proto;../lib/api/services/OutOfProcessURLListener.proto;../lib/api/services/PersistenceService.proto;../lib/api/services/PositionService.proto;../lib/api/services/ShellDiagnosticService.proto;../lib/api/services/ShellLatencyService.proto;../lib/api/services/SyntheticSecurityService.proto;../lib/domain_model/Security.proto;../lib/md/protos/co2.proto;../lib/sparkproto/SparkProtos.proto;api/build.sh;api/node_modules/exit/test/fixtures/create-files.sh;api/node_modules/protobufjs/cli/node_modules/acorn-jsx/node_modules/acorn/bin/update_authors.sh;api/node_modules/watch/release.sh;api/quiet_build.sh;api/src/DynamicService.ts;api/src/ServiceContext.ts;api/src/bts_assert.ts;ui/src/Access.fs;ui/src/Accounts/ClipLimits.fs;ui/src/Accounts/ProductLimits.fs;ui/src/Accounts/SecurityLimits.fs;ui/src/Accounts/State.fs;ui/src/Accounts/Types.fs;ui/src/Accounts/View.fs;ui/src/App.fs;ui/src/Cmd.fs;ui/src/Controls/Button.fs;ui/src/Controls/DataGrid.fs;ui/src/Controls/Fields.fs;ui/src/Controls/Modal.fs;ui/src/Controls/ReactGrid.fs;ui/src/Controls/Table.fs;ui/src/Controls/nbsp.fs;ui/src/Global.fs;ui/src/Imports.fs;ui/src/Lens.fs;ui/src/Login/Login.fs;ui/src/Nav.fs;ui/src/State.fs;ui/src/Types.fs;ui/src/Users/State.fs;ui/src/Users/Types.fs;ui/src/Users/View.fs;ui/src/Util/EnumUtils.fs;ui/src/Util/JSMap.fs;ui/src/Util/StreamUtils.fs;ui/src/obj/Debug/netstandard2.0/ui.AssemblyInfo.fs)
+set(sources ../../externals/linux/include/google/protobuf/descriptor.proto;../../externals/linux/include/google/protobuf/timestamp.proto;../lib/api/apicore.proto;../lib/api/apigateway.proto;../lib/api/apioptions.proto;../lib/api/services/AdminService.proto;../lib/api/services/AlgoDefinitionService.proto;../lib/api/services/AlgoService.proto;../lib/api/services/ApiUnitTestService.proto;../lib/api/services/AuthService.proto;../lib/api/services/BasePositionService.proto;../lib/api/services/ErrorReportService.proto;../lib/api/services/HelloService.proto;../lib/api/services/MBOService.proto;../lib/api/services/MarginService.proto;../lib/api/services/MarketDataAuxiliaryService.proto;../lib/api/services/MarketDataRelay.proto;../lib/api/services/MarketDataService.proto;../lib/api/services/MoneyFileService.proto;../lib/api/services/OrderService.proto;../lib/api/services/OutOfProcessURLListener.proto;../lib/api/services/PersistenceService.proto;../lib/api/services/PositionService.proto;../lib/api/services/ShellDiagnosticService.proto;../lib/api/services/ShellLatencyService.proto;../lib/api/services/SyntheticSecurityService.proto;../lib/domain_model/Security.proto;../lib/md/protos/co2.proto;../lib/sparkproto/SparkProtos.proto;api/build.sh;api/node_modules/exit/test/fixtures/create-files.sh;api/node_modules/object.assign/test.sh;api/node_modules/watch/release.sh;api/quiet_build.sh;api/src/DynamicService.ts;api/src/ServiceContext.ts;api/src/bts_assert.ts;ui/src/Access.fs;ui/src/Accounts/ClipLimits.fs;ui/src/Accounts/ProductLimits.fs;ui/src/Accounts/SecurityLimits.fs;ui/src/Accounts/State.fs;ui/src/Accounts/Types.fs;ui/src/Accounts/View.fs;ui/src/App.fs;ui/src/Cmd.fs;ui/src/Controls/Button.fs;ui/src/Controls/DataGrid.fs;ui/src/Controls/Fields.fs;ui/src/Controls/Modal.fs;ui/src/Controls/ReactGrid.fs;ui/src/Controls/Table.fs;ui/src/Controls/nbsp.fs;ui/src/Global.fs;ui/src/Imports.fs;ui/src/Lens.fs;ui/src/Login/Login.fs;ui/src/Nav.fs;ui/src/State.fs;ui/src/Types.fs;ui/src/Users/State.fs;ui/src/Users/Types.fs;ui/src/Users/View.fs;ui/src/Util/EnumUtils.fs;ui/src/Util/JSMap.fs;ui/src/Util/StreamUtils.fs;ui/src/obj/Debug/netstandard2.0/ui.AssemblyInfo.fs)

src/web/ui/src/Access.fs

@@ -102,7 +102,7 @@ module Accounts =
         }
       riskLimitEnforcement =
         {  productLimitInNetTerms = true
-           ignoreSpreadPositionLimits = false }
+           ignoreSpreadPositionLimits = true }
       accountLossLimit = 
         { enforce = false
           totalLossLimit = 0u }

testing/manual/algos/TASCounter.txt

@@ -0,0 +1,78 @@
+0. This needs sim market data **not** to be merged with replay (or cert), so we can create our own markets.
+
+1. This is a proprietary algo so its hidden behind a config key. Enable it by adding the following to config
+DynamicAlgos{
+   value {
+      enable_tascounter: true
+   }
+}
+
+2. We will run this test on the grain markets, speficially corn, so make sure you have ZC (it could also be run for other
+grain products, but this test is written for ZC)
+
+3. Open up Algo Studio and make a TASCounter algo.
+Concepts : 
+ZC expirations = Mar, May, Jul, Sep, Dec
+T1 contracts are adjacent calendar spreads so if the front month is May 
+T1 Front = May-Jul 
+T1 Second = Jul-Sep
+T1 Third = Sep-Dec 
+
+T2 skips an expiration
+T2 Front = May-Sep
+T2 Second = Jul-Dec
+T2 Third is not used
+
+T3 skips 2 expirations
+T3 Front = May-Dec
+
+Set up those contracts for the algo.
+
+4. The TAS contracts are the corresponding TAS contracts for ZC (ZCT is the TAS product for ZC).
+T1 Front = ZCT May-Jul
+T1 Second = ZCT Jul-Sep
+T1 Third = ZCT Sep-Dec 
+
+(Since we are running SIM, if these contracts are not available, choose any other contracts that are not used above)
+
+5. For T1 Front time, T1 Qty, and T1 Order Enabled enter 60, 10 and false respectively. Do the same for 
+T2 Front time, T2 Qty, and T2 Order Enabled, as well as T3 Front time, T3 Qty, and T3 Order Enabled.
+
+6. This algo runs during the settlement window of the product (ZC in this case). This is an input parameter which will 
+also make it easier for us to run this test. So enter a time (remember if your server is on Central time, make the adjustment)
+3 or 4 mins from now. Enter the settlement period as 180 seconds.
+
+7. Give the algo a name and save it.
+
+8. Bring up ladders and timesales for all 6 non TAS contracts. Bring up ladder for the TAS contracts.
+
+9. The contracts, price and qty are already part of the algo, so we can launch it from any ladder, with any price, or qty. Launch the algo
+
+10. Bring up the orderbook, and make sure the input params are correct.
+
+11. Make sure there is an output params grid under the input params. All other values except for "Seconds To Settle" should be NAN.
+"Seconds To Settle" should downcount toward the settle window, and should go to 0 when in settlement period.
+
+12. Now once you enter the settlement period, enter a few orders and cross them, across all 6 non TAS contracts. Enter them at a few
+price levels. Make sure they show up in the timesales window.
+
+13. Make sure the VWAP for all these 6 contracts is correct. VWAP is simply sum(price*qty) for all trades / sum (qty).
+
+14. Now make sure the Projected Settles for T1 Front, Second and Third is correct. This is how they are calculated
+T1 Front Projected Settle = VWAP (T1 Front)
+T1 Second Projected Settle = VWAP (T2 Front) - VWAP (T1 Front)
+T1 Third Projected Settle = VWAP(T3 Front) - VWAP(T1 Front) - VWAP(T1 Second)
+These values are rounded to the nearest tick (0.25 in this case)
+
+15. The last step it to make the algo actually shoot orders. I will explain what the concept is and that can be used for testing.
+This is a market making algo, so the trader wants to buy at the bid or sell at the offer. The TAS contract is a special contract that
+gives delivery of the actual contract at the price the TAS was traded at. So for example if you bought N TAS contracts on product X 
+at -1.0, it means you will get delivery of X at settlement (of X) +(-1). If you sold the TAS contract at 0, you will go short on X
+at settlement + 0.
+With this in mind, the algo watches the TAS market and crosses an order (ie buys at the offer or sells at the bid) if it gets delivery
+on the actual product such that its a buy at or below the bid or a sell at or above the offer.
+So for example the TAS Market is Bid=0, Ask=0-2. If we sell at Bid we will go short in the main product at Settlement + 0. 
+If our projected settle (from 14.) is >= the offer of the main product we will sell the TAS.
+The algo will buy or sell the TAS contract if the condition above holds AND if its Order Enabled = True and Time is within the window.
+
+Repeat the above for T1 Front, Second and Third.

tools/linux_control/build_in_docker.sh

@@ -1,7 +1,9 @@
 #!/bin/bash
+BUILD=clang_debug
+BUILD_LOC=/home/ppinto/temp/spark_build
 if [ -f /.dockerenv ]; then
-    ninja
+    cd $BUILD_LOC/$BUILD && ninja
 else
     cid=`docker ps | grep -P "emacs_ide|bluetradingsystems/linux_docker" | awk '{print $1}'`
-    docker exec $cid /bin/bash -c "cd /home/ppinto/temp/spark_build/clang_debug && ninja"
+    docker exec $cid /bin/bash -c "cd $BUILD_LOC/$BUILD && ninja"
 fi