Compare commits

..
220 Commits
Author SHA1 Message Date
Pete Fisher e974edfc6e Merge branch 'master' of bitbucket.org:bluetradesys/spark 2021-03-18 16:22:27 -04:00
Pete Fisher 3b7740cee0 updating submodule to latest 2021-03-18 16:22:12 -04:00
John Colvin 258d4a3fc1 import compute_synthetic_book method from spartan 2020-12-08 23:00:18 -06:00
John Colvin c53f6425cd import IntPrice arithmetic operators from spartan
The compute_synthetic_book method (also to be imported
from spartan) depends on a couple of them.
2020-12-08 22:52:28 -06:00
Pedro Pinto e5d77593d1 Changed logging 2020-10-15 14:24:02 -04:00
Pedro Pinto 6580f3883b Added logging of cancel/replace 2020-10-15 14:18:57 -04:00
Pete Fisher a66eae66bb fixing for python3 2020-10-14 21:56:49 -04:00
Pedro Pinto 090a9dfb17 Added iLink3 documentation. 2020-09-14 12:02:37 -04:00
Pete Fisher 27aa43f44d fixed time on ilink2 audit 2020-09-09 12:16:57 -04:00
Pedro Pinto 19e087aa69 Fixed time reporting in iLink3. 2020-09-09 10:58:34 -04:00
Pedro Pinto 167c8c23a0 Fixed batching of orders under iLink3. 2020-08-27 08:19:47 -04:00
Pete Fisher d235f4be41 fixed jetfix.conf ports on relay 2020-08-25 10:03:09 -04:00
Pedro Pinto ec43dbd82d Maybe fixed the self match prevention problem. 2020-08-24 17:02:58 -04:00
Pedro Pinto f07998d6a6 Fixed order rounding error in ilink3 2020-08-24 16:17:31 -04:00
Pedro Pinto b3dc2cdcd4 Changed logging subsystem from ilink3 to iLink3 2020-08-24 11:06:44 -04:00
Pete Fisher 0963aee137 added audit log scripts to be run by the docker container 2020-08-19 17:49:47 -04:00
Pedro Pinto c2f8e92b10 Hopefully fixed the ilink3 filled qty reporting problem.
This was crashing the spreader.
2020-08-13 08:42:43 -04:00
Pedro Pinto 7ae0142dff Fixed reset of sequence numbers problem 2020-08-12 11:56:08 -04:00
Pedro Pinto db0ffba8ed Fixed windows build 2020-07-10 13:00:20 -04:00
Pedro Pinto a274a214b4 Merged with master 2020-07-10 11:31:47 -04:00
Pedro Pinto bbcfa3c75a merged with prod 2020-07-10 10:45:10 -04:00
Pedro Pinto 89229d65f1 Merge branch 'master' of bitbucket.org:bluetradesys/spark 2020-07-10 10:43:34 -04:00
Pedro Pinto 0e75cde409 Fixed a bad asser in recovery, a few typos, haderned against bad securities. 2020-07-10 10:36:03 -04:00
Pedro Pinto 42fa6dfaff Fixed life cycle of exe reports and config during replay. 2020-07-09 16:06:39 -04:00
Pedro Pinto 400143c84d Protected against ticker subs for expired securities. 2020-07-09 14:45:51 -04:00
Pedro Pinto e915945880 Trying to replay fix logs. 2020-07-09 14:27:22 -04:00
Taha Afzal ca6673e98f Merge branch 'prod' 2020-07-09 13:09:57 -04:00
Taha Afzal 479f3dccff Adding OnSignal (this is a new algo which is hidden by default) 2020-07-09 13:07:28 -04:00
Pedro Pinto 9601ebab20 Adding replay of stored messages. 2020-07-09 10:16:31 -04:00
Taha Afzal c272d8a8c2 Pyramid Algo: Takes care of the rounding error when Top == sum (child orders) 2020-07-08 13:35:21 -04:00
Pedro Pinto b344ab3377 Fixed an issue with order recovery from Exec reports. 2020-07-08 12:07:48 -04:00
Pedro Pinto ae3080b031 Refactoring. 2020-07-08 10:10:14 -04:00
Pedro Pinto f0464331cb Added better handling of cancel rejects. 2020-07-08 09:52:18 -04:00
Pedro Pinto 0648b618bf Refactored. 2020-07-08 08:42:53 -04:00
Pedro Pinto 59282b8c70 Added one more test to manual order entry list. 2020-07-08 08:42:18 -04:00
Pedro Pinto 34a86009ce Doc and tool minor update. 2020-07-08 07:44:01 -04:00
Pedro Pinto 0ad9ae9001 Small tweak to logging tool. 2020-07-07 20:19:59 -04:00
Pedro Pinto a296e1e772 Fixed parsing of stop px in link3. 2020-07-07 20:19:41 -04:00
Pedro Pinto 7f53d8b161 Improved logging, fix iceberg order, removed duplicate logging. 2020-07-07 17:01:24 -04:00
John Colvin fbd8bbd4a8 algo abbreviation code uses string.Join instead of list_string (the latter method is redundant, should be removed) 2020-07-07 13:16:18 -05:00
John Colvin 3da52b350f fix pure crash (algo abbreviation code choked on names that end with a space) 2020-07-07 13:15:55 -05:00
Pedro Pinto 243d6827d4 Merge remote-tracking branch 'refs/remotes/origin/ilink3_final' into ilink3_final 2020-07-07 12:12:09 -04:00
Pedro Pinto 00221f9bd9 Fixed spread order entry. 2020-07-07 12:11:11 -04:00
Pedro Pinto a6171ae405 Fixed spread order entry. 2020-07-07 11:55:05 -04:00
Pedro Pinto be5d01a797 Fixed fills. 2020-07-07 06:47:58 -04:00
Pedro Pinto e5d9d324bf Order order entry manual test. 2020-07-06 18:22:39 -04:00
Pedro Pinto c34dee9d96 Taked order status from execution report, parse error message in il3 2020-07-06 18:21:44 -04:00
Pedro Pinto e5a98aab41 Merged with master 2020-07-06 12:49:24 -04:00
Pedro Pinto 10cd35372b Can enter, modify, cancel orders. 2020-07-06 11:38:17 -04:00
Pedro Pinto ad4515be39 Renamed NativeOrder to GLBXNativeOrder 2020-07-06 09:47:34 -04:00
Pedro Pinto ff4d87dd6b Fixed IL3 execution report parser to convert exchange prices. 2020-07-06 09:34:26 -04:00
Pedro Pinto 8ddde3f9da Part 2 of MBO service fix. 2020-07-06 09:34:09 -04:00
Pedro Pinto 51093dc3a5 Added simple tool to watch spark logs. 2020-07-06 09:33:35 -04:00
Pedro Pinto 5d5db9ad9f Fixed MBO parser to extract the right exchange key. 2020-07-06 09:33:06 -04:00
Pedro Pinto fca808fe0d Made ilink3 gw wait for start() before its starts. 2020-07-03 14:11:22 -04:00
Pedro Pinto 07d14f877b More nonsense 2020-07-02 20:28:49 -04:00
Pedro Pinto 5ea6dccdd4 Something here breaks 2020-07-02 19:04:49 -04:00
Pedro Pinto ac956a5c02 Renamed ExchangeOrderIDu64 to ExchangeOrderID64 2020-07-02 19:00:32 -04:00
Pedro Pinto 1fba09cf94 Refactor of ids. 2020-07-02 11:05:04 -04:00
Pedro Pinto 0baa6b875e Fixed merge error. 2020-07-02 09:09:20 -04:00
Pedro Pinto cdacc2d4d4 Merged 2020-07-02 07:26:43 -04:00
Pedro Pinto 1433541947 Automatically advanced our own seq number on connection failure.
It is fairly easy for our sequence number to be behind CME if we
crash. Since we can detect this issue (CME tells us about it) we can
try to fix it automatically by incrementing our own sequence
number. This should allow us to recover automatically in some
scenarios.
2020-07-02 06:57:01 -04:00
Pedro Pinto 1070ca7ace Ran clang-format. 2020-07-02 06:28:32 -04:00
Pedro Pinto 56fa27dd7e Handled non text timestamp in native order. 2020-07-02 06:23:07 -04:00
Pedro Pinto d87b522bc6 Added better logging when terminated by exchange. 2020-07-02 06:22:38 -04:00
Eric Harding 166cb03782 Provide account when parsing execution reports. 2020-07-01 14:40:10 -04:00
John Colvin 40bad84c48 algo studio includes algos with undefined children in the dropdown 2020-07-01 12:09:12 -05:00
Eric Harding c3786d9d64 ilink3: Provide SecurityOracle to execution report parser 2020-07-01 11:50:03 -04:00
Pedro Pinto 9080e3124b Filled in cust_order_handing_inst 2020-07-01 11:29:47 -04:00
Pedro Pinto d9c536124a Fixed logging of session seq numbers. 2020-07-01 11:29:30 -04:00
Pedro Pinto 7de6b2907e Filled in sendercompid in routing record.
iLink3 requires this.
2020-07-01 10:48:01 -04:00
Eric Harding 75f017e9f8 ilink3: Get PartyDetails from sender_cmp_id 2020-07-01 10:29:20 -04:00
Pedro Pinto 7757ab3ea0 Fixed a few bugs. 2020-06-30 19:35:53 -04:00
Pedro Pinto 3186dd383c We can connect now. 2020-06-30 19:03:35 -04:00
Pedro Pinto 45f751c9c6 Bad merge. 2020-06-30 19:03:35 -04:00
Pedro Pinto e4d3774980 Better logging. 2020-06-30 19:03:35 -04:00
Eric Harding eedb3ce05d Clear hmac for json logging 2020-06-30 17:11:01 -04:00
Eric Harding 6a7d38c0fd flip session bool 2020-06-30 16:10:26 -04:00
Eric Harding 7f9967834d add readme note 2020-06-30 15:34:03 -04:00
Eric Harding 31ccbfcd19 Switch to ilink3 logging group 2020-06-30 14:43:38 -04:00
Eric Harding c2abf90cc4 IL3MessageFramer owns outgoing messages so it needs to outlive ILinkGW 2020-06-30 14:36:02 -04:00
Eric Harding 8dc4735248 ilink3: set_state(Error) only after disconnect 2020-06-30 14:25:59 -04:00
Pedro Pinto d940d9f11d For now, caught serialization error in some ilink3 msgs. 2020-06-30 14:10:07 -04:00
Pedro Pinto 1e0c75c6c7 Added iLink3 subsystem for logging. 2020-06-30 14:09:55 -04:00
Eric Harding fbaa17bc40 Add some comments to marketdata.proto 2020-06-30 13:16:30 -04:00
Eric Harding 38e3cd6197 Update market data readme 2020-06-30 13:13:48 -04:00
Pedro Pinto 3bb9961af4 Refactor. 2020-06-30 11:35:40 -04:00
Pedro Pinto da8088fd8d Rebased 2020-06-30 11:27:56 -04:00
Eric Harding 748a65f671 Merge branch 'ilink3_gateway' of bitbucket.org:bluetradesys/spark into ilink3_gateway 2020-06-30 10:05:33 -04:00
Eric Harding 66d80136bd ilink3: Keep outgoing messages alive forever. 2020-06-30 10:04:42 -04:00
Eric Harding 10386311d4 Update market data readme 2020-06-29 12:25:30 -04:00
Eric Harding 629a7f8130 Remove unused class in MD 2020-06-29 12:25:14 -04:00
Pedro Pinto 81189bb2cc Forgot to add message logger. 2020-06-29 12:08:22 -04:00
Pedro Pinto e5681a50f2 Made session name come from date. 2020-06-29 10:59:15 -04:00
Pedro Pinto 69fe4569e3 Made FIX session name available as uint64_t as well as strings. 2020-06-29 10:57:40 -04:00
Pedro Pinto c7caf66f9c Changed the ILink3 log format to json. 2020-06-26 19:11:05 -04:00
Pedro Pinto 14504f4a94 Cleaned up comment. 2020-06-26 19:10:49 -04:00
Pedro Pinto 5b601a4016 Fixed encofing of clorid. 2020-06-26 19:10:23 -04:00
John Colvin e3207b921a truncate list of securities in transitive compare test (was timing out due to too many permutations) 2020-06-26 13:32:27 -05:00
Taha Afzal 98db8e61a5 Removing hardcoded ref's to Katallage 2020-06-25 17:05:10 -04:00
Pedro Pinto 45aa75e8bb Brought back the ilink3/ilink2 encoding abstraction. 2020-06-25 16:49:46 -04:00
Pedro Pinto b2471d45b2 Added tool 2020-06-25 16:49:10 -04:00
Pedro Pinto 11e2df8c76 Tweaked build tools. 2020-06-25 16:49:10 -04:00
Pedro Pinto cba6852655 Updated docker 2020-06-25 16:49:10 -04:00
Pedro Pinto 5eeaedde26 Added a few build tools. 2020-06-25 16:49:10 -04:00
Pedro Pinto b63052dbaa Updated ICE Onixs 2020-06-25 16:49:10 -04:00
Pedro Pinto 29f5bc798e Added +x 2020-06-25 16:49:10 -04:00
Pedro Pinto 04603acbb9 Added publishing of new routing info when orders are recorevered from disk 2020-06-25 16:49:10 -04:00
Pedro Pinto bc6890055e temp commit, fixing glbxnative order 2020-06-25 16:49:10 -04:00
Pedro Pinto ea94c09c78 Eric intermediate ilink3 fixes 2020-06-25 16:49:10 -04:00
Eric Harding e3c3bc78fc Note what this address list is for 2020-06-24 20:55:20 -04:00
Eric Harding a11d390dcd ilink3 message logging 2020-06-24 20:52:21 -04:00
Eric Harding 7c74a961d4 Add canned cert sessions for ilink3 nrcert 2020-06-24 20:43:27 -04:00
Eric Harding 5db21ad2ca fix build, session and clordid changes 2020-06-24 19:32:12 -04:00
Eric Harding f703dabdfe Raw msg should be 2 strings 2020-06-24 17:53:30 -04:00
Eric Harding 237d3eea4b encode_clordid needs security key 2020-06-24 17:29:06 -04:00
Taha Afzal e009316bfb Merge branch 'DJ_ICE' of bitbucket.org:bluetradesys/spark into DJ_ICE 2020-06-24 17:13:38 -04:00
Taha Afzal 5528efd2d2 Cleaning up 2020-06-24 17:13:31 -04:00
Eric Harding 818bdec130 Add NrCert keys 2020-06-24 15:07:02 -04:00
John Colvin 09f91735fe Security sort tests run with ice securities 2020-06-24 12:58:29 -05:00
John Colvin d9fc18547b SecurityHelper.Compare fallsback on 'ToString' then 'Id' comparisons to guarentee deterministic sort 2020-06-24 12:56:29 -05:00
Eric Harding d257cf2c77 Disable broken ilink3 tests for now 2020-06-24 13:28:33 -04:00
Eric Harding 079f46404f Add some testing notes 2020-06-24 10:57:15 -04:00
Eric Harding edcc73e934 Remove old ILink3ExecutionGateway. Instead use GlbxIlink3ExecutionGateway 2020-06-23 22:41:33 -04:00
Taha Afzal 5905ca3e92 Merging frmo master 2020-06-23 13:51:14 -04:00
Pedro Pinto bc7ac8523b Added tool 2020-06-23 13:06:15 -04:00
Eric Harding 0c4c2b7231 cme autocert msgw_failover test script 2020-06-23 13:05:32 -04:00
Eric Harding 420351c093 Replay refactor #4 2020-06-23 12:01:52 -04:00
Eric Harding f69e6b89c5 Sequence number fixes 2020-06-22 22:23:13 -04:00
Eric Harding c3e22e4b23 Recovery revamp #3 2020-06-22 20:16:21 -04:00
John Colvin 32b7b7f40d quantity conversion for ladder right click order 2020-06-22 17:02:23 -05:00
Pedro Pinto 54e5a68730 Tweaked build tools. 2020-06-22 08:41:19 -04:00
Pedro Pinto 880c95d92b Tweaked build tools. 2020-06-20 17:17:22 -04:00
Eric Harding 3231ffb343 Meaningless autocert code 2020-06-15 22:48:28 -04:00
Eric Harding 8d7f1738cf update cert secure logon 2020-06-15 22:47:30 -04:00
Eric Harding 9101214e66 Add message logger and revamp replay sequence handling 2020-06-15 14:51:41 -04:00
Pedro Pinto 59774ccbb5 Updated docker 2020-06-14 15:27:18 -04:00
Pedro Pinto fcb6c9b8d9 Added a few build tools. 2020-06-14 15:27:01 -04:00
Eric Harding c6b2e2f21e OMF setting to true, always 2020-06-12 17:49:37 -04:00
Eric Harding 2eac97c1a9 meaningless cert change 2020-06-12 15:18:20 -04:00
Eric Harding b4b0e10f29 Fix IOC=FAK order type (not FOK) 2020-06-12 15:17:38 -04:00
Eric Harding 38d9db4a90 Session and sequence number fixes 2020-06-11 11:36:37 -04:00
Eric Harding 6a679ff78c Fix for cancel/replace 2020-06-10 14:13:28 -04:00
Eric Harding aba3f21020 stop order does not carry price 2020-06-09 19:18:36 -04:00
Eric Harding fede19fd11 stop-limit success 2020-06-09 16:30:00 -04:00
Eric Harding 3fa8dbf112 Cannot send price with market order 2020-06-08 23:09:10 -04:00
Eric Harding 77ae4703b3 Fix clOrdID and custOrdCapacity 2020-06-08 22:58:53 -04:00
Eric Harding a28bfa1175 execution report handling tweaks 2020-06-04 10:05:44 -04:00
Eric Harding 4e4a48283d bool parse_report(string&, SecurityOracle&, ExecutionReport*); 2020-06-03 13:09:56 -04:00
Pedro Pinto 7a761d00ce Added security oracle to SessionStore so we can resolve exe reports. 2020-06-03 08:51:31 -04:00
Pedro Pinto 968be6a99a Added storage for sessions. Still missing parsing of exec reports. 2020-06-02 17:13:53 -04:00
Pedro Pinto 03404eb3f1 Made some progress in GLBXiLink3ExecutionGateway 2020-06-02 09:16:02 -04:00
Pedro Pinto ff1a970982 Fixed build some more 2020-05-28 14:29:45 -04:00
Pedro Pinto ed890b4d84 Fixed broken tests for subscribe_exe_report change 2020-05-28 13:50:04 -04:00
Pedro Pinto e87003cc41 Merged with master 2020-05-28 12:21:51 -04:00
Eric Harding bc06037843 Fix the build 2020-05-28 12:20:30 -04:00
Pedro Pinto c4c7cebfc5 Updated externals 2020-05-28 12:12:46 -04:00
Pedro Pinto f18bcd2995 Build tweaks 2020-05-28 12:12:46 -04:00
Pedro Pinto a6a1b6ac53 Fixed clang build. 2020-05-28 12:12:46 -04:00
Eric Harding 8c3da4e43c Re-arrange some methods in ILinkGW.h 2020-05-27 16:51:36 -04:00
Eric Harding 06b955dceb Exchange seq number fix 2020-05-27 11:49:08 -04:00
Eric Harding e75d15146c cme cert test update 2020-05-26 23:55:27 -04:00
Eric Harding cb07384c9f Fix problem with party details 2020-05-26 23:55:13 -04:00
Eric Harding fc9e502539 cert refactor 2020-05-26 12:40:16 -04:00
Eric Harding fcbf3388c0 update ilinkbinaryxml templates 2020-05-26 12:40:04 -04:00
Eric Harding 899bea940f improve ilinkgw logging 2020-05-26 12:39:55 -04:00
Eric Harding 14b7509607 ensure local launch timezone 2020-05-26 12:39:41 -04:00
Eric Harding 25668e8df0 dont' build web client on wsl2 2020-05-26 11:54:19 -04:00
Eric Harding 5345ba8ff8 securities for sending orders ilink3 test 2020-05-19 15:58:56 -05:00
Eric Harding 9bf05973bd Add in/out to logs 2020-05-14 20:51:15 -05:00
Eric Harding cea4d05bc4 log every msg 2020-05-13 20:19:48 -05:00
Eric Harding 0b7208784b Can't take the double negatives anymore 2020-05-13 14:01:16 -05:00
Eric Harding 059ccbdab6 Merge branch 'master' of bitbucket.org:bluetradesys/spark into ilink3_gateway 2020-05-13 11:05:31 -05:00
Eric Harding 75eaf8c3d4 IL3ExecutionReportParser 2020-05-12 13:54:28 -05:00
Eric Harding 1e8fcd794f Update ExecutionReport::exchange_order_id 2020-05-06 12:22:33 -05:00
Eric Harding 726329c4bf Add ilink3 execution report parser and order test 2020-05-06 11:56:52 -05:00
Eric Harding 2d01f3f0f4 CME Certification requires very specific UUID generation 2020-05-06 11:56:01 -05:00
Eric Harding 2a595c0fd7 cme cert hard coding 2020-05-06 11:54:32 -05:00
Eric Harding 5f9866c9e8 SecurityCache now hands out raw pointers. SecurityOracle now indexes exchange security key 2020-05-05 11:17:16 -05:00
Eric Harding 78f1cf88e0 cmake needs love too 2020-05-04 11:16:21 -05:00
Eric Harding 4877436877 Ilink3 order unit test 2020-05-04 11:32:30 -04:00
Eric Harding 1b43124756 Add ReentrancyGuard.h 2020-05-02 13:10:49 -04:00
Eric Harding a1feb3fddf PrimaryGatewayConnectionEnforcement 2020-05-01 13:22:35 -04:00
Eric Harding 024328bf8b Refactor empty session 2020-05-01 11:44:03 -04:00
Eric Harding 80d7e34bf1 Fixes for updated sbe codegen 2020-04-28 18:16:45 -04:00
Eric Harding 8ab69c919b Improve cme provided error message 2020-04-27 16:56:32 -04:00
Eric Harding 99531bcfa0 Update ilink3 generated templates 2020-04-27 16:50:54 -04:00
Eric Harding 6bd516d4f2 cleanup 2020-04-27 11:26:42 -04:00
Eric Harding 1a1d771e1b addition ilink3 tests 2020-04-25 13:18:58 -04:00
Eric Harding 050fcf7c6e Passing simple log in test 2020-04-24 12:31:30 -04:00
Eric Harding aadeae3201 update cmake, avoid web build on wsl 2020-04-17 22:38:53 -04:00
Eric Harding b72e8aed02 ilink3 test code 2020-04-17 20:30:11 -05:00
Eric Harding 0820cc3e9a Fix a very subtle bug in serializer 2020-04-17 13:15:12 -05:00
Eric Harding b1ba9a1315 update ilink code gen readme 2020-04-16 14:03:53 -05:00
Eric Harding 39ce4323a1 Add CME cert command line tool 2020-04-13 16:18:59 -05:00
Eric Harding d9fcd290d9 source files don't need to be executable 2020-04-13 09:53:28 -05:00
Eric Harding 05939b02c5 save 2020-04-12 15:11:46 -05:00
Eric Harding aa91638fe4 ilink3 receive buffer handler + tests 2020-04-10 10:55:53 -05:00
Eric Harding 065fc54f73 IL3Reactor 2020-04-07 14:05:31 -05:00
Eric Harding 95fb13f58b Fix some build issues 2020-04-01 15:21:04 -05:00
Eric Harding 4eb2931cb8 Additional ilink3 refactoring 2020-03-28 21:08:47 -05:00
Eric Harding 5326015a48 Testcase for framer 2020-03-27 13:46:29 -05:00
Eric Harding fcbe319470 Split out functional pieces for easier testing 2020-03-27 09:14:30 -05:00
Eric Harding 5ad21c88b2 Add tests and refactor includes for faster build 2020-03-25 21:15:19 -05:00
Eric Harding 3a9c1d557b Split Retransmit of old vs new session logic 2020-03-09 16:29:15 -05:00
Eric Harding 6fb5b83249 Only send Negotiate if the session is new 2020-02-28 15:33:27 -06:00
Eric Harding 95830fafdf send_cancel 2020-02-27 14:48:56 -06:00
Eric Harding d68b03bb91 send_cancel_replace 2020-02-26 17:56:50 -06:00
Eric Harding e7062ce01e send_order call for link3 2020-02-26 17:36:18 -06:00
Pedro Pinto 932738f98e Merge branch 'master' into ilink3_gateway 2020-02-20 08:39:47 -05:00
Pedro Pinto 1426b00e83 Added fake iLink3 gateway 2020-02-19 16:21:06 -05:00
Pedro Pinto e724409b58 Added jetfix support for string_view 2020-02-19 10:55:56 -05:00
Pedro Pinto 9cf828e9a4 Implemented iLink3 order 2020-02-18 15:37:06 -05:00
Eric Harding 41408fb609 ILink3ExecutionGateway placeholder 2020-02-14 16:07:44 -06:00
Eric Harding 4c21e32838 ILink3GW core gateway 2020-02-14 16:06:58 -06:00
Eric Harding 385f3ea402 add unix_nano_now() to datetime.h 2020-02-14 16:05:00 -06:00
Eric Harding 8d6d8cab18 Update SecureLogon for ilink3 2020-02-14 16:05:00 -06:00
Eric Harding 01615f824a Generated ilink3 messages 2020-02-14 16:05:00 -06:00
Pedro Pinto c6779ddc90 Reactore GLBX native order for iLink3. 2020-02-13 15:26:33 -05:00
Pedro Pinto 46eff65435 Changed IExecutionGateway to support non FIX gateways. 2020-02-12 10:11:53 -05:00
272 changed files with 152689 additions and 1956 deletions

No files matched your search

+1
View File
@@ -37,3 +37,4 @@ trace_data/
*.rej
/build
.clangd/
*CMakeDeps.cmake
+9 -5
View File
@@ -8,13 +8,15 @@
"program": "${workspaceRoot}/build/ninja_debug/server/sparkd",
"args": [
"run",
// "test",
// "-a",
// "[ilink3]"
"test",
"-a",
"OutrightVanillaOrder"
],
"stopAtEntry": false,
"cwd": "${workspaceRoot}/build/ninja_debug/server/",
"environment": [],
"environment": [
{ "name": "TZ", "value": "America/Chicago" }
],
"externalConsole": false,
"MIMode": "gdb",
"setupCommands": [
@@ -31,7 +33,9 @@
"request": "launch",
"program": "/home/bts/spark/build/ninja_debug/server/sparkd",
"args": [
"run"
"test",
"-a",
"[ilink3]"
],
"stopAtEntry": false,
"cwd": "/home/bts/spark/build/ninja_debug/server/",
+1
View File
@@ -41,6 +41,7 @@ else ()
-Wno-missing-braces -Wno-ignored-qualifiers \
--std=c++17 -fdiagnostics-color")
set (CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_DEBUG} -fno-omit-frame-pointer -DNDEBUG -O3")
set (CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -DDEBUG")
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wno-undef")
add_subdirectory(src/server/sparkd)
add_subdirectory(src/lib)
+1 -1
View File
@@ -1 +1 @@
set(sources /home/ppinto/Dropbox/bts/spark/doc/user/build.sh;content/_index.md;content/bts_spark/Glossary/_index.md;content/bts_spark/Help/_index.md;content/bts_spark/Help/about_spark.png;content/bts_spark/Help/image39.png;content/bts_spark/Help/image58.png;content/bts_spark/Launcher/_index.md;content/bts_spark/Launcher/image38.png;content/bts_spark/Launcher/image68.png;content/bts_spark/Logging-In-to-the-System/_index.md;content/bts_spark/Logging-In-to-the-System/image3.png;content/bts_spark/Logging-In-to-the-System/image64.png;content/bts_spark/Market/_index.md;content/bts_spark/Market/change_securities.png;content/bts_spark/Market/image11.png;content/bts_spark/Market/image14.png;content/bts_spark/Market/image20.png;content/bts_spark/Market/image21.png;content/bts_spark/Market/image23.png;content/bts_spark/Market/image24.png;content/bts_spark/Market/image34.png;content/bts_spark/Market/image49.png;content/bts_spark/Market/image53.png;content/bts_spark/Market/image66.png;content/bts_spark/Market/image69.png;content/bts_spark/Market/market_view.png;content/bts_spark/Market/settings_icon.png;content/bts_spark/Market/time_and_sales.png;content/bts_spark/My-Windows/_index.md;content/bts_spark/My-Windows/image15.png;content/bts_spark/My-Windows/image16.png;content/bts_spark/My-Windows/image37.png;content/bts_spark/Settings/_index.md;content/bts_spark/Settings/image51.png;content/bts_spark/Settings/image9.png;content/bts_spark/Spreader/_index.md;content/bts_spark/Spreader/image10.png;content/bts_spark/Spreader/image13.png;content/bts_spark/Spreader/image18.png;content/bts_spark/Spreader/image19.png;content/bts_spark/Spreader/image28.png;content/bts_spark/Spreader/image29.png;content/bts_spark/Spreader/image31.png;content/bts_spark/Spreader/image35.png;content/bts_spark/Spreader/image40.png;content/bts_spark/Spreader/image43.png;content/bts_spark/Spreader/image54.png;content/bts_spark/Spreader/image55.png;content/bts_spark/Spreader/image56.png;content/bts_spark/Spreader/image6.png;content/bts_spark/Spreader/image7.png;content/bts_spark/Spreader/image70.png;content/bts_spark/Spreader/image72.png;content/bts_spark/Spreader/image8.png;content/bts_spark/Trading/OrderTypes/index.md;content/bts_spark/Trading/_index.md;content/bts_spark/Trading/algo_studio.png;content/bts_spark/Trading/collapse_all_icon.png;content/bts_spark/Trading/cover_algo_control.png;content/bts_spark/Trading/expand_all_icon.png;content/bts_spark/Trading/image1.png;content/bts_spark/Trading/image17.png;content/bts_spark/Trading/image2.png;content/bts_spark/Trading/image25.png;content/bts_spark/Trading/image27.png;content/bts_spark/Trading/image33.png;content/bts_spark/Trading/image42.png;content/bts_spark/Trading/image44.png;content/bts_spark/Trading/image47.png;content/bts_spark/Trading/image48.png;content/bts_spark/Trading/image5.png;content/bts_spark/Trading/image50.png;content/bts_spark/Trading/image52.png;content/bts_spark/Trading/image57.png;content/bts_spark/Trading/image59.png;content/bts_spark/Trading/image60.png;content/bts_spark/Trading/image61.png;content/bts_spark/Trading/image62.png;content/bts_spark/Trading/image73.png;content/bts_spark/Trading/image74.png;content/bts_spark/Trading/join_algo_control.png;content/bts_spark/Trading/ladder.png;content/bts_spark/Trading/ladder_badges.png;content/bts_spark/Trading/ladder_main_part.png;content/bts_spark/Trading/lean_algo_control.png;content/bts_spark/Trading/mini_orderbook.png;content/bts_spark/Trading/oco_algo_control.png;content/bts_spark/Trading/offset_algo_control.png;content/bts_spark/Trading/pnl_calculation.png;content/bts_spark/Trading/pnl_window.png;content/bts_spark/Trading/recreate_icon.png;content/bts_spark/Trading/settings_icon.png;content/bts_spark/Trading/spark_menu_order_book.png;content/bts_spark/Trading/top_algo_control.png;content/bts_spark/Trading/trigger_algo_control.png;content/bts_spark/_index.md;content/release_notes/_index.md;content/system_requirements/_index.md;static/book.js;static/favicon.ico;static/image.js;templates/index.html;templates/page.html;templates/section.html;templates/shortcodes/back_button.html)
set(sources /home/pfisher/bts/spark/doc/user/build.sh;content/_index.md;content/bts_spark/Glossary/_index.md;content/bts_spark/Help/_index.md;content/bts_spark/Help/about_spark.png;content/bts_spark/Help/image39.png;content/bts_spark/Help/image58.png;content/bts_spark/Launcher/_index.md;content/bts_spark/Launcher/image38.png;content/bts_spark/Launcher/image68.png;content/bts_spark/Logging-In-to-the-System/_index.md;content/bts_spark/Logging-In-to-the-System/image3.png;content/bts_spark/Logging-In-to-the-System/image64.png;content/bts_spark/Market/_index.md;content/bts_spark/Market/change_securities.png;content/bts_spark/Market/image11.png;content/bts_spark/Market/image14.png;content/bts_spark/Market/image20.png;content/bts_spark/Market/image21.png;content/bts_spark/Market/image23.png;content/bts_spark/Market/image24.png;content/bts_spark/Market/image34.png;content/bts_spark/Market/image49.png;content/bts_spark/Market/image53.png;content/bts_spark/Market/image66.png;content/bts_spark/Market/image69.png;content/bts_spark/Market/market_view.png;content/bts_spark/Market/settings_icon.png;content/bts_spark/Market/time_and_sales.png;content/bts_spark/My-Windows/_index.md;content/bts_spark/My-Windows/image15.png;content/bts_spark/My-Windows/image16.png;content/bts_spark/My-Windows/image37.png;content/bts_spark/Settings/_index.md;content/bts_spark/Settings/image51.png;content/bts_spark/Settings/image9.png;content/bts_spark/Spreader/_index.md;content/bts_spark/Spreader/image10.png;content/bts_spark/Spreader/image13.png;content/bts_spark/Spreader/image18.png;content/bts_spark/Spreader/image19.png;content/bts_spark/Spreader/image28.png;content/bts_spark/Spreader/image29.png;content/bts_spark/Spreader/image31.png;content/bts_spark/Spreader/image35.png;content/bts_spark/Spreader/image40.png;content/bts_spark/Spreader/image43.png;content/bts_spark/Spreader/image54.png;content/bts_spark/Spreader/image55.png;content/bts_spark/Spreader/image56.png;content/bts_spark/Spreader/image6.png;content/bts_spark/Spreader/image7.png;content/bts_spark/Spreader/image70.png;content/bts_spark/Spreader/image72.png;content/bts_spark/Spreader/image8.png;content/bts_spark/Trading/OrderTypes/index.md;content/bts_spark/Trading/_index.md;content/bts_spark/Trading/algo_studio.png;content/bts_spark/Trading/collapse_all_icon.png;content/bts_spark/Trading/cover_algo_control.png;content/bts_spark/Trading/expand_all_icon.png;content/bts_spark/Trading/image1.png;content/bts_spark/Trading/image17.png;content/bts_spark/Trading/image2.png;content/bts_spark/Trading/image25.png;content/bts_spark/Trading/image27.png;content/bts_spark/Trading/image33.png;content/bts_spark/Trading/image42.png;content/bts_spark/Trading/image44.png;content/bts_spark/Trading/image47.png;content/bts_spark/Trading/image48.png;content/bts_spark/Trading/image5.png;content/bts_spark/Trading/image50.png;content/bts_spark/Trading/image52.png;content/bts_spark/Trading/image57.png;content/bts_spark/Trading/image59.png;content/bts_spark/Trading/image60.png;content/bts_spark/Trading/image61.png;content/bts_spark/Trading/image62.png;content/bts_spark/Trading/image73.png;content/bts_spark/Trading/image74.png;content/bts_spark/Trading/join_algo_control.png;content/bts_spark/Trading/ladder.png;content/bts_spark/Trading/ladder_badges.png;content/bts_spark/Trading/ladder_main_part.png;content/bts_spark/Trading/lean_algo_control.png;content/bts_spark/Trading/mini_orderbook.png;content/bts_spark/Trading/oco_algo_control.png;content/bts_spark/Trading/offset_algo_control.png;content/bts_spark/Trading/pnl_calculation.png;content/bts_spark/Trading/pnl_window.png;content/bts_spark/Trading/recreate_icon.png;content/bts_spark/Trading/settings_icon.png;content/bts_spark/Trading/spark_menu_order_book.png;content/bts_spark/Trading/top_algo_control.png;content/bts_spark/Trading/trigger_algo_control.png;content/bts_spark/_index.md;content/release_notes/_index.md;content/system_requirements/_index.md;static/book.js;static/favicon.ico;static/image.js;templates/index.html;templates/page.html;templates/section.html;templates/shortcodes/back_button.html)
+209 -203
View File
@@ -1,40 +1,40 @@
using bts.api;
using bts.api.core;
using bts.api.orderman;
using bts.utils;
using mbts.spark.utilities;
using bts.api;
using bts.api.core;
using bts.api.orderman;
using bts.utils;
using mbts.spark.utilities;
using mbts.ui.utility;
using mbts.utils;
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using System.Reactive.Linq;
namespace mbts.spark {
interface IAlgoProvider {
AlgoTypeSetImmutable get_permitted_algos(SecurityImmutable security);
AlgoTypeSetImmutable get_permitted_algos(SecurityCriteriaClause clause);
IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_algos();
IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_algos(SecurityImmutable security);
IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_algos(SecurityCriteriaClause clause);
}
// This interface is used to distinguish the attachable algo provider
// from the standard one wrt MEF export/import.
interface IAttachableAlgoProvider : IAlgoProvider { }
class AlgoProvider : IAlgoProvider {
readonly IEnumerable<SecurityAlgoTypeSetImmutable> _security_algo_types;
readonly IAlgoStudioModel _studio_model;
readonly ISecurityAuxiliaryDataModel _security_auxiliary_model;
readonly IObservable<TableUpdate<AlgoData, AlgoKey>> _studio_algos_source;
readonly DynamicOrderHelper _dynamic_helper;
readonly AlgoTypeSetImmutable _full_set;
public AlgoProvider(IEnumerable<SecurityAlgoTypeSetImmutable> algo_sets, IAlgoStudioModel studio_model, DynamicOrderHelper dynamic_helper, ISecurityAuxiliaryDataModel security_auxiliary_model) {
_security_algo_types = algo_sets;
_studio_model = studio_model;
using mbts.utils;
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using System.Reactive.Linq;
namespace mbts.spark {
interface IAlgoProvider {
AlgoTypeSetImmutable get_permitted_algos(SecurityImmutable security);
AlgoTypeSetImmutable get_permitted_algos(SecurityCriteriaClause clause);
IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_algos();
IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_algos(SecurityImmutable security);
IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_algos(SecurityCriteriaClause clause);
}
// This interface is used to distinguish the attachable algo provider
// from the standard one wrt MEF export/import.
interface IAttachableAlgoProvider : IAlgoProvider { }
class AlgoProvider : IAlgoProvider {
readonly IEnumerable<SecurityAlgoTypeSetImmutable> _security_algo_types;
readonly IAlgoStudioModel _studio_model;
readonly ISecurityAuxiliaryDataModel _security_auxiliary_model;
readonly IObservable<TableUpdate<AlgoData, AlgoKey>> _studio_algos_source;
readonly DynamicOrderHelper _dynamic_helper;
readonly AlgoTypeSetImmutable _full_set;
public AlgoProvider(IEnumerable<SecurityAlgoTypeSetImmutable> algo_sets, IAlgoStudioModel studio_model, DynamicOrderHelper dynamic_helper, ISecurityAuxiliaryDataModel security_auxiliary_model) {
_security_algo_types = algo_sets;
_studio_model = studio_model;
_dynamic_helper = dynamic_helper;
_full_set = new AlgoTypeSetImmutable(
ImmutableHashSet.CreateRange(
@@ -51,42 +51,42 @@ namespace mbts.spark {
_studio_algos_source = studio_model.definitions
.StartWith(TableUpdate<AlgoData, AlgoKey>.mk_empty(AlgoData.key_fn))
.Cache();
}
public AlgoTypeSetImmutable get_permitted_algos(SecurityImmutable security) =>
_security_algo_types.Where(p => p.security_criteria.includes(security))
.Select(sec_algos => sec_algos.algo_type_set)
.Aggregate(_full_set, (result, algos) => result.intersect_with(algos));
public AlgoTypeSetImmutable get_permitted_algos(SecurityCriteriaClause clause) =>
_security_algo_types.Where(p => p.security_criteria.is_subset_of(clause))
.Select(sec_algos => sec_algos.algo_type_set)
.Aggregate(_full_set, (result, algos) => result.intersect_with(algos));
public IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_algos() =>
_studio_algos_source
.AppendTable(_security_algo_types
.Select(s => s.algo_type_set)
.Aggregate(AlgoTypeSetImmutable.empty, (a, b) => a.union_with(b))
.define_algos(_dynamic_helper));
public IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_algos(SecurityCriteriaClause security_clause) {
var system_algos = _security_algo_types.match_algos(
security_clause,
_dynamic_helper,
def => _fit_to_type(def, security_clause, _security_algo_types));
return _studio_algos_source.Select(
a => (algo: a, fitted_def: _fit_to_type(a.value, security_clause, _security_algo_types)),
tup => tup.algo.key)
.Select(tup => AlgoData.create(tup.fitted_def, tup.algo.name, tup.algo.key), a => a.key)
.AppendTable(system_algos.Values);
}
public IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_algos(SecurityImmutable security) {
var clause = new SecurityCriteriaClause(
exchanges: new[] { security.Exchange },
security_types: new[] { security.security_type() });
}
public AlgoTypeSetImmutable get_permitted_algos(SecurityImmutable security) =>
_security_algo_types.Where(p => p.security_criteria.includes(security))
.Select(sec_algos => sec_algos.algo_type_set)
.Aggregate(_full_set, (result, algos) => result.intersect_with(algos));
public AlgoTypeSetImmutable get_permitted_algos(SecurityCriteriaClause clause) =>
_security_algo_types.Where(p => p.security_criteria.is_subset_of(clause))
.Select(sec_algos => sec_algos.algo_type_set)
.Aggregate(_full_set, (result, algos) => result.intersect_with(algos));
public IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_algos() =>
_studio_algos_source
.AppendTable(_security_algo_types
.Select(s => s.algo_type_set)
.Aggregate(AlgoTypeSetImmutable.empty, (a, b) => a.union_with(b))
.define_algos(_dynamic_helper));
public IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_algos(SecurityCriteriaClause security_clause) {
var system_algos = _security_algo_types.match_algos(
security_clause,
_dynamic_helper,
def => _fit_to_type(def, security_clause, _security_algo_types));
return _studio_algos_source.Select(
a => (algo: a, fitted_def: _fit_to_type(a.value, security_clause, _security_algo_types)),
tup => tup.algo.key)
.Select(tup => AlgoData.create(tup.fitted_def, tup.algo.name, tup.algo.key), a => a.key)
.AppendTable(system_algos.Values);
}
public IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_algos(SecurityImmutable security) {
var clause = new SecurityCriteriaClause(
exchanges: new[] { security.Exchange },
security_types: new[] { security.security_type() });
// i.e., non-studio algos
var system_algos = _security_algo_types.match_algos(
clause,
@@ -94,135 +94,141 @@ namespace mbts.spark {
def => _fit_to_type(def, security, _security_algo_types),
_security_auxiliary_model.get_display_quantity_divisor(security));
return _studio_algos_source.Select(
a => (algo: a, fitted_def: _fit_to_type(a.value, security, _security_algo_types)),
tup => tup.algo.key)
// filter out algos that use an unsupported root, because this would have the effect of duplicating the default algo
// (e.g., GlbxIceberg orders would be converted to Limit/Spreader orders for non-Glbx securities).
.Filter(tup => tup.fitted_def.actual().CustomCase == tup.algo.value.actual().CustomCase, tup => tup.algo.key)
.Select(tup => AlgoData.create(tup.fitted_def, tup.algo.name, tup.algo.key), AlgoData.key_fn)
return _studio_algos_source.Select(
a => (algo: a, fitted_def: _fit_to_type(a.value, security, _security_algo_types)),
tup => tup.algo.key)
// filter out algos that use an unsupported root, because this would have the effect of duplicating the default algo
// (e.g., GlbxIceberg orders would be converted to Limit/Spreader orders for non-Glbx securities).
.Filter(tup => tup.fitted_def.actual().CustomCase == tup.algo.value.actual().CustomCase, tup => tup.algo.key)
.Select(tup => AlgoData.create(tup.fitted_def, tup.algo.name, tup.algo.key), AlgoData.key_fn)
.AppendTable(system_algos.Values);
}
protected virtual void assign_fallback_algo(CustomOrderDefinition mdef, SecurityCriteriaClause security_clause) {
if (security_clause.security_types.empty_if_null().Contains(Security.ContractOneofCase.Synthetic.security_type())) {
mdef.Spreader = new SpreaderOrderDefinition();
} else {
mdef.Limit = new LimitOrderDefinition();
}
}
protected virtual void assign_fallback_algo(CustomOrderDefinition mdef, SecurityImmutable security) {
if (security.ContractCase == Security.ContractOneofCase.Synthetic) {
mdef.Spreader = new SpreaderOrderDefinition {
Security = security.CloneToMutable()
};
} else {
mdef.Limit = new LimitOrderDefinition();
}
}
CustomOrderDefinitionImmutable _fit_to_type(CustomOrderDefinitionImmutable defi, SecurityCriteriaClause security_clause, IEnumerable<SecurityAlgoTypeSetImmutable> algo_sets) {
var mroot = defi.CloneToMutable();
if (_fit_to_type(mroot, security_clause, algo_sets)) {
defi = new CustomOrderDefinitionImmutable(mroot);
}
return defi;
}
CustomOrderDefinitionImmutable _fit_to_type(CustomOrderDefinitionImmutable defi, SecurityImmutable security, IEnumerable<SecurityAlgoTypeSetImmutable> algo_sets) {
var mroot = defi.CloneToMutable();
if (_fit_to_type(mroot, security, algo_sets)) {
defi = new CustomOrderDefinitionImmutable(mroot);
}
return defi;
}
bool _fit_to_type(CustomOrderDefinition mdef, SecurityCriteriaClause security_clause, IEnumerable<SecurityAlgoTypeSetImmutable> algo_sets) {
while (mdef.CustomCase == CustomOrderDefinition.CustomOneofCase.Composite)
mdef = mdef.Composite.Definition;
var sec_sets = algo_sets.Where(s => security_clause.is_subset_of(s.security_criteria));
if (mdef.CustomCase == CustomOrderDefinition.CustomOneofCase.Dynamic) {
if (!algo_sets.All(s => s.algo_type_set.dynamic_types.Contains(mdef.Dynamic.DynamicOrderType))) {
assign_fallback_algo(mdef, security_clause);
return true;
}
} else if (!sec_sets.All(s => s.algo_type_set.factory_types.Contains(mdef.CustomCase))) {
assign_fallback_algo(mdef, security_clause);
return true;
}
return OrderDataHelper.get_child_algo_data(mdef).Count(cad => _fit_to_type(cad.get_child(mdef), security_clause, algo_sets)) > 0;
}
bool _fit_to_type(CustomOrderDefinition mdef, SecurityImmutable security, IEnumerable<SecurityAlgoTypeSetImmutable> algo_sets) {
while (mdef.CustomCase == CustomOrderDefinition.CustomOneofCase.Composite)
mdef = mdef.Composite.Definition;
var sec_sets = algo_sets.Where(s => s.security_criteria.includes(security));
if (mdef.CustomCase == CustomOrderDefinition.CustomOneofCase.Dynamic) {
if (!sec_sets.All(s => s.algo_type_set.dynamic_types.Contains(mdef.Dynamic.DynamicOrderType))) {
assign_fallback_algo(mdef, security);
return true;
}
} else if (!sec_sets.All(s => s.algo_type_set.factory_types.Contains(mdef.CustomCase))) {
assign_fallback_algo(mdef, security);
return true;
}
return OrderDataHelper.get_child_algo_data(mdef).Count(cad => _fit_to_type(cad.get_child(mdef), security, algo_sets)) > 0;
}
}
class AttachableAlgoProvider : AlgoProvider, IAttachableAlgoProvider {
public AttachableAlgoProvider(IEnumerable<SecurityAlgoTypeSetImmutable> algo_sets, IAlgoStudioModel studio_model, DynamicOrderHelper dynamic_helper, ISecurityAuxiliaryDataModel security_auxiliary_model)
: base(algo_sets, studio_model, dynamic_helper, security_auxiliary_model) { }
// arbitrarily assign LimitOrderDefinition (this is just a placeholder; the real child is the attach target algo).
// Note: although 'None' might seem approrpriate, the server requires a valid definition (perhaps this will change in the future).
protected override void assign_fallback_algo(CustomOrderDefinition mdef, SecurityCriteriaClause security_clause) =>
mdef.Limit = new LimitOrderDefinition();
// arbitrarily assign LimitOrderDefinition (this is just a placeholder; the real child is the attach target algo).
// Note: although 'None' might seem approrpriate, the server requires a valid definition (perhaps this will change in the future).
protected override void assign_fallback_algo(CustomOrderDefinition mdef, SecurityImmutable security) =>
mdef.Limit = new LimitOrderDefinition();
}
static class AlgoProviderExtensions {
// This stream filters out composite algos that are not subsets of the non-composite algo types in the table
// (normally, such composites are 'auto-fit,' meaning the algo nodes that are not part of the set are replaced with ones that are).
// For example, consider a composite algo the following structure: Offset -> GlbxIceberg.
// If an update arrives with these noncomposite types: "Iceberg, Limit, Market," that composite algo is not a subset of the non-composite algo types,
// because GlbxIceberg is not in the table, so it would only be added to the table if/when GlbxIceberg arrives.
public static IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_complete_algos(this IAlgoProvider algo_provider) =>
subscribe_complete_algos(algo_provider.subscribe_algos());
public static IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_complete_algos(IObservable<TableUpdate<AlgoData, AlgoKey>> source) =>
Observable.Create<TableUpdate<AlgoData, AlgoKey>>(o => {
var last_out = TableUpdate<AlgoData, AlgoKey>.mk_empty(AlgoData.key_fn);
var incomplete_composites = ImmutableHashSet<AlgoData>.Empty;
return source.Subscribe(table => {
var bd = last_out.to_builder();
foreach (var algo in incomplete_composites.Where(a => OrderDataHelper.enumerate_algo_structure(a.value).All(tup => _is_part_contained(tup, table.state)))) {
incomplete_composites = incomplete_composites.Remove(algo);
bd.add_or_update(algo);
}
foreach (var a in table.removed.Concat(table.changes.Old()))
bd.remove(a.key);
foreach (var a in table.added.Concat(table.changed)) {
var structure = OrderDataHelper.enumerate_algo_structure(a.value);
if (structure.All(t => _is_part_contained(t, table.state))) {
bd.add_or_update(a);
} else {
incomplete_composites = incomplete_composites.Add(a);
}
}
o.OnNext(last_out = bd.to_table());
});
});
static bool _is_part_contained((CustomOrderDefinition.CustomOneofCase type, string dyn, uint studio_id, uint spreader_sec) structure, IReadOnlyDictionary<AlgoKey, AlgoData> state) =>
structure.type == CustomOrderDefinition.CustomOneofCase.Composite ||
structure.type == CustomOrderDefinition.CustomOneofCase.Dynamic ?
state.ContainsKey(AlgoKey.from_dynamic_type(structure.dyn)) :
state.ContainsKey(AlgoKey.from_known_type(structure.type));
}
}
}
protected virtual void assign_fallback_algo(CustomOrderDefinition mdef, SecurityCriteriaClause security_clause) {
if (security_clause.security_types.empty_if_null().Contains(Security.ContractOneofCase.Synthetic.security_type())) {
mdef.Spreader = new SpreaderOrderDefinition();
} else {
mdef.Limit = new LimitOrderDefinition();
}
}
protected virtual void assign_fallback_algo(CustomOrderDefinition mdef, SecurityImmutable security) {
if (security.ContractCase == Security.ContractOneofCase.Synthetic) {
mdef.Spreader = new SpreaderOrderDefinition {
Security = security.CloneToMutable()
};
} else {
mdef.Limit = new LimitOrderDefinition();
}
}
CustomOrderDefinitionImmutable _fit_to_type(CustomOrderDefinitionImmutable defi, SecurityCriteriaClause security_clause, IEnumerable<SecurityAlgoTypeSetImmutable> algo_sets) {
var mroot = defi.CloneToMutable();
if (_fit_to_type(mroot, security_clause, algo_sets)) {
defi = new CustomOrderDefinitionImmutable(mroot);
}
return defi;
}
CustomOrderDefinitionImmutable _fit_to_type(CustomOrderDefinitionImmutable defi, SecurityImmutable security, IEnumerable<SecurityAlgoTypeSetImmutable> algo_sets) {
var mroot = defi.CloneToMutable();
if (_fit_to_type(mroot, security, algo_sets)) {
defi = new CustomOrderDefinitionImmutable(mroot);
}
return defi;
}
bool _fit_to_type(CustomOrderDefinition mdef, SecurityCriteriaClause security_clause, IEnumerable<SecurityAlgoTypeSetImmutable> algo_sets) {
while (mdef.CustomCase == CustomOrderDefinition.CustomOneofCase.Composite)
mdef = mdef.Composite.Definition;
var sec_sets = algo_sets.Where(s => security_clause.is_subset_of(s.security_criteria));
if (mdef.CustomCase == CustomOrderDefinition.CustomOneofCase.Dynamic) {
if (!algo_sets.All(s => s.algo_type_set.dynamic_types.Contains(mdef.Dynamic.DynamicOrderType))) {
assign_fallback_algo(mdef, security_clause);
return true;
}
} else if (!sec_sets.All(s => s.algo_type_set.factory_types.Contains(mdef.CustomCase))) {
assign_fallback_algo(mdef, security_clause);
return true;
}
return OrderDataHelper.get_child_algo_data(mdef).Count(cad => _fit_to_type(cad.get_child(mdef), security_clause, algo_sets)) > 0;
}
bool _fit_to_type(CustomOrderDefinition mdef, SecurityImmutable security, IEnumerable<SecurityAlgoTypeSetImmutable> algo_sets) {
while (mdef.CustomCase == CustomOrderDefinition.CustomOneofCase.Composite)
mdef = mdef.Composite.Definition;
var sec_sets = algo_sets.Where(s => s.security_criteria.includes(security));
if (mdef.CustomCase == CustomOrderDefinition.CustomOneofCase.Dynamic) {
if (!sec_sets.All(s => s.algo_type_set.dynamic_types.Contains(mdef.Dynamic.DynamicOrderType))) {
assign_fallback_algo(mdef, security);
return true;
}
} else if (!sec_sets.All(s => s.algo_type_set.factory_types.Contains(mdef.CustomCase))) {
assign_fallback_algo(mdef, security);
return true;
}
return OrderDataHelper.get_child_algo_data(mdef).Count(cad => _fit_to_type(cad.get_child(mdef), security, algo_sets)) > 0;
}
}
class AttachableAlgoProvider : AlgoProvider, IAttachableAlgoProvider {
public AttachableAlgoProvider(IEnumerable<SecurityAlgoTypeSetImmutable> algo_sets, IAlgoStudioModel studio_model, DynamicOrderHelper dynamic_helper, ISecurityAuxiliaryDataModel security_auxiliary_model)
: base(algo_sets, studio_model, dynamic_helper, security_auxiliary_model) { }
// arbitrarily assign LimitOrderDefinition (this is just a placeholder; the real child is the attach target algo).
// Note: although 'None' might seem approrpriate, the server requires a valid definition (perhaps this will change in the future).
protected override void assign_fallback_algo(CustomOrderDefinition mdef, SecurityCriteriaClause security_clause) =>
mdef.Limit = new LimitOrderDefinition();
// arbitrarily assign LimitOrderDefinition (this is just a placeholder; the real child is the attach target algo).
// Note: although 'None' might seem approrpriate, the server requires a valid definition (perhaps this will change in the future).
protected override void assign_fallback_algo(CustomOrderDefinition mdef, SecurityImmutable security) =>
mdef.Limit = new LimitOrderDefinition();
}
static class AlgoProviderExtensions {
// This stream filters out composite algos that are not subsets of the non-composite algo types in the table
// (normally, such composites are 'auto-fit,' meaning the algo nodes that are not part of the set are replaced with ones that are).
// For example, consider a composite algo the following structure: Offset -> GlbxIceberg.
// If an update arrives with these noncomposite types: "Iceberg, Limit, Market," that composite algo is not a subset of the non-composite algo types,
// because GlbxIceberg is not in the table, so it would only be added to the table if/when GlbxIceberg arrives.
public static IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_complete_algos(this IAlgoProvider algo_provider, bool allow_undefined_child = false) =>
subscribe_complete_algos(algo_provider.subscribe_algos(), allow_undefined_child);
public static IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_complete_algos(IObservable<TableUpdate<AlgoData, AlgoKey>> source, bool allow_undefined_child = false) =>
Observable.Create<TableUpdate<AlgoData, AlgoKey>>(o => {
var last_out = TableUpdate<AlgoData, AlgoKey>.mk_empty(AlgoData.key_fn);
var incomplete_composites = ImmutableHashSet<AlgoData>.Empty;
return source.Subscribe(table => {
var bd = last_out.to_builder();
foreach (var algo in incomplete_composites.Where(a => _validate_algo_structure(a.value, table.state, allow_undefined_child))) {
incomplete_composites = incomplete_composites.Remove(algo);
bd.add_or_update(algo);
}
foreach (var a in table.removed.Concat(table.changes.Old()))
bd.remove(a.key);
foreach (var a in table.added.Concat(table.changed)) {
if (_validate_algo_structure(a.value, table.state, allow_undefined_child)) {
bd.add_or_update(a);
} else {
incomplete_composites = incomplete_composites.Add(a);
}
}
o.OnNext(last_out = bd.to_table());
});
});
static bool _validate_algo_structure(CustomOrderDefinitionImmutable custom_def, IReadOnlyDictionary<AlgoKey, AlgoData> allowed_algo_parts, bool allow_undefined_child) =>
OrderDataHelper.enumerate_algo_structure(custom_def)
.Select((algo_node, index) =>
// None is an acceptable algo type as long as it's not the root algo (i.e., index > 0)
(index > 0 && allow_undefined_child && algo_node.type == CustomOrderDefinition.CustomOneofCase.None) || _is_part_contained(algo_node, allowed_algo_parts))
.All();
static bool _is_part_contained((CustomOrderDefinition.CustomOneofCase type, string dyn, uint studio_id, uint spreader_sec) structure, IReadOnlyDictionary<AlgoKey, AlgoData> state) =>
structure.type == CustomOrderDefinition.CustomOneofCase.Composite ||
structure.type == CustomOrderDefinition.CustomOneofCase.Dynamic ?
state.ContainsKey(AlgoKey.from_dynamic_type(structure.dyn)) :
state.ContainsKey(AlgoKey.from_known_type(structure.type));
}
}
@@ -73,7 +73,7 @@ namespace mbts.spark {
_revert_cmd_tooltip = backing(nameof(revert_cmd_tooltip), get_revert_tooltip);
//
_dispose = new CompositeDisposable(
_model.subscribe_complete_algos().OnDispatcher().Subscribe(_definitions_update_buffer),
_model.subscribe_complete_algos(allow_undefined_child: true).OnDispatcher().Subscribe(_definitions_update_buffer),
_definitions_update_buffer.OnDispatcher()
.SubscribeEach(u => {
u.FilterIncremental(a => a.key.is_composite, out var added, out var changes, out var removed);
@@ -144,7 +144,7 @@ namespace mbts.spark {
string candidate = string.Empty;
if (a.HasValue) {
var algo_name = a.Value.name;
var words = algo_name.Split(delim);
var words = algo_name.Split(delim, StringSplitOptions.RemoveEmptyEntries);
if (words.Length == 1) {
var single_word = words[0];
if (single_word.Length > 1) {
@@ -153,7 +153,7 @@ namespace mbts.spark {
candidate = single_word;
}
} else {
candidate = words.Take(2).Select(w => w.Substring(0, 1)).list_string(string.Empty);
candidate = string.Join(string.Empty, words.Take(2).Select(w => w.Substring(0, 1)));
}
}
return candidate.ToUpper();
@@ -398,7 +398,8 @@ namespace mbts.spark {
public uint last_column_width { get => _last_column_width.Value; set => _last_column_width.set(value); }
public uint piq_column_width { get => _piq_column_width.Value; set => _piq_column_width.set(value); }
public uint volume_column_width { get => _volume_column_width.Value; set => _volume_column_width.set(value); }
public uint alternate_order_size_from_settings => current_settings.alternate_order_size;
public uint actual_alternate_order_size =>
(current_settings?.alternate_order_size ?? 0) * Math.Max(1, display_quantity_divisor);
public bool show_warn_on_cross_from_settings => current_settings.show_warn_on_cross;
public bool enforce_max_clip_from_settings => current_settings.enforce_max_clip;
public uint max_clip_from_settings => current_settings.max_clip;
@@ -489,11 +490,11 @@ namespace mbts.spark {
public Task send_order_async((double price, Side? side, MouseButton button) args) {
if (order_properties_vm != null && args.button != MouseButton.Middle) {
var size_from_alternate = args.button == MouseButton.Right;
if (!DoubleHelper.IsNaNOrInfinity(args.price) && args.side.HasValue && !HasErrors && !order_properties_vm.HasErrors && (!size_from_alternate || current_settings.alternate_order_size > 0)) {
if (!DoubleHelper.IsNaNOrInfinity(args.price) && args.side.HasValue && !HasErrors && !order_properties_vm.HasErrors && (!size_from_alternate || actual_alternate_order_size > 0)) {
if (!size_from_alternate && (!size_calculator.value.HasValue || size_calculator.value.Value == 0)) {
size_calculator.flash_error();
} else {
uint size = size_from_alternate ? current_settings.alternate_order_size : (uint)Math.Abs(size_calculator.value ?? 0);
uint size = size_from_alternate ? actual_alternate_order_size : (uint)Math.Abs(size_calculator.value ?? 0);
// enforce max clip
if (enforce_max_clip_from_settings && size > max_clip_from_settings) {
size = max_clip_from_settings;
@@ -353,7 +353,9 @@ namespace mbts.spark {
return;
}
} else if (_vm != null) {
uint size = e.ChangedButton == MouseButton.Right ? _vm.alternate_order_size_from_settings : _vm.size_calculator.value ?? 0;
uint size = e.ChangedButton == MouseButton.Right ?
_vm.actual_alternate_order_size :
_vm.size_calculator.value ?? 0;
// does the size violate the local max clip
if (_vm.enforce_max_clip_from_settings && size > _vm.max_clip_from_settings) {
@@ -622,9 +622,7 @@ namespace bts.shell.pnl {
static class PNLDataExtensions {
public static string pnl_symbol(this SecurityImmutable security) {
if(security.HasIce)
return PNLICEHelper.get_symbol(security);
return security.Symbol;
return bts.api.ExchangeSpecificUitlities.get_pnl_symbol(security);
}
// while override_cpp could use double.NaN to indicate the lack of an override,
@@ -636,7 +634,7 @@ namespace bts.shell.pnl {
public static (double default_cpp, double? override_cpp) pnl_cpp_tuple(this PNLUnitImmutable pnl_unit, SecurityImmutable security) {
// TODO: actual impl
double calced_pnl = PNLICEHelper.get_cpp(pnl_unit,security);
double calced_pnl = bts.api.ExchangeSpecificUitlities.get_cpp(pnl_unit,security);
return (calced_pnl, pnl_unit.HasCppOverride ? pnl_unit.CppOverride.Value : (double?)null);
}
}
@@ -1,75 +0,0 @@
using bts.api;
using bts.api.orderman;
using bts.api.position;
using bts.utils;
using mbts;
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
namespace bts.shell.pnl {
static class PNLICEHelper{
public static bool is_spread(SecurityImmutable sec){
return sec.HasSpread && sec.Ice.StripName.Contains("/");
}
public static bool is_strip(SecurityImmutable sec){
return sec.HasSpread && !is_spread(sec);
}
public static bool is_ice_lots(SecurityImmutable sec){
return sec.Ice.MarketDescription.Contains("ICE Lots");
}
public static double get_future_cpp (SecurityImmutable sec){
if(sec.Symbol == "HHD")
return 1;
if(is_ice_lots(sec))
return 2500;
else{
var days_in_month = DateTime.DaysInMonth(Convert.ToInt32(sec.Future.ExpirationYear),
Convert.ToInt32(sec.Future.ExpirationMonth));
return days_in_month; // because the size is multiplied by 2500 already
}
}
public static double get_spread_cpp(SecurityImmutable sec){
return 0;
}
public static double get_strip_cpp(PNLUnitImmutable pnl,
SecurityImmutable sec){
double rv = 0;
foreach (var l in sec.Spread.Legs)
if(l.Security.HasFuture)
rv += get_cpp(pnl,l.Security);
return rv;
}
public static string get_symbol (SecurityImmutable sec){
var rv = String.Format("{0}{1}{2}",
sec.Symbol,
is_ice_lots(sec)? " ICE LOTS" : String.Empty,
is_spread(sec)? " Spread" : String.Empty);
return rv;
}
public static double get_cpp (PNLUnitImmutable pnl,
SecurityImmutable sec){
if (sec.HasGlbx)
return pnl.CashValue;
else if (sec.HasIce){
if(sec.HasFuture)
return get_future_cpp(sec);
else if(is_spread(sec))
return get_spread_cpp(sec);
else if(is_strip(sec))
return get_strip_cpp(pnl,sec);
}
return Double.NaN;
}
}}
+2 -2
View File
@@ -337,8 +337,8 @@ namespace mbts.spark {
public IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_algos(SecurityImmutable security, bool attachable_only = false) =>
(attachable_only ? _attachable_algo_provider : _algo_provider).subscribe_algos(security);
public IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_complete_algos(bool attachable_only = false) =>
(attachable_only ? _attachable_algo_provider : _algo_provider).subscribe_complete_algos();
public IObservable<TableUpdate<AlgoData, AlgoKey>> subscribe_complete_algos(bool attachable_only = false, bool allow_undefined_child = false) =>
(attachable_only ? _attachable_algo_provider : _algo_provider).subscribe_complete_algos(allow_undefined_child);
#endregion
#region Input
@@ -259,6 +259,7 @@ namespace mbts.co2 {
}
}
// If Size is wrong remember to update from
[StructLayout(LayoutKind.Sequential, Size = 704)]
internal struct ColdEntry {
public int exchange_status;
@@ -0,0 +1,128 @@
using bts.api;
using bts.api.orderman;
using bts.api.position;
using bts.utils;
using mbts;
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
namespace bts.api {
static class ExchangeSpecificUitlities{
private static string _default_hub = "Henry";
static class ICE{
#region Contract Type
public static bool is_spread(SecurityImmutable sec){
return sec.HasSpread && sec.Ice.StripName.Contains("/");
}
public static bool is_strip(SecurityImmutable sec){
return sec.HasSpread && !is_spread(sec);
}
public static bool is_ice_lots(SecurityImmutable sec){
return sec.Ice.MarketDescription.Contains("ICE Lots");
}
public static bool is_tas(SecurityImmutable sec){
return sec.Ice.MarketDescription.Contains("TAS");
}
#endregion
#region CPP
public static double get_future_cpp (SecurityImmutable sec){
if(sec.Symbol == "HHD")
// only futures in HHD are the dailies, everything
// else is a strip or spread
return 1;
if(is_ice_lots(sec))
return 2500;
else{
var days_in_month = DateTime.DaysInMonth(Convert.ToInt32(sec.Future.ExpirationYear),
Convert.ToInt32(sec.Future.ExpirationMonth));
return days_in_month; // because the size is multiplied by 2500 already
}
}
public static double get_spread_cpp(SecurityImmutable sec){
return 0;
}
public static double get_strip_cpp(PNLUnitImmutable pnl,
SecurityImmutable sec){
// add up all the legs
double rv = 0;
foreach (var l in sec.Spread.Legs)
if(l.Security.HasFuture)
rv += get_cpp(pnl,l.Security);
return rv;
}
public static double get_cpp (PNLUnitImmutable pnl,
SecurityImmutable sec){
if(sec.HasFuture)
return get_future_cpp(sec);
else if(is_spread(sec))
return get_spread_cpp(sec);
else if(is_strip(sec))
return get_strip_cpp(pnl,sec);
return Double.NaN;
}
#endregion
#region Symbol
public static string get_pnl_symbol (SecurityImmutable sec){
// for pnl purposes, for a symbol, the division is :
// Futures and Strips
// ICE Lots
// Spreads
var rv = String.Format("{0}{1}{2}",
sec.Symbol,
is_ice_lots(sec)? " ICE LOTS" : String.Empty,
is_spread(sec)? " Spread" : String.Empty);
return rv;
}
#endregion
#region Formatting
public static string format_security (SecurityImmutable? security){
if(security.HasValue && security.Value.HasIce){
bool ice_lots = is_ice_lots(security.Value),
tas = is_tas(security.Value);
return String.Format("{0}{1}{2}{3}{4}",
security?.Symbol,
ice_lots? " ICE Lots" : String.Empty,
tas ? " TAS" : String.Empty,
String.Compare(security.Value.Ice.HubAlias, _default_hub,true)==0 ? String.Empty :
" " + security.Value.Ice.HubAlias,
" " + security?.Ice.StripName);
}
return String.Empty;
}
#endregion
}
public static double get_cpp (PNLUnitImmutable pnl,
SecurityImmutable sec){
if(sec.HasIce)
return ICE.get_cpp(pnl,sec);
return pnl.CashValue;
}
public static string get_pnl_symbol (SecurityImmutable sec){
if(sec.HasIce)
return ICE.get_pnl_symbol(sec);
return sec.Symbol;
}
public static string format_security(SecurityImmutable? security){
if(security?.HasIce == true)
return ICE.format_security(security);
return security?.ToString();
}
}}
@@ -41,7 +41,7 @@ namespace bts.api {
switch (security?.ContractCase) {
case ContractOneofCase.Future:
if(security?.HasIce == true)
return _format_ICE(security);
return bts.api.ExchangeSpecificUitlities.format_security(security);
return format_future(security?.Symbol, security?.security_desc(), security.Value.Future);
case ContractOneofCase.Spread:
if (security?.HasGlbx == true) {
@@ -50,7 +50,7 @@ namespace bts.api {
return security?.Glbx.SecurityDesc;
}
else if(security?.HasIce == true)
return _format_ICE(security);
return bts.api.ExchangeSpecificUitlities.format_security(security);
return string.Join("", _format_future_legs(SecurityHelper.get_flat_future_legs(security?.Spread.Legs)));
case ContractOneofCase.Synthetic: return security?.Synthetic.Name;
default: return "Unknown-security";
@@ -84,28 +84,5 @@ namespace bts.api {
default: return $"{fl.quantity:+0;-0}{sym_exp_name}";
}
}
static string _format_ICE (SecurityImmutable? security){
string symbol = security?.Symbol,
strip = security?.Ice.StripName,
md_desc = security?.Ice.MarketDescription,
type_append = "";
if(md_desc.Contains("25k ICE Lots")){
if(md_desc.Contains("TAS"))
type_append = "TAS";
}
else if(md_desc.Contains("ICE Lots")){
if(md_desc.Contains("TAS"))
type_append = "ICE Lots TAS";
else
type_append = "ICE Lots";
}
else if(md_desc.Contains("TAS"))
type_append = "TAS";
type_append = string.IsNullOrEmpty(type_append) ?
" " : $" {type_append} ";
return $"{symbol}{type_append}{strip}";
}
}
}
+30 -19
View File
@@ -83,26 +83,37 @@ namespace bts.api {
}
public static int Compare(SecurityImmutable x, SecurityImmutable y) {
if (x.ContractCase == ContractOneofCase.Future) {
return y.ContractCase == ContractOneofCase.Future ? Compare(x.Symbol, x.Future, y.Symbol, y.Future) : -1;
} else if (y.ContractCase == ContractOneofCase.Future) {
return 1;
int c = non_deterministic_impl();
if (c == 0) {
c = x.ToString().CompareTo(y.ToString());
if (c == 0)
c = x.Id.CompareTo(y.Id);
}
var xlegs = get_flat_future_legs(x).ToList();
var ylegs = get_flat_future_legs(y).ToList();
int max = Math.Min(xlegs.Count, ylegs.Count);
for (int i = 0; i < max; i++) {
int rv = Compare(xlegs[i].symbol, xlegs[i].future, ylegs[i].symbol, ylegs[i].future);
if (rv != 0)
return rv;
rv = xlegs[i].quantity.CompareTo(ylegs[i].quantity);
if (rv != 0)
return rv;
rv = xlegs[i].exchange.CompareTo(ylegs[i].exchange);
if (rv != 0)
return rv;
return c;
// this has been observed to be non-determinisitc with certain ICE securities
// (it is determenistic with other exchanges..as far as we know).
int non_deterministic_impl() {
if (x.ContractCase == ContractOneofCase.Future) {
return y.ContractCase == ContractOneofCase.Future ? Compare(x.Symbol, x.Future, y.Symbol, y.Future) : -1;
} else if (y.ContractCase == ContractOneofCase.Future) {
return 1;
}
var xlegs = get_flat_future_legs(x).ToList();
var ylegs = get_flat_future_legs(y).ToList();
int max = Math.Min(xlegs.Count, ylegs.Count);
for (int i = 0; i < max; i++) {
int rv = Compare(xlegs[i].symbol, xlegs[i].future, ylegs[i].symbol, ylegs[i].future);
if (rv != 0)
return rv;
rv = xlegs[i].quantity.CompareTo(ylegs[i].quantity);
if (rv != 0)
return rv;
rv = xlegs[i].exchange.CompareTo(ylegs[i].exchange);
if (rv != 0)
return rv;
}
return xlegs.Count.CompareTo(ylegs.Count);
}
return xlegs.Count.CompareTo(ylegs.Count);
}
public static int Compare(string x_sym, FutureContract x, string y_sym, FutureContract y) {
@@ -116,4 +127,4 @@ namespace bts.api {
}
}
}
}
}
+1
View File
@@ -36,6 +36,7 @@ namespace bts {
Plugin,
Sdk,
Http,
iLink3
// Please keep this list in sync with logging.h/logging.cpp
}
@@ -7,6 +7,22 @@ using System.Reactive.Linq;
namespace System.Collections.Generic {
public static class EnumerableExtensions {
/// <summary>
/// Determines whether all elements of a sequence are true.
/// </summary>
/// <param name="source">An <see cref="IEnumerable{bool}"/> that contains the elements to which to apply the predicate.</param>
/// <returns>True if every element of the source sequence is true, or if the sequence is empty; otherwise false.</returns>
/// <exception cref="ArgumentNullException"><paramref name="source"/> is null</exception>
public static bool All(this IEnumerable<bool> source) {
if (source is null)
throw new ArgumentNullException(nameof(source));
foreach (var v in source) {
if (!v)
return false;
}
return true;
}
/// <summary>
/// Similar to <see cref="Enumerable.All{TSource}(IEnumerable{TSource}, Func{TSource, bool})"/>, but returns false if the <paramref name="source"/> is empty.
/// </summary>
+24 -13
View File
@@ -1,4 +1,5 @@
using bts.api;
using bts.collections;
using bts.resources;
using bts.test;
using bts.utils.proto;
@@ -10,7 +11,6 @@ namespace bts.mcore {
class SecurityTests {
[Test]
public void test_security_equal() {
SecurityImmutable a = new SecurityImmutable(new Security() { Id = 42 });
SecurityImmutable b = new SecurityImmutable(new Security() { Id = 42 });
SecurityImmutable c = new SecurityImmutable(new Security() { Id = 4 });
@@ -29,20 +29,31 @@ namespace bts.mcore {
[Test]
public void test_security_compare() {
const string securities_resource = "KrandallOrderBookCrash.security_proto";
// verify that the sample size doesn't shrink for some reason.
// if it grows a bit that's fine; the generic comparison test
// asserts against collections that are too large.
const int min_sample_size = 91;
foreach (var file_path in SecurityFiles.all())
_test_security_compare(_read_securities(file_path));
}
var securities = StreamHelper.ReadProtos<Security>(
ResourceHelper.LoadResource(securities_resource))
.Select(s => new SecurityImmutable(s))
.ToList();
Assert.IsTrue(securities.Count >= min_sample_size);
void _test_security_compare(IEnumerable<SecurityImmutable> securities) {
const int max_count = 2048;
var sec_list = securities as IList<SecurityImmutable> ?? securities.ToList();
if (sec_list.Count > max_count)
sec_list = sec_list.WrapSubRange(0, max_count);
ComparisonTests.transitive_comparison(sec_list, SecurityHelper.Comparer.InstanceImmutable);
ComparisonTests.identity_comparison(sec_list, SecurityHelper.Comparer.InstanceImmutable);
}
ComparisonTests.transitive_comparison(securities, SecurityHelper.Comparer.InstanceImmutable);
ComparisonTests.identity_comparison(securities, SecurityHelper.Comparer.InstanceImmutable);
IEnumerable<SecurityImmutable> _read_securities(string file_path) =>
StreamHelper.ReadProtos<Security>(ResourceHelper.LoadResource(file_path)).Select(s => new SecurityImmutable(s));
static class SecurityFiles {
public const string
KrandallOrderBookCrash = "KrandallOrderBookCrash.security_proto",
NonDeterministicICEFutures = "NonDeterministicICEFutures.security_proto";
public static IEnumerable<string> all() {
yield return KrandallOrderBookCrash;
yield return NonDeterministicICEFutures;
}
}
}
}
@@ -4,14 +4,14 @@ using System.Linq;
namespace bts.mcore {
static class ComparisonTests {
public static void transitive_comparison<T>(IEnumerable<T> items, IComparer<T> comparer) {
// Verify that the sample size isn't too big.
// This test compares each item to every other item
// (technically twice; lhs/rhs is flipped the second time),
// so performance is O(n^2).
const int max_sample_size = 4092;
/// <summary>
/// Compares every permutation of pairs in the specified items.
/// Be wary of use with large collections, as performance is O(n^2).
/// </summary>
public static void transitive_comparison<T>(IEnumerable<T> items, IComparer<T> comparer, int min_item_count = 2, int max_item_count = 5000) {
var sorted_items = items.ToList();
Assert.That(sorted_items.Count <= max_sample_size);
Assert.That(sorted_items.Count >= min_item_count);
Assert.That(sorted_items.Count <= max_item_count);
sorted_items.Sort(comparer);
for (int i = 0; i < sorted_items.Count - 1; i++) {
for (int j = i + 1; j < sorted_items.Count; j++) {
+1
View File
@@ -16,5 +16,6 @@
</ItemGroup>
<ItemGroup>
<EmbeddedResource Include="Resources\KrandallOrderBookCrash.security_proto" />
<EmbeddedResource Include="Resources\NonDeterministicICEFutures.security_proto" />
</ItemGroup>
</Project>
+1 -1
View File
@@ -1 +1 @@
set(sources files/cert_secure_logon.txt;files/futures.coconut.xz;files/onixs_ice/ConnectivityConfiguration.APITest.xml;files/onixs_ice/ConnectivityConfiguration.PerfTest.xml;files/onixs_ice/ConnectivityConfiguration.Production.xml;files/perf_trace/compiled.css;files/perf_trace/index.html;files/perf_trace/jquery-3.3.1.min.js;files/perf_trace/main.js;files/perf_trace/perf_log.sql;files/perf_trace/perf_trace_analysis.sql;files/perf_trace/plotly-latest.min.js)
set(sources files/audit_ilink3.py;files/audit_upload.sh;files/audit_uploader.py;files/cert_secure_logon.txt;files/example.conf;files/futures.coconut.xz;files/onixs_ice/ConnectivityConfiguration.APITest.xml;files/onixs_ice/ConnectivityConfiguration.APITest.xml.orig;files/onixs_ice/ConnectivityConfiguration.PerfTest.xml;files/onixs_ice/ConnectivityConfiguration.Production.xml;files/perf_trace/compiled.css;files/perf_trace/index.html;files/perf_trace/jquery-3.3.1.min.js;files/perf_trace/main.js;files/perf_trace/perf_log.sql;files/perf_trace/perf_trace_analysis.sql;files/perf_trace/plotly-latest.min.js)
+405
View File
@@ -0,0 +1,405 @@
import json
import csv
import datetime, time
import argparse, os
def valid_file(f):
if not os.path.exists(f):
# Argparse uses the ArgumentTypeError to give a rejection message like:
# error: argument input: x does not exist
raise argparse.ArgumentTypeError("{0} does not exist".format(f))
return f
def valid_date(s):
try:
return datetime.datetime.strptime(s, "%Y%m%d")
except ValueError:
raise argparse.ArgumentTypeError("Not a valid date: '{0}'.".format(s))
def print_headers():
# From https://www.cmegroup.com/confluence/display/EPICSANDBOX/iLink+3+-+Minimum+Acceptable+Audit+Trail+Elements+-+Data+Definitions+and+Validation+Rules
line_to_write = "Sending Timestamps,Message Direction,Operator ID,Session ID,Manual Order Indicator,Message Type,Instrument,Order Flow ID,\
CME Globex Order ID,Client Order ID,Order Request ID,Buy/Sell Indicator,Quantity,Limit Price,Stop Price,Order Type,Order Qualifier,IFM Flag,\
Display Quantity,Minimum Quantity,Country of Origin,Fill Price,Fill Quantity,Cumulative Quantity,Remaining Quantity,Aggressor Flag,Source of Cancellation,\
Reject Reason,CME Globex Message ID,Cross ID,Quote Request ID,Message Quote ID,Quote Set ID,Quote Entry ID,Bid Price,Bid Size,Offer Price,Offer Size,\
Processed Entries,Party Details List Request ID,ListUpdateAction,Self-Matching Prevention ID,Self-Match Prevention Instr,Customer Type Indicator,\
AvgPxGroupID,Average Px Ind,CmtaGiveUpCD,Origin,Clearing Trade Price Type,Customer Handling Instr,Executing Firm ID,Account Number,Take Up Firm,Take up Account"
with open (args.output_file, 'w') as the_file:
the_file.write(line_to_write)
the_file.write("\n")
def set_blanks():
return ""
def get_value(json_key):
prices = ["Price", "LastPx", "StopPx"]
if json_key in json_data:
if json_key in prices:
x = json_data[json_key]
return x['mantissa']/100000000000
else:
return json_data[json_key]
else:
return ""
# Column 1 - Sending Timestamps
def SendingTimeEpoch(epochtime): #1
#TIME = time.strftime('%Y%m%d-%H:%M:%S.{}'.format(epochtime/1000000000), time.gmtime(epochtime/1000000000))
TIME = datetime.datetime.fromtimestamp(int(epochtime) / 1e9).strftime('%Y%m%d-%H:%M:%S.%f')
return TIME
# Column 2 - Message Direction
def direction(whichway):
if whichway == '<':
DIRECTION = "FROM CME"
else:
DIRECTION = "TO CME"
return DIRECTION
# Column 3 - Operator ID
def SenderID(sender):
if "SenderID" in json_data:
SENDER = json_data["SenderID"]
return SENDER
else:
return SESSIONID
# Column 4 - Session ID
# Column 5 - Manual Order Indicator
def ManualOrderIndicator(manual):
if "ManualOrderIndicator" in json_data:
if manual == "Manual":
return "1"
else:
return "0"
# Column 6 - Message Type
def MessageType(msg_type,orderstatus):
if msg_type == "NewOrderSingle514":
return "D"
elif msg_type == "ExecutionReportNew522":
return "8--0"
if "ExecutionReportTrade" in msg_type:
if orderstatus == "PartiallyFilled":
return "8--1"
else:
return "8--2"
elif msg_type == "ExecutionReportCancel534":
return "8--4"
elif msg_type == "ExecutionReportModify531":
return "8--5"
elif msg_type == "ExecutionReportCancel534":
return "8--H"
elif msg_type == "ExecutionReportReject523":
return "8--8"
elif msg_type == "OrderCancelReplaceReject536":
return "9--2"
elif msg_type == "OrderCancelReject535":
return "9--1"
elif msg_type == "PartyDetailsDefinitionRequest518":
return "CX"
elif msg_type == "PartyDetailsDefinitionRequestAck519":
return "CY"
elif msg_type == "OrderCancelRequest516":
return "F"
elif msg_type == "OrderCancelReplaceRequest515":
return "G"
# Column 12 - Buy-Sell Indicator/Side
def Side(indicator):
if "Side" in json_data:
if indicator == "Buy":
return "1"
else:
return "2"
# Column 16 - Order Type/OrdType
def OrdType(ordertype):
if "OrdType" in json_data:
if ordertype == "MarketWithProtection" or ordertype == "MarketwithProtection":
return "1"
elif ordertype == "Limit":
return "2"
elif ordertype == "StopWithProtection" or ordertype == "StopwithProtection":
return "3"
elif ordertype == "StopLimit":
return "4"
elif ordertype == "MarketLimit":
return "K"
else:
return ""
# Column 17 - Order Qualifier/TimeInForce
def TimeInForce(orderqual):
if "TimeInForce" in json_data:
if orderqual == "Day":
return "0"
elif orderqual == "GTC":
return "1"
elif orderqual == "FAK":
return "3"
elif orderqual == "GTD":
return "6"
else:
return ""
# Column 18 IFM Flag/OFMOverride
def OFMOverride(imfflag):
if "OFMOverride" in json_data:
if imfflag == "Enabled":
return "1"
else:
return "0"
# Column 19 Display Quantity/DisplayQty
def DisplayQty(qty):
if "DisplayQty" in json_data:
if str(qty) == "0":
return ""
elif str(qty) == "4294967295":
return ""
else:
return qty
# Column 26 - Aggressor Flag/AggressorIndicator
def AggressorIndicator(aggressor):
if "AggressorIndicator" in json_data:
if aggressor == "True":
return "1"
else:
return "0"
# Column 27 - Source of Cancellation/ExecRestatementReason
def ExecRestatementReason(sourcecancel):
if "ExecRestatementReason" in json_data:
if sourcecancel == "Exchange":
return "8"
elif sourcecancel == "CancelledNotBest":
return "9"
elif sourcecancel == "CancelOnDisconnect":
return "100"
elif sourcecancel == "CancelRestingDueToSelfMatchPrevention":
return "103"
elif sourcecancel == "CancelFromCMEGlobexCreditControlsViolation":
return "104"
elif sourcecancel == "CancelFromCMEOne":
return "105"
elif sourcecancel == "CancelFromRiskManagementAPI":
return "106"
elif sourcecancel == "CancelAggressingDueToSelfMatchPrevention":
return "107"
elif sourcecancel == "CancelDueToVolatilityQuoteOptionLessThanMinOrderSize":
return "108"
else:
return ""
# Column 43 - Self-Match Prevention Instr/SelfMatchPreventionInstruction
def SelfMatchPreventionInstruction(instr):
if "SelfMatchPreventionInstruction" in json_data:
if instr == "CancelOldest":
return "O"
elif instr == "CancelNewest":
return "N"
else:
return ""
# Column 44 - Customer Type Indicator/CustOrderCapacity
def CustOrderCapacity(custord):
if "CustOrderCapacity" in json_data:
if custord == "NULL_VALUE":
return ""
elif custord == "Membertradingfortheirownaccount":
return "1"
elif custord == "ClearingFirmtradingforitsproprietaryaccount":
return "2"
elif custord == "Membertradingforanothermember":
return "3"
else:
return "4"
# Column 46 - Average Px Ind/AvgPxInd
def AvgPxInd(avgpxindicator):
if "AvgPxInd" in json_data:
if avgpxindicator == "NULL_VALUE":
return ""
else:
return avgpxindicator
# Column 48 - Origin/ClearingAccountType
def ClearingAccountType(accounttype):
if "ClearingAccountType" in json_data:
if accounttype == "Customer":
return "0"
elif accounttype == "Firm":
return "1"
else:
return ""
# Column 49 - Clearing Trade Price Type/ClearingTradePriceType
def ClearingTradePriceType(clrtrdprice):
if "ClearingTradePriceType" in json_data:
if clrtrdprice == "NULL_VALUE":
return "0"
else:
return "3"
# Column 50 - Customer Handling Instr/CustOrderHandlingInst
def CustOrderHandlingInst(custorderinst):
if "CustOrderHandlingInst" in json_data:
if custorderinst == "ClientElectronic":
return "C"
elif custorderinst == "TEST":
return "D"
else:
return ""
# Column 51 and 52 - Getting the Firm and Account Number
def partyid(partydetailid,column):
for x in partydetailid:
role = x['PartyDetailRole']
id = x['PartyDetailID']
if column == "51" and role == "ExecutingFirm":
return id
if column == "52" and role == "CustomerAccount":
return id
parser = argparse.ArgumentParser()
parser.add_argument("json_log", help="the json log[.gz] you wish to process", type=valid_file)
parser.add_argument("output_file", help="the destination csv file you wish to write")
parser.add_argument("-d", "--date", help="the date you wish to parse from the log, specified as YYYYmmdd",
default=datetime.datetime.today().strftime("%Y%m%d"), type=valid_date)
args = parser.parse_args()
print_headers()
list_types = ['NewOrderSingle514', 'OrderCancelReplaceRequest515', 'OrderCancelRequest516', 'PartyDetailsDefinitionRequest518', 'PartyDetailsDefinitionRequestAck519', \
'ExecutionReportNew522', 'ExecutionReportReject523', 'ExecutionReportTradeOutright525', 'ExecutionReportTradeSpread526', 'ExecutionReportTradeSpreadLeg527', \
'ExecutionReportModify531', 'ExecutionReportCancel534', 'OrderCancelReject535', 'OrderCancelReplaceReject536', ]
# Note that the timedeltas specificed in trade_day_start and trade_day_end
# are based on Chicago times; as such, this script should run on a machine
# set to a Chicago timezone. A future impromovement would be to determine
# the current machine's timezone and determine the appropriate offset
# versus Chicago time, but for now we assume this will run on a Chicago
# tz machine.
input_as_datetime = args.date
trade_day_start = input_as_datetime - datetime.timedelta(days=1) + datetime.timedelta(hours=16, minutes=30)
trade_day_utc_start = time.gmtime((time.mktime(trade_day_start.timetuple())))
trade_day_end = input_as_datetime + datetime.timedelta(hours=16)
trade_day_utc_end = time.gmtime((time.mktime(trade_day_end.timetuple())))
file_opener = None
if args.json_log.endswith(".gz"):
import gzip
file_opener = gzip.open
else:
file_opener = open
#inputFile = open(fileInput) #open json file
#outputFile = open(fileOutput, 'w') #load csv file
# Get Session
with file_opener(args.json_log) as json_file:
for line in json_file:
data = json.loads(line)
json_data = data['msg_json']
if "Session" in json_data:
SESSIONID = json_data["Session"]
#inputFile = open(fileInput) #open json file
#outputFile = open(fileOutput, 'w') #load csv file
with file_opener(args.json_log) as json_file:
for line in json_file:
data = json.loads(line)
line_timestamp = time.strptime(data['time'], "%Y%m%d-%H:%M:%S.%f")
if trade_day_utc_start <= line_timestamp <= trade_day_utc_end:
json_data = data['msg_json']
NAME = json_data["Name"]
if NAME in list_types:
EXECUTING_FIRM_ID = ""
ACCOUNT_NUMBER = ""
DIRECTION = direction(data['direction'])
# Set Blanks
blank_json = {'QuoteReqID': '', 'QuoteID': '', 'QuoteSetID': '', 'QuoteEntryID': '', 'BidPx': '', 'BidSize': '', 'OfferPx': '', \
'OfferSize': '', 'NoProcessedEntries': '', 'ListUpdateAction': '', 'MinQty': '', 'AvgPxGroupID': '', \
'CmtaGiveUpCD': '', 'TAKE_UP_FIRM': '', 'TAKE_UP_ACCOUNT': '',}
# No Data Processing
straight_json = {'SecurityID': '', 'ClOrdID': '', 'OrderID': '', 'OrderRequestID': '', 'OrderQty': '', \
'Location': '', 'LastQty': '', 'CumQty': '', 'LeavesQty': '', 'Text': '', 'SideTradeID': '', 'CrossID': '', 'SelfMatchPreventionID': '', \
'Price': '', 'StopPx': '', 'LastPx': '', 'PartyDetailsListReqID': '', 'OrdStatus': ''}
# Processing Data
process_json = {'SendingTimeEpoch': '', 'ManualOrderIndicator': '', 'SenderID': '', 'Side': '', 'OrdType': '', 'TimeInForce': '', \
'OFMOverride': '', 'AggressorIndicator': '', 'ExecRestatementReason': '', 'SelfMatchPreventionInstruction': '', 'CustOrderCapacity': '', \
'AvgPxInd': '', 'ClearingAccountType': '', 'ClearingTradePriceType': '', 'CustOrderHandlingInst': '', 'DisplayQty': ''}
for d in straight_json:
test = get_value(d)
if test != "None" or test != "":
if "," in str(test):
test = test.replace(",",";")
if "4294967295" in str(test):
test = ""
if "us;" in str(test):
test = test.upper()
if d == "SelfMatchPreventionID" and len(str(test)) > 8:
test = ""
straight_json[d] = test
else:
straight_json[d] = ''
for d in process_json:
if d in json_data:
function = locals()[d]
test = function(json_data[d])
if test != "None":
process_json[d] = test
else:
process_json[d] = ''
elif d == "SenderID":
function = locals()[d]
test = function(SenderID)
if test != "None":
process_json[d] = test
# Party Message Fields
# Column 51 - Executing Firm ID
if "NoPartyDetails" in json_data:
EXECUTING_FIRM_ID = partyid(json_data['NoPartyDetails'], "51")
# Column 52 - Party Account Number
ACCOUNT_NUMBER = partyid(json_data['NoPartyDetails'], "52")
line_to_write = process_json['SendingTimeEpoch'] + "," + DIRECTION + "," + process_json['SenderID'] + "," + SESSIONID + "," + process_json['ManualOrderIndicator'] + "," + \
MessageType(NAME,straight_json['OrdStatus']) + "," + str(straight_json['SecurityID']) + "," + str(straight_json['ClOrdID']) + "," + str(straight_json['OrderID']) + "," + str(straight_json['ClOrdID']) + "," + \
str(straight_json['OrderRequestID']) + "," + process_json['Side'] + "," + str(straight_json['OrderQty']) + "," + str(straight_json['Price']) + "," + \
str(straight_json['StopPx']) + "," + process_json['OrdType'] + "," + process_json['TimeInForce'] + "," + process_json['OFMOverride'] + "," + str(process_json['DisplayQty']) + "," + \
blank_json['MinQty'] + "," + straight_json['Location'] + "," + str(straight_json['LastPx']) + "," + str(straight_json['LastQty']) + "," + str(straight_json['CumQty']) + "," + \
str(straight_json['LeavesQty']) + "," + process_json['AggressorIndicator'] + "," + process_json['ExecRestatementReason'] + "," + str(straight_json['Text']) + "," + \
str(straight_json['SideTradeID']) + "," + str(straight_json['CrossID']) + "," + blank_json['QuoteReqID'] + "," + blank_json['QuoteID'] + "," + blank_json['QuoteSetID'] + "," + \
blank_json['QuoteEntryID'] + "," + blank_json['BidPx'] + "," + blank_json['BidSize'] + "," + blank_json['OfferPx'] + "," + blank_json['OfferSize'] + "," + \
blank_json['NoProcessedEntries'] + "," + str(straight_json['PartyDetailsListReqID']) + "," + blank_json['ListUpdateAction'] + "," + str(straight_json['SelfMatchPreventionID']) + "," + \
process_json['SelfMatchPreventionInstruction'] + "," + process_json['CustOrderCapacity'] + "," + blank_json['AvgPxGroupID'] + "," + process_json['AvgPxInd'] + "," + \
blank_json['CmtaGiveUpCD'] + "," + str(process_json['ClearingAccountType']) + "," + process_json['ClearingTradePriceType'] + "," + process_json['CustOrderHandlingInst'] + "," + \
EXECUTING_FIRM_ID + "," + str(ACCOUNT_NUMBER) + "," + blank_json['TAKE_UP_FIRM'] + "," + blank_json['TAKE_UP_ACCOUNT']
#print(line_to_write)
with open (args.output_file, 'a') as the_file:
the_file.write(line_to_write)
the_file.write("\n")
+73
View File
@@ -0,0 +1,73 @@
#/bin/bash
function help_and_exit()
{
echo
echo Usage: $0 [option]... config_file
echo
echo Can also be used in cron like the following
echo 40 16 * * 1-5 /usr/bin/docker exec spark_10000 bash /sparkbin/data_files/audit_upload.sh /sparkdata/data/audit_uploads/example.conf
echo example.conf included. The location of the conf file should be relative to the docker container, not the host
echo
echo Options are as follows:
echo " -d|--date YYYYMMDD Example is 20180731"
echo " Help: "
echo " -h|--help Display this message."
echo
exit 1
}
DATE=$(date +%Y%m%d)
ORIG_DIR=$(pwd)
ERROR_LEVEL="0"
while getopts "d:h" option; do
case $option in
d) DATE=$OPTARG ;;
h) help_and_exit;;
\?) echo "Invalid option: -$OPTARG" >&2;
exit 1;;
esac
done
shift $((OPTIND-1))
CONFIG_FILE=$1
declare -A conf_vars; while read line; do conf_vars[${line%=*}]=${line#*=}; done <$CONFIG_FILE
SESSION_ID=${conf_vars[session_id]}
FCM=${conf_vars[fcm]}
CLIENT=${conf_vars[client]}
cd /sparkbin/data_files
for SESSION in $(ls /sparkdata/data/ilink3/); do
if [[ $SESSION == *$SESSION_ID* ]]; then
echo "Session is $SESSION"
SESSION_DAY=$(date +%Y-%m-%d)
if [[ $SESSION == "cme_"* ]]; then
SESSION_DAY=$(date -d "$DATE -$(date -d $DATE +%u) days" +"%Y-%m-%d")
AUDIT_SCRIPT="audit_ilink3.py"
fi
echo $SESSION_DAY
if [[ ! -d /sparkdata/data/ilink3/$SESSION/audit_logs ]]; then
mkdir -p /sparkdata/data/ilink3/$SESSION/audit_logs
fi
if [[ -d /sparkdata/data/ilink3/$SESSION/$SESSION_DAY ]]; then
echo "Starting the creation of the audit log"
python $AUDIT_SCRIPT -d $DATE /sparkdata/data/ilink3/$SESSION/$SESSION_DAY/log /sparkdata/data/ilink3/$SESSION/audit_logs/"$FCM"_audit_"$SESSION"_"$DATE".csv
fi
if [[ ! -d /sparkdata/data/md_audit ]]; then
mkdir /sparkdata/data/md_audit
fi
if [[ -d /sparkdata/data/md_audit ]]; then
# Running the MD Audit csv generation
cd /sparkbin
./sparkd adminreport md > /sparkdata/data/md_audit/"$FCM"_md_"$CLIENT"_"$DATE".csv
cd /sparkbin/data_files
fi
echo "Uploading to the (s)FTP Server"
python audit_uploader.py --config_file="$CONFIG_FILE" --datestamp $DATE --session="$SESSION" --client="$CLIENT" --fcm="$FCM"
fi
done
+150
View File
@@ -0,0 +1,150 @@
import getopt, sys, datetime
import paramiko, ftplib
#########################################################
# Commandline Args #
#########################################################
def usage():
print "-------------------------------------------------------"
print "-Usage: -"
print "-------------------------------------------------------"
print "- --config_file= : Path to upload config file -"
print "- -if no path is specified look -"
print "- in the working directory. -"
print "- -"
print "- --help : Print this message -"
print "- -"
print "- --datestamp= : Optional, defaults to today -"
print "- -replaces *datestamp in the -"
print "- the upload config file -"
print "- -"
print "- --fcm : Optional, spark input -"
print "- -"
print "- --session : Optional, spark jf session -"
print "-------------------------------------------------------"
fcm = ''
session = ''
try:
options, arguments = getopt.getopt(sys.argv[1:],"h",['config_file=', 'datestamp=', 'help', 'fcm=', 'client=', 'session='])
except getopt.GetoptError, err:
print str(err)
usage()
sys.exit(2)
for option, value in options:
if option in ("-h","--h","-help","--help"):
usage()
sys.exit()
elif option in ("-config_file","--config_file"):
config_file = value
elif option in ("-datestamp","--datestamp"):
datestamp = value
elif option in ("-fcm","--fcm"):
fcm = value
elif option in ("-client","--client"):
client = value
elif option in ("-session","--session"):
session = value
for argument in arguments:
if argument in ("h", "help"):
usage()
sys.exit()
else:
print "Unrecognized commandline argument. Type help for usage."
sys.exit()
if len(options) <= 0:
print "Requires a config file to run."
usage()
sys.exit()
#########################################################
# Read Config Function #
#########################################################
def read_config_file(config_file, datestamp=datetime.datetime.today().strftime('%Y%m%d'), fcm=fcm, session=session):
'''
Parse our config file that gets passed in as a commandline parameter.
All config keys must be of the form key=value. White space is ignored.
All config keys must be seperated by a newline character.
Called during initialization.
'''
upload_config = {}
config_file = open(config_file, 'r')
config_file = config_file.read()
for config in config_file.split("\n"):
if '=' in config:
cleaned_key = config.split('=')[0].strip().upper()
cleaned_value = config.split('=')[1].strip()
if '*datestamp*' in cleaned_value:
cleaned_value = cleaned_value.replace('*datestamp*',datestamp)
if '*fcm*' in cleaned_value:
cleaned_value = cleaned_value.replace('*fcm*',fcm)
if '*client*' in cleaned_value:
cleaned_value = cleaned_value.replace('*client*',client)
if '*session*' in cleaned_value:
cleaned_value = cleaned_value.replace('*session*',session)
upload_config[cleaned_key] = cleaned_value
else:
pass
return upload_config
#########################################################
# FTP File Transfer Function #
#########################################################
def putfile_ftp(file, put_path, site, user, password, port=21, verbose=True):
if verbose: print 'Uploading', file
local = open(file, 'rb')
remote = ftplib.FTP(site)
remote.login(user, password)
remote.storbinary(put_path, local, 1024)
remote.quit()
local.close()
if verbose: print 'Upload done.'
#########################################################
# SFTP File Transfer Function #
#########################################################
def putfile_sftp(file, put_path, site, username, password, port=22, verbose=True):
transport = paramiko.Transport((site, port))
transport.connect(username = username, password = password)
sftp = paramiko.SFTPClient.from_transport(transport)
sftp.put(file, put_path)
sftp.close()
transport.close()
#########################################################
# Main #
#########################################################
#if datestamp != None:
# upload_config = read_config_file(config_file, datestamp)
#else:
# upload_config = read_config_file(config_file)
try:
datestamp
except NameError:
upload_config = read_config_file(config_file)
else:
upload_config = read_config_file(config_file, datestamp)
print upload_config
if upload_config["PROTOCOL_PUT"] in ['FTP','ftp']:
print "Protocol = FTP"
putfile_ftp(upload_config["GET_ILINK3_FILE_PATH"] + upload_config["GET_ILINK3_FILE_NAME"], upload_config["PUT_ILINK3_FILE_PATH"] + upload_config["PUT_ILINK3_FILE_NAME"], upload_config["PUT_HOST"], upload_config["USERNAME_PUT"], upload_config["PASSWORD_PUT"], upload_config["PORT_PUT"] )
putfile_ftp(upload_config["GET_MD_FILE_PATH"] + upload_config["GET_MD_FILE_NAME"], upload_config["PUT_MD_FILE_PATH"] + upload_config["PUT_MD_FILE_NAME"], upload_config["PUT_HOST"], upload_config["USERNAME_PUT"], upload_config["PASSWORD_PUT"], upload_config["PORT_PUT"] )
elif upload_config["PROTOCOL_PUT"] in ['SFTP','sftp']:
print "Protocol = SFTP"
putfile_sftp(upload_config["GET_ILINK3_FILE_PATH"] + upload_config["GET_ILINK3_FILE_NAME"], upload_config["PUT_ILINK3_FILE_PATH"] + upload_config["PUT_ILINK3_FILE_NAME"], upload_config["PUT_HOST"], upload_config["USERNAME_PUT"], upload_config["PASSWORD_PUT"], int(upload_config["PORT_PUT"]) )
putfile_sftp(upload_config["GET_MD_FILE_PATH"] + upload_config["GET_MD_FILE_NAME"], upload_config["PUT_MD_FILE_PATH"] + upload_config["PUT_MD_FILE_NAME"], upload_config["PUT_HOST"], upload_config["USERNAME_PUT"], upload_config["PASSWORD_PUT"], int(upload_config["PORT_PUT"]) )
+19 -12
View File
@@ -1,12 +1,19 @@
Session ID Access ID Secret Key Creation Date Expiration Date Environment
M4D lLtjH74y1Kh5XUJTLyH2 f1K_tROQp2lTIplVp34WxKa7xDUgfFnIteolfik061o 2019-09-03 2020-09-03 CERT
MC1 VEMlkDuLz74OVHSuacQB IuHyYT3b-9KUo3WsuXykt-fXsjzp7L-JOrdCSQZLNl0 2019-09-03 2020-09-03 CERT
MC5 77lzSUwPLlM9obaXh2mK z6aA9qfASDKfZkYTVKFqJd-78i0HZ2Q9ec3CKCYIqHg 2019-09-03 2020-09-03 CERT
MC7 RphzI3rn71oGUEiYnX9e y6OvvPPyoyj2S8YAnE1gFduXT8v29HIMjO5EngYxSNg 2019-09-03 2020-09-03 CERT
MC8 PYBIm2TI9gFPIxmHbuYl 6o-ZolGH2DdeQOlGtJ5Zqoh_kg7VdbxkmjeOHmigKwU 2019-09-03 2020-09-03 CERT
Session ID Access ID Secret Key Creation Date Expiration Date Environment
M4D lLtjH74y1Kh5XUJTLyH2 f1K_tROQp2lTIplVp34WxKa7xDUgfFnIteolfik061o 2019-09-03 2020-09-03 CERT
MC1 VEMlkDuLz74OVHSuacQB IuHyYT3b-9KUo3WsuXykt-fXsjzp7L-JOrdCSQZLNl0 2019-09-03 2020-09-03 CERT
MC5 77lzSUwPLlM9obaXh2mK z6aA9qfASDKfZkYTVKFqJd-78i0HZ2Q9ec3CKCYIqHg 2019-09-03 2020-09-03 CERT
MC7 RphzI3rn71oGUEiYnX9e y6OvvPPyoyj2S8YAnE1gFduXT8v29HIMjO5EngYxSNg 2019-09-03 2020-09-03 CERT
MC8 PYBIm2TI9gFPIxmHbuYl 6o-ZolGH2DdeQOlGtJ5Zqoh_kg7VdbxkmjeOHmigKwU 2019-09-03 2020-09-03 CERT
NGG RYcigz3bSLIIBBF2A2Hq qB1dm9VkTKeXcdQpFzla8B3_jwTgya_-Fk5iyEJvr2E 2020-06-15 2021-06-15 NEWRELEASE
NGF o4OTmmTLQ7Lv4g6bCm26 HbD3FneMyb9Fu-FkCDLscCczHqFcFE2PyWv978RXGyA 2019-12-20 2020-12-20 NEWRELEASE
UZE EeFdZ1FPepk6U4k3xBZA 2LjEtdLvBLLB1QuDJ4KiJW6M4dF6IHWd_UKmo7p3_Us 2020-06-24 2021-06-24 NEWRELEASE
VAE Y8iqatTewuoHX16gghw8 FNcCB3Ks1qleS56quYzH4V7zAu78YY2bCdLRi9tmU9E 2020-06-24 2021-06-24 NEWRELEASE
+16
View File
@@ -0,0 +1,16 @@
fcm=FCM
client=NAME
session_id=ses
put_host=ftpc.example.com
port_put=2222
username_put=user
password_put=password
protocol_put=sftp
get_ilink3_file_path=/sparkdata/data/ilink3/*session*/audit_logs/
put_ilink3_file_path=/
get_ilink3_file_name=*fcm*_audit_*session*_*datestamp*.csv
put_ilink3_file_name=*fcm*_audit_*session*_*datestamp*.csv
get_md_file_path=/sparkdata/data/md_audit/
put_md_file_path=/
get_md_file_name=*fcm*_md_*client*_*datestamp*.csv
put_md_file_name=*fcm*_md_*client*_*datestamp*.csv
+1 -1
View File
@@ -1 +1 @@
set(sources api/apicore.proto;api/apigateway.proto;api/apioptions.proto;api/core/ApiClientInterop.cpp;api/core/ApiServerInterop.cpp;api/public/ApiServer.cpp;api/public/ClientChannel.cpp;api/public/QuickbusClientChannel.cpp;api/public/ServerChannel.cpp;api/public/ServiceContext.cpp;api/service_helpers/AdminServiceUtils.cpp;api/service_helpers/AlgoServiceUtils.cpp;api/service_helpers/PositionUtilities.cpp;api/services/AdminService.proto;api/services/AlgoDefinitionService.proto;api/services/AlgoService.proto;api/services/ApiUnitTestService.proto;api/services/AuthService.proto;api/services/BasePositionService.proto;api/services/ErrorReportService.proto;api/services/HelloService.proto;api/services/MBOService.proto;api/services/MarginService.proto;api/services/MarketDataAuxiliaryService.proto;api/services/MarketDataRelay.proto;api/services/MarketDataService.proto;api/services/MoneyFileService.proto;api/services/OrderService.proto;api/services/OutOfProcessURLListener.proto;api/services/PersistenceService.proto;api/services/PositionService.proto;api/services/ShellDiagnosticService.proto;api/services/ShellLatencyService.proto;api/services/SyntheticSecurityService.proto;api/test/ApiTests.cpp;api/test/HelloServiceServer.cpp;config/ConfigCheck.cpp;config/ConfigDownloader.cpp;config/ConfigErrorCollector.cpp;config/ConfigLog.cpp;config/ConfigPropertyEvaluator.cpp;config/ConfigValueQueue.cpp;config/ConfigurationException.cpp;config/ConfigurationState.cpp;config/PreprocessInputStream.cpp;config/config.cpp;config/defaults/AlgoTypes.conf;config/defaults/MarketData.conf;config/defaults/Ports.conf;config/defaults/Symbols.conf;config/defaults/cme_reference_data.conf;config/defaults/edge.conf;config/defaults/jetfix.conf;config/defaults/keycode.conf;config/defaults/logging.conf;config/defaults/orderman_cfg.conf;config/defaults/orders_fills_cleanup.conf;config/defaults/position_cfg.conf;config/defaults/span.conf;config/defaults/spreader.conf;config/defaults/testkeys.conf;config/defaults/threads.conf;config/defaults/toggles.conf;config/extensions/extensions.proto;config/invalid/invalid.proto;config/protos/ExchangeSimCfg.proto;config/protos/MarketData.proto;config/protos/Ports.proto;config/protos/Symbols.proto;config/protos/UI.proto;config/protos/cme_reference_data.proto;config/protos/jetfix.proto;config/protos/keycode.proto;config/protos/logging.proto;config/protos/mercury_cfg.proto;config/protos/orderman_cfg.proto;config/protos/orders_fills_cleanup.proto;config/protos/perflog_cfg.proto;config/protos/position_cfg.proto;config/protos/quickbus_cfg.proto;config/protos/span.proto;config/protos/spreader.proto;config/protos/subprocess.proto;config/protos/system.proto;config/protos/testkeys.proto;config/protos/threads.proto;config/protos/toggles.proto;config/tests/HTTPServer.cpp;config/tests/config_ref_tests.cpp;config/tests/config_tests.cpp;config/text_format/text_format.cpp;domain_model/Security.cpp;domain_model/Security.proto;domain_model/SecurityTests.cpp;domain_model/sod.cpp;domain_model/tests/domain_tests.cpp;expression/BooleanGrammar.cpp;expression/ExpressionEvaluator.cpp;expression/ParserExceptions.cpp;expression/Value.cpp;expression/tests/expression_parser_tests.cpp;jetfix/codec/FIXArray.cpp;jetfix/common/OutFIX.cpp;jetfix/connection/AsyncReader.cpp;jetfix/connection/IConnection.cpp;jetfix/connection/SocketConnection.cpp;jetfix/gateway/CoreGateway.cpp;jetfix/gateway/JetFIXGateway.cpp;jetfix/gateway/Session.cpp;jetfix/store/FIXFileStore.cpp;jetfix/store/session_name.cpp;jetfix/tests/async_reader_tests.cpp;jetfix/tests/decoding_tests.cpp;jetfix/tests/fix_encoding_stream_tests.cpp;jetfix/tests/fix_fragment_tests.cpp;jetfix/tests/jetfix_performance_tests.cpp;jetfix/tests/sample_fix.cpp;jetfix/tests/store_tests.cpp;logging/LoggerHub.cpp;logging/Sink.cpp;logging/logging.cpp;logging/tests/logging_tests.cpp;md/Diag/MDPeriodicDumper.cpp;md/Diag/ShmDumper.cpp;md/MBOProcessor.cpp;md/MBOServiceServer.cpp;md/MDClient.cpp;md/MDConsumer.cpp;md/MDConsumerHandle.cpp;md/MDConsumerInterop.cpp;md/MDProducer.cpp;md/MDProducerHandle.cpp;md/MemorySegmentUtils.cpp;md/PlatformLoader.cpp;md/PriceSize.cpp;md/Product.cpp;md/SecurityCache.cpp;md/SecurityDB.cpp;md/SecurityIDRecorder.cpp;md/SecurityMapper.cpp;md/SecurityOracle.cpp;md/SettlementRecorder.cpp;md/Shm/FixedSecurity.cpp;md/SyntheticSecurityService.cpp;md/cme/MDP3Utils.cpp;md/common.cpp;md/ice/ICEUtils.cpp;md/protos/co2.proto;md/relay/QueryServer.cpp;md/relay/RelayClient.cpp;md/relay/RelayRecorder.cpp;md/service/MarketDataServiceServer.cpp;md/test/Co2ManagedTestHelper.cpp;md/test/Co2TestInstrument.cpp;md/test/mbo_tests.cpp;md/test/md2_tests.cpp;md/test/md_test_utils.cpp;md/test/synthetic_book_merge_tests.cpp;md/vendor/OnixsUtils.cpp;mercury/actors/IPollableActor.cpp;mercury/broker/Broker.cpp;mercury/broker/BrokerProxy.cpp;mercury/broker/ILocalBroker.cpp;mercury/broker/IRemoteBroker.cpp;mercury/common/mercury_common.cpp;mercury/events/TimerEvent.cpp;mercury/executor/Executor.cpp;mercury/topics/Topic.Line truncated
set(sources api/apicore.proto;api/apigateway.proto;api/apioptions.proto;api/core/ApiClientInterop.cpp;api/core/ApiServerInterop.cpp;api/public/ApiServer.cpp;api/public/ClientChannel.cpp;api/public/QuickbusClientChannel.cpp;api/public/ServerChannel.cpp;api/public/ServiceContext.cpp;api/service_helpers/AdminServiceUtils.cpp;api/service_helpers/AlgoServiceUtils.cpp;api/service_helpers/PositionUtilities.cpp;api/services/AdminService.proto;api/services/AlgoDefinitionService.proto;api/services/AlgoService.proto;api/services/ApiUnitTestService.proto;api/services/AuthService.proto;api/services/BasePositionService.proto;api/services/ErrorReportService.proto;api/services/HelloService.proto;api/services/MBOService.proto;api/services/MarginService.proto;api/services/MarketDataAuxiliaryService.proto;api/services/MarketDataRelay.proto;api/services/MarketDataService.proto;api/services/MoneyFileService.proto;api/services/OrderService.proto;api/services/OutOfProcessURLListener.proto;api/services/PersistenceService.proto;api/services/PositionService.proto;api/services/ShellDiagnosticService.proto;api/services/ShellLatencyService.proto;api/services/SyntheticSecurityService.proto;api/test/ApiTests.cpp;api/test/HelloServiceServer.cpp;config/ConfigCheck.cpp;config/ConfigDownloader.cpp;config/ConfigErrorCollector.cpp;config/ConfigLog.cpp;config/ConfigPropertyEvaluator.cpp;config/ConfigValueQueue.cpp;config/ConfigurationException.cpp;config/ConfigurationState.cpp;config/PreprocessInputStream.cpp;config/config.cpp;config/defaults/MarketData.conf;config/defaults/Ports.conf;config/defaults/Symbols.conf;config/defaults/cme_reference_data.conf;config/defaults/edge.conf;config/defaults/jetfix.conf;config/defaults/keycode.conf;config/defaults/logging.conf;config/defaults/orderman_cfg.conf;config/defaults/orders_fills_cleanup.conf;config/defaults/position_cfg.conf;config/defaults/span.conf;config/defaults/spreader.conf;config/defaults/testkeys.conf;config/defaults/threads.conf;config/defaults/toggles.conf;config/extensions/extensions.proto;config/invalid/invalid.proto;config/protos/AlgoTypes.proto;config/protos/ExchangeSimCfg.proto;config/protos/MarketData.proto;config/protos/Ports.proto;config/protos/Symbols.proto;config/protos/UI.proto;config/protos/cme_reference_data.proto;config/protos/jetfix.proto;config/protos/keycode.proto;config/protos/logging.proto;config/protos/mercury_cfg.proto;config/protos/orderman_cfg.proto;config/protos/orders_fills_cleanup.proto;config/protos/perflog_cfg.proto;config/protos/position_cfg.proto;config/protos/quickbus_cfg.proto;config/protos/span.proto;config/protos/spreader.proto;config/protos/subprocess.proto;config/protos/system.proto;config/protos/testkeys.proto;config/protos/threads.proto;config/protos/toggles.proto;config/tests/HTTPServer.cpp;config/tests/config_ref_tests.cpp;config/tests/config_tests.cpp;config/text_format/text_format.cpp;domain_model/Security.cpp;domain_model/Security.proto;domain_model/SecurityTests.cpp;domain_model/sod.cpp;domain_model/tests/domain_tests.cpp;expression/BooleanGrammar.cpp;expression/ExpressionEvaluator.cpp;expression/ParserExceptions.cpp;expression/Value.cpp;expression/tests/expression_parser_tests.cpp;jetfix/codec/FIXArray.cpp;jetfix/common/OutFIX.cpp;jetfix/connection/AsyncReader.cpp;jetfix/connection/IConnection.cpp;jetfix/connection/SocketConnection.cpp;jetfix/gateway/CoreGateway.cpp;jetfix/gateway/JetFIXGateway.cpp;jetfix/gateway/Session.cpp;jetfix/store/FIXFileStore.cpp;jetfix/store/session_name.cpp;jetfix/tests/async_reader_tests.cpp;jetfix/tests/decoding_tests.cpp;jetfix/tests/fix_encoding_stream_tests.cpp;jetfix/tests/fix_fragment_tests.cpp;jetfix/tests/jetfix_performance_tests.cpp;jetfix/tests/sample_fix.cpp;jetfix/tests/store_tests.cpp;logging/LoggerHub.cpp;logging/Sink.cpp;logging/logging.cpp;logging/tests/logging_tests.cpp;md/Diag/MDPeriodicDumper.cpp;md/Diag/ShmDumper.cpp;md/MBOProcessor.cpp;md/MBOServiceServer.cpp;md/MDClient.cpp;md/MDConsumer.cpp;md/MDConsumerHandle.cpp;md/MDConsumerInterop.cpp;md/MDProducer.cpp;md/MDProducerHandle.cpp;md/MemorySegmentUtils.cpp;md/PlatformLoader.cpp;md/PriceSize.cpp;md/Product.cpp;md/SecurityCache.cpp;md/SecurityDB.cpp;md/SecurityIDRecorder.cpp;md/SecurityMapper.cpp;md/SecurityOracle.cpp;md/SettlementRecorder.cpp;md/Shm/FixedSecurity.cpp;md/SyntheticSecurityService.cpp;md/cme/MDP3Utils.cpp;md/common.cpp;md/ice/ICEUtils.cpp;md/protos/co2.proto;md/relay/QueryServer.cpp;md/relay/RelayClient.cpp;md/relay/RelayRecorder.cpp;md/service/MarketDataServiceServer.cpp;md/test/Co2ManagedTestHelper.cpp;md/test/Co2TestInstrument.cpp;md/test/mbo_tests.cpp;md/test/md2_tests.cpp;md/test/md_test_utils.cpp;md/test/synthetic_book_merge_tests.cpp;md/vendor/OnixsUtils.cpp;mercury/actors/IPollableActor.cpp;mercury/broker/Broker.cpp;mercury/broker/BrokerProxy.cpp;mercury/broker/ILocalBroker.cpp;mercury/broker/IRemoteBroker.cpp;mercury/common/mercury_common.cpp;mercury/events/TimerEvent.cpp;mercury/executor/Executor.cpp;mercury/topics/Topic.cLine truncated
+1 -1
View File
@@ -319,7 +319,7 @@ if __name__ == "__main__":
if DEBUG:
for f in response.file:
print f.content
print (f.content)
else:
output = response.SerializeToString()
sys.stdout.write(output)
@@ -580,7 +580,9 @@ bts::api::orderman::RoutingRecord AdminServiceUtils::get_routing_record(const bt
rv.set_session(conn_ptr->name());
rv.set_operator_id(cme_user_data->operator_id());
rv.set_location(cme_user_data->location());
rv.set_smp_id(cme_user_data->smp_id());
if (cme_user_data->smp_instruction()== "O")
rv.set_smp_instruction(api::orderman::SelfMatchPreventionInstruction::SELF_MATCH_PREVENTION_INSTRUCTION_CANCEL_RESTING);
else if (cme_user_data->smp_instruction()== "N")
+4 -1
View File
@@ -386,7 +386,9 @@ message NativeOrderState {
RoutingRecord routing_record = 5;
NativeOrderType native_type = 6;
string last_transact_time = 7;
uint64 last_transact_time_unix_ns = 9;
double stop_px = 8;
uint64 exchange_order_id_u64 = 10;
}
// Custom order states
@@ -455,7 +457,8 @@ enum TimeInForce {
TIME_IN_FORCE_UNDEFINED = 0;
TIME_IN_FORCE_DAY = 48;
TIME_IN_FORCE_GTC = 49;
TIME_IN_FORCE_IOC = 51;
TIME_IN_FORCE_IOC = 51; // FAK
TIME_IN_FORCE_DATE = 52;
}
enum ManualOrderIndicator {
+6
View File
@@ -45,6 +45,12 @@ CMEOptions {
}
}
ICEPublisherOptions {
value {
network_interface: "eth1"
}
}
#to make this work in dev clone the operations repo
#go here: c:\\code\\bts\\operations\\ice\\ICE_iMpactMulticastFeed_GettingStarted-1_1_36\\bin\\easyTunnel>
#modify the easyTunnelProxy.properties file to replace
File diff suppressed because it is too large. Load diff
+5
View File
@@ -103,6 +103,7 @@ message CMEOptions
// Defines amount of time Feed Engine keeps cycling before going to sleep when no useful activity can be done
optional uint32 spin_time = 4 [default = 1];
optional uint32 channels_per_thread = 5 [default = 3];
// rmem_max
optional uint32 udp_socket_buffer_size = 11 [default = 268435456]; //256mb
}
message CMEChannel {
@@ -110,9 +111,13 @@ message CMEOptions
optional string feed_engine = 3 [default = "default"];
// for re-orders
optional uint32 max_gap_recovery_cache_size = 11 [default = 6];
// for re-orders
optional uint32 lost_packet_wait_timeout_ms = 12 [default = 10];
optional OnixSLogMode onixs_log_mode = 13 [default = Important];
// If using both feeds prefer the B feed for snapshots
optional bool prefer_backup_feeds = 15 [default = false];
optional bool use_tcp_replay = 20 [default = true];
+5 -4
View File
@@ -9,9 +9,9 @@ message JetFIX {
enum JetFIXPlatform {
GLBX = 1;
CBOE = 6;
SIM = 2;
ICE = 3;
SIMICE = 4;
SIM = 2;
ICE = 3;
SIMICE = 4;
}
optional JetFIXPlatform platform = 15 [default = GLBX];
@@ -87,6 +87,7 @@ message JetFIX {
optional uint32 max_connection_attempts = 346 [default=10];
optional bool auto_fix_seq_num = 347 [default=false];
optional bool use_ilink3 = 348 [default=false] ;
}
repeated Connection connection = 10;
@@ -116,4 +117,4 @@ message CMECannedSecureLogon {
"MC8 AbEEtBn63WIAuItPDz1c n3ndjYb03Hx6wBO3me35HoONyDgO5TZKWk0tDSaX9KA 2018-04-05 2019-04-05 CERT \n"
"T39 UWggBgHWAe07aqW3ps2I tbb3it0rUQSSTa0TC9QcRwkrHEMDLWhaJH4VQ0FqGB4 2018-04-05 2019-04-05 NEWRELEASE \n"
];
}
}
+1
View File
@@ -73,4 +73,5 @@ message DynamicAlgos{
optional bool enable_tas = 3 [default = false];
optional bool enable_tascounter = 4 [default = false];
optional bool enable_limit_cover = 5 [default = false];
optional bool enable_onsignal = 6 [default = false];
}
+8 -19
View File
@@ -212,31 +212,20 @@ inline bool gr_than(double p1, double p2, double tick_size) {
return gr_than_eq(p1, p2, tick_size) &&
!test_price_equality(p1, p2, tick_size);
}
inline bool test_price_equality(double price_1, double price_2) {
return std::fabs(price_2 - price_1) < market_epsilon();
}
// Works only if ICEExchangeData is part of the a spread security
// inline bool is_ice_strip(const bts::api::ICEExchangeData& ice_data){
// return ice_data.is_strip();
// }
// ice specialization /////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////
inline bool is_ice_strip(const bts::api::ICEExchangeData& ice_data){
// TODO, put this in the security
inline bool is_ice_spread(const bts::api::ICEExchangeData& ice_data){
const auto& strip = ice_data.strip_name();
auto find = [&](const std::string& sub) -> bool {
return strip.find(sub) != std::string::npos;
};
if(!find("/"))
return find("-") ||
find("Q1") || find("Q2") || find("Q3") || find("Q4") ||
find("Cal");
else
return false;
return strip.find("/") != std::string::npos;
}
inline bool is_strip(const bts::api::Security& sec){
if(sec.has_spread())
if (sec.has_ice())
return is_ice_strip(sec.ice());
return false;
return sec.has_ice() && sec.has_spread() && !is_ice_spread(sec.ice());
}
inline bool is_spread_of_strips(const bts::api::Security& sec){
+1 -1
View File
@@ -52,7 +52,7 @@ class FIXEncodingStream {
auto msg_offset = _buffer.size();
_current_msg = FIXFragment(_buffer);
_begin_string_bm = _current_msg.add<fix::BeginString>("FIX.4.2");
_begin_string_bm = _current_msg.add<fix::BeginString>(std::string_view {"FIX.4.2"});
_body_length_bm = _current_msg.add<fix::BodyLength>(0, 6);
char mt_encoding[4];
+1
View File
@@ -85,6 +85,7 @@ typedef StringField<307> UnderlyingSecurityDesc;
typedef StringField<336> TradingSessionID;
typedef StringField<35> MsgType;
typedef StringField<37> OrderID;
typedef BigIntField<37> OrderIDu64;
typedef StringField<41> OrigClOrdID;
typedef StringField<440> ClearingAccount;
typedef StringField<48> SecurityID;
+9
View File
@@ -2,6 +2,8 @@
#include <memory>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <string_view>
#include "lib/jetfix/codec/ReaderCursor.h"
#include "lib/utils/meta_programming.h"
#include "lib/utils/datetime.h"
@@ -203,6 +205,13 @@ public:
static std::pair<uint32_t,uint32_t> write (char*buffer,std::string const &str)
{return write (buffer,str.data (),str.size ());}
#if __cplusplus >= 201703L
static uint32_t max_value_size (std::string_view const&str) {return str.size ();}
static std::pair<uint32_t,uint32_t> write (char*buffer,std::string_view const &str)
{return write (buffer,str.data (),str.size ());}
#endif
static std::pair<uint32_t,uint32_t> write (char*buffer,const char*str,uint32_t sz)
{
memcpy (buffer,str,sz);
@@ -23,9 +23,9 @@ SocketConnection::SocketConnection(bts::config::JetFIX::Connection const &cfg,
: _fd(-1), _cfg(cfg), _log(cfg.log_fix_stream()), _in_log(), _out_log(),
_perf_log(perf_log) {
if (_log) {
_in_log.open(std::string("/tmp/fix_raw_read_log_") + session_name(cfg),
_in_log.open(std::string("/tmp/fix_raw_read_log_") + session_name(cfg).as_string,
std::ios_base::app);
_out_log.open(std::string("/tmp/fix_raw_write_log_") + session_name(cfg),
_out_log.open(std::string("/tmp/fix_raw_write_log_") + session_name(cfg).as_string,
std::ios_base::app);
}
}
+22 -23
View File
@@ -786,9 +786,10 @@ OutFIXPtr make_logon(uint32_t seq_num, uint32_t last_seq_num_processed,
if(session_cfg.platform() == bts::config::JetFIX_Connection_JetFIXPlatform_ICE ||
session_cfg.platform() == bts::config::JetFIX_Connection_JetFIXPlatform_SIMICE)
{
buffer.add<ICEApplicationSystemName>("Katallage");
buffer.add<ICEApplicationSystemVersion>("1.0");
buffer.add<ICEApplicationSystemVendor>("Katallage");
std::string sender_comp_id = session_cfg.sender_subid();
buffer.add<ICEApplicationSystemName>(sender_comp_id);
buffer.add<ICEApplicationSystemVersion>(std::string_view{"1.0"});
buffer.add<ICEApplicationSystemVendor>(sender_comp_id);
buffer.add<UserName>(session_cfg.user_name());
buffer.add<RawDataLength>(session_cfg.raw_data().size());
@@ -796,19 +797,18 @@ OutFIXPtr make_logon(uint32_t seq_num, uint32_t last_seq_num_processed,
buffer.add<EncryptMethod>(0);
buffer.add<ICEStrategyPreference>(1);
buffer.add<ICETimeStampPreference>(1);
}
else
{
buffer.add<ApplicationSystemName>(application);
buffer.add<ApplicationSystemVersion>("1.0");
buffer.add<ApplicationSystemVendor>("BTS");
} else {
buffer.add<ApplicationSystemName>(std::string_view{application});
buffer.add<ApplicationSystemVersion>(std::string_view{"1.0"});
buffer.add<ApplicationSystemVendor>(std::string_view{"BTS"});
if (session_cfg.has_trading_session_id()) {
buffer.add<fix::NoTradingSessions>(1);
buffer.add<fix::TradingSessionID>(session_cfg.trading_session_id());
}
if (session_cfg.has_udf_support_indicator())
buffer.add<fix::UDFSupportIndicator>(session_cfg.udf_support_indicator());
buffer.add<fix::UDFSupportIndicator>(
session_cfg.udf_support_indicator());
if (session_cfg.use_cme_secure_logon()) {
bts_assert(session_cfg.sender_compid().size() >= 3);
@@ -819,25 +819,24 @@ OutFIXPtr make_logon(uint32_t seq_num, uint32_t last_seq_num_processed,
std::string last_seq_num_str(9, ' ');
i32_toa(last_seq_num_processed, &last_seq_num_str[0], 9);
auto ld = glbx::calculate_hmac(
session_id, seq_num_str, session_cfg.sender_compid(),
session_cfg.sender_subid(), time_str, session_cfg.target_subid(),
std::to_string(session_cfg.seconds_between_hbs()),
session_cfg.sender_locationid(), last_seq_num_str, application, "1.0",
"BTS");
#ifndef BTS_EDGE
LOG(Jetfix, Info, "Canonical string for secure logon: {}", ld.canonical);
#endif
session_id, seq_num_str, session_cfg.sender_compid(),
session_cfg.sender_subid(), time_str, session_cfg.target_subid(),
std::to_string(session_cfg.seconds_between_hbs()),
session_cfg.sender_locationid(), last_seq_num_str, application,
"1.0", "BTS");
#ifndef BTS_EDGE
LOG(Jetfix, Info, "Canonical string for secure logon: {}",
ld.canonical);
#endif
buffer.add<EncodedTextLen>(ld.access_key.size());
buffer.add<EncodedText>(ld.access_key);
buffer.add<EncryptedPasswordMethod>("CME-1-SHA-256");
buffer.add<EncryptedPasswordMethod>(std::string_view{"CME-1-SHA-256"});
buffer.add<EncryptedPasswordLen>(ld.signature.size());
buffer.add<EncryptedPassword>(ld.signature);
}
else
{
} else {
buffer.add<RawDataLength>(session_cfg.raw_data().size());
buffer.add<RawData>(session_cfg.raw_data());
if(session_cfg.user_name() != "")
if (session_cfg.user_name() != "")
buffer.add<UserName>(session_cfg.user_name());
}
}
+1 -1
View File
@@ -185,7 +185,7 @@ void load_seq_nums(std::string const &path, uint32_t *last_seq_num,
FIXFileStore::FIXFileStore(config::JetFIX::Connection const &cfg,
std::string base_path)
: me(new PImpl(cfg)) {
auto sn = session_name(cfg);
auto sn = session_name(cfg).as_string;
me->active = sn.size() > 0;
if (me->active) {
+19 -14
View File
@@ -75,49 +75,55 @@ bool in_session (datetime::PTime const &ref_date,
return closest_earlier_end<closest_earlier_start;
}
std::string session_name_from_ptime (datetime::PTime t,
SessionName session_name_from_ptime (datetime::PTime t,
boost::posix_time::time_duration time_of_day_start,
boost::posix_time::time_duration time_of_day_end)
{
std::string result (10,' ');
SessionName result;
result.as_string = std::string(10,' ');
auto start_of_session = t;
if (time_of_day_start > time_of_day_end &&
start_of_session.time_of_day () <= time_of_day_end)
start_of_session -= boost::gregorian::date_duration (1);
sprintf (&result[0], "%04d-%02d-%02d",
using namespace boost::posix_time;
ptime epoch(boost::gregorian::date(1970,1,1));
ptime session_start_date(start_of_session.date());
result.as_uint64_t = (session_start_date - epoch).total_milliseconds();
sprintf (&(result.as_string[0]), "%04d-%02d-%02d",
(unsigned short) start_of_session.date ().year (),
start_of_session.date ().month ().as_number (),
start_of_session.date ().day ().as_number ());
return result;
}
std::string session_name (datetime::PTime const &ref_date,
boost::gregorian::greg_weekday weekday_start,
boost::posix_time::time_duration time_of_day_start,
boost::gregorian::greg_weekday weekday_end,
boost::posix_time::time_duration time_of_day_end)
SessionName session_name (datetime::PTime const &ref_date,
boost::gregorian::greg_weekday weekday_start,
boost::posix_time::time_duration time_of_day_start,
boost::gregorian::greg_weekday weekday_end,
boost::posix_time::time_duration time_of_day_end)
{
if (in_session (ref_date,weekday_start,time_of_day_start,weekday_end,time_of_day_end))
return session_name_from_ptime (closest_before (ref_date,weekday_start,time_of_day_start),
boost::posix_time::time_duration(0, 0, 0),
boost::posix_time::time_duration(0, 0, 1));
else
return std::string ();
return {};
}
std::string session_name (datetime::PTime const &ref_date,
SessionName session_name (datetime::PTime const &ref_date,
boost::posix_time::time_duration time_of_day_start,
boost::posix_time::time_duration time_of_day_end)
{
if (in_session (ref_date,time_of_day_start,time_of_day_end))
return session_name_from_ptime (ref_date, time_of_day_start, time_of_day_end);
else
return "";
return {};
}
std::string session_name (config::JetFIX::Connection const &session_cfg)
SessionName session_name (config::JetFIX::Connection const &session_cfg)
{
auto now = datetime::local_now ();
@@ -134,7 +140,6 @@ std::string session_name (config::JetFIX::Connection const &session_cfg)
parse_time (session_cfg.end_time ()));
}
}}
+34 -28
View File
@@ -1,9 +1,9 @@
#pragma once
#include "lib/config/protos/jetfix.pb.h"
#include "lib/utils/datetime.h"
#include <memory>
#include <string>
#include "lib/utils/datetime.h"
#include "lib/config/protos/jetfix.pb.h"
namespace boost {
namespace gregorian {
@@ -11,39 +11,45 @@ class greg_weekday;
}
namespace posix_time {
class time_duration;
}}
}
} // namespace boost
namespace bts {
namespace jetfix {
boost::gregorian::greg_weekday parse_weekday (std::string weekday_str);
bool parse_time (std::string const &time_str,boost::posix_time::time_duration *out);
boost::gregorian::greg_weekday parse_weekday(std::string weekday_str);
bool parse_time(std::string const &time_str,
boost::posix_time::time_duration *out);
datetime::PTime closest_before (datetime::PTime const &ref_date,
boost::gregorian::greg_weekday weekday,
boost::posix_time::time_duration time_of_day);
datetime::PTime closest_before(datetime::PTime const &ref_date,
boost::gregorian::greg_weekday weekday,
boost::posix_time::time_duration time_of_day);
bool in_session (datetime::PTime const &ref_date,
boost::gregorian::greg_weekday weekday_start,
boost::posix_time::time_duration time_of_day_start,
boost::gregorian::greg_weekday weekday_end,
boost::posix_time::time_duration time_of_day_end);
bool in_session(datetime::PTime const &ref_date,
boost::gregorian::greg_weekday weekday_start,
boost::posix_time::time_duration time_of_day_start,
boost::gregorian::greg_weekday weekday_end,
boost::posix_time::time_duration time_of_day_end);
bool in_session (datetime::PTime const &ref_date,
boost::posix_time::time_duration time_of_day_start,
boost::posix_time::time_duration time_of_day_end);
bool in_session(datetime::PTime const &ref_date,
boost::posix_time::time_duration time_of_day_start,
boost::posix_time::time_duration time_of_day_end);
std::string session_name (datetime::PTime const &ref_date,
boost::gregorian::greg_weekday weekday_start,
boost::posix_time::time_duration time_of_day_start,
boost::gregorian::greg_weekday weekday_end,
boost::posix_time::time_duration time_of_day_end);
struct SessionName {
std::string as_string;
uint64_t as_uint64_t;
bool empty() const {return as_string.empty();};
};
std::string session_name (datetime::PTime const &ref_date,
boost::posix_time::time_duration time_of_day_start,
boost::posix_time::time_duration time_of_day_end);
SessionName session_name(datetime::PTime const &ref_date,
boost::gregorian::greg_weekday weekday_start,
boost::posix_time::time_duration time_of_day_start,
boost::gregorian::greg_weekday weekday_end,
boost::posix_time::time_duration time_of_day_end);
SessionName session_name(datetime::PTime const &ref_date,
boost::posix_time::time_duration time_of_day_start,
boost::posix_time::time_duration time_of_day_end);
SessionName session_name(config::JetFIX::Connection const &session_cfg);
std::string session_name (config::JetFIX::Connection const &session_cfg);
}}
} // namespace jetfix
} // namespace bts
+7 -7
View File
@@ -81,9 +81,9 @@ TEST_CASE ("JetFIX Field2Tests", "[jetfix]")
std::string out (20, ' ');
typedef StringField<123> Field;
REQUIRE(Field::max_value_size ("hello")==5);
REQUIRE(Field::max_value_size (std::string_view{"hello"})==5);
auto result = Field::write (&out[0],"hello");
auto result = Field::write (&out[0],std::string_view{"hello"});
REQUIRE (result.first==5);
REQUIRE (result.second=='h'+'e'+'l'+'l'+'o');
REQUIRE (out.compare (0,5,"hello") == 0);
@@ -112,7 +112,7 @@ TEST_CASE ("JetFIX FIXFragmentTests" , "[jetfix]")
FIXFragment g (0);
g.add<CharField<1>> ('a');
g.add<FloatField<2>> (1.2,-1,6);
g.add<StringField<3>> ("hello");
g.add<StringField<3>> (std::string_view{"hello"});
g.add<IntField<4>> (123,3);
const char*t_s = "20150427-19:05:18.922";
g.add<TimeField<5>> (datetime::parse_FIX_timestamp (t_s));
@@ -141,14 +141,14 @@ TEST_CASE ("JetFIX FIXFragmentTests" , "[jetfix]")
FIXFragment g;
auto c_bm = g.add<CharField<1>> ('a');
auto f_bm = g.add<FloatField<2>> (1.2,-1,6);
auto s_bm = g.add<StringField<3>> ("hello");
auto s_bm = g.add<StringField<3>> (std::string_view{"hello"});
auto i_bm = g.add<IntField<4>> (123,3);
auto t_bm = g.add<TimeField<5>> (datetime::parse_FIX_timestamp (t_s1));
auto b_bm = g.add<BoolField<6>> (true);
g.overwrite (c_bm,'b');
g.overwrite (f_bm,1.3,-1,6);
g.overwrite (s_bm,"olleh");
g.overwrite (s_bm,std::string_view {"olleh"});
g.overwrite (i_bm,321,3);
g.overwrite (t_bm,(datetime::parse_FIX_timestamp (t_s2)));
g.overwrite (b_bm,false);
@@ -172,11 +172,11 @@ TEST_CASE ("JetFIX FIXFragmentTests" , "[jetfix]")
FIXFragment g1;;
g1.add<CharField<1>> ('a');
g1.add<FloatField<2>> (1.2,-1,6);
g1.add<StringField<3>> ("hello");
g1.add<StringField<3>> (std::string_view{"hello"});
auto chk1 = g1.checksum ();
FIXFragment g2;
g2.add<StringField<4>> ("goodbye");
g2.add<StringField<4>> (std::string_view{"goodbye"});
g2.add<IntField<5>> (321,3);
auto chk2 = g2.checksum ();
+202 -211
View File
@@ -6,279 +6,270 @@
#include "lib/jetfix/common/OutFIXPtr.h"
#include "lib/jetfix/store/FIXFileStore.h"
#include "lib/jetfix/store/session_name.h"
#include "lib/logging/logging.h"
#include "lib/utils/TempDir.h"
#include "lib/utils/TimeStamp.h"
#include "lib/utils/datetime.h"
#include "lib/logging/logging.h"
#include <boost/date_time.hpp>
#include <boost/date_time/gregorian/gregorian.hpp>
#include <boost/filesystem.hpp>
#include <boost/date_time.hpp>
namespace bts {
namespace jetfix {
TEST_CASE ("JetFIX SessionNameTests", "[jetfix]")
{
SECTION("greg_weekday_works")
{
const char*days [] = {"sun","Monday","tuesday","Wedn","Thursday","Friday","sat"};
TEST_CASE("JetFIX SessionNameTests", "[jetfix]") {
SECTION("greg_weekday_works") {
const char *days[] = {"sun", "Monday", "tuesday", "Wedn",
"Thursday", "Friday", "sat"};
int i =0;
int i = 0;
for (auto d : days)
REQUIRE (parse_weekday (d)==boost::gregorian::greg_weekday (i++));
}
SECTION("closest_before_works")
{
auto same_day_before = closest_before (datetime::make_ptime (2015,7,17,17,0,0,0),
boost::gregorian::Friday,
boost::posix_time::time_duration(16, 59, 59));
REQUIRE (datetime::FIX_date_time_string (same_day_before)=="20150717-16:59:59");
auto same_day_after = closest_before (datetime::make_ptime (2015,7,17,17,0,0,0),
boost::gregorian::Friday,
boost::posix_time::time_duration(17,0,1));
REQUIRE (datetime::FIX_date_time_string (same_day_after)=="20150710-17:00:01");
auto another_day_same_week = closest_before (datetime::make_ptime (2015,7,17,17,0,0,0),
boost::gregorian::Thursday,
boost::posix_time::time_duration(16, 59, 59));
REQUIRE (datetime::FIX_date_time_string (another_day_same_week)=="20150716-16:59:59");
auto another_day_previous_week = closest_before (datetime::make_ptime (2015,7,17,17,0,0,0),
boost::gregorian::Saturday,
boost::posix_time::time_duration(16, 59, 59));
REQUIRE (datetime::FIX_date_time_string (another_day_previous_week)=="20150711-16:59:59");
REQUIRE(parse_weekday(d) == boost::gregorian::greg_weekday(i++));
}
SECTION("in_session_works")
{
REQUIRE (in_session (datetime::make_ptime (2015,7,17,17,0,0,0),
boost::gregorian::Sunday,
boost::posix_time::time_duration(16, 59, 59),
boost::gregorian::Friday,
boost::posix_time::time_duration(17, 0, 1)));
SECTION("closest_before_works") {
auto same_day_before =
closest_before(datetime::make_ptime(2015, 7, 17, 17, 0, 0, 0),
boost::gregorian::Friday,
boost::posix_time::time_duration(16, 59, 59));
REQUIRE(datetime::FIX_date_time_string(same_day_before) ==
"20150717-16:59:59");
REQUIRE (!in_session (datetime::make_ptime (2015,7,17,17,0,0,0),
boost::gregorian::Sunday,
boost::posix_time::time_duration(16, 59, 59),
boost::gregorian::Friday,
boost::posix_time::time_duration(16, 59, 59)));
auto same_day_after = closest_before(
datetime::make_ptime(2015, 7, 17, 17, 0, 0, 0),
boost::gregorian::Friday, boost::posix_time::time_duration(17, 0, 1));
REQUIRE(datetime::FIX_date_time_string(same_day_after) ==
"20150710-17:00:01");
REQUIRE (in_session (datetime::make_ptime (2015,7,17,17,0,0,0),
boost::posix_time::time_duration(16, 59, 59),
boost::posix_time::time_duration(17, 59, 59)));
auto another_day_same_week =
closest_before(datetime::make_ptime(2015, 7, 17, 17, 0, 0, 0),
boost::gregorian::Thursday,
boost::posix_time::time_duration(16, 59, 59));
REQUIRE(datetime::FIX_date_time_string(another_day_same_week) ==
"20150716-16:59:59");
REQUIRE (!in_session (datetime::make_ptime (2015,7,17,17,0,0,0),
boost::posix_time::time_duration(0,0,0),
boost::posix_time::time_duration(16, 0,0)));
REQUIRE (in_session (datetime::make_ptime (2015,7,17,17,0,0,0),
boost::posix_time::time_duration(17, 50, 59),
boost::posix_time::time_duration(17, 45, 59)));
REQUIRE (!in_session (datetime::make_ptime (2015,7,17,17,47,0,0),
boost::posix_time::time_duration(17, 50, 59),
boost::posix_time::time_duration(17, 45, 59)));
auto another_day_previous_week =
closest_before(datetime::make_ptime(2015, 7, 17, 17, 0, 0, 0),
boost::gregorian::Saturday,
boost::posix_time::time_duration(16, 59, 59));
REQUIRE(datetime::FIX_date_time_string(another_day_previous_week) ==
"20150711-16:59:59");
}
SECTION("session_name_works")
{
REQUIRE (session_name (datetime::make_ptime (2015,7,17,17,0,0,0),
boost::gregorian::Sunday,
boost::posix_time::time_duration(16, 59, 59),
boost::gregorian::Friday,
boost::posix_time::time_duration(17, 0, 1)) ==
"2015-07-12");
SECTION("in_session_works") {
REQUIRE(in_session(datetime::make_ptime(2015, 7, 17, 17, 0, 0, 0),
boost::gregorian::Sunday,
boost::posix_time::time_duration(16, 59, 59),
boost::gregorian::Friday,
boost::posix_time::time_duration(17, 0, 1)));
REQUIRE (session_name (datetime::make_ptime (2015,7,17,17,0,0,0),
boost::gregorian::Sunday,
boost::posix_time::time_duration(16, 59, 59),
boost::gregorian::Friday,
boost::posix_time::time_duration(16, 59, 59)) == "");
REQUIRE(!in_session(datetime::make_ptime(2015, 7, 17, 17, 0, 0, 0),
boost::gregorian::Sunday,
boost::posix_time::time_duration(16, 59, 59),
REQUIRE (session_name (datetime::make_ptime (2015,7,17,17,0,0,0),
boost::posix_time::time_duration(16, 59, 59),
boost::posix_time::time_duration(17, 59, 59)) ==
"2015-07-17");
boost::gregorian::Friday,
boost::posix_time::time_duration(16, 59, 59)));
REQUIRE (session_name (datetime::make_ptime (2015,7,17,18,0,0,0),
boost::posix_time::time_duration(16, 59, 59),
boost::posix_time::time_duration(17, 59, 59)) ==
"");
REQUIRE(in_session(datetime::make_ptime(2015, 7, 17, 17, 0, 0, 0),
boost::posix_time::time_duration(16, 59, 59),
boost::posix_time::time_duration(17, 59, 59)));
REQUIRE (session_name (datetime::make_ptime (2015,7,17,17,0,0,0),
boost::posix_time::time_duration(17, 50, 59),
boost::posix_time::time_duration(17, 45, 59)) ==
"2015-07-16");
REQUIRE(!in_session(datetime::make_ptime(2015, 7, 17, 17, 0, 0, 0),
boost::posix_time::time_duration(0, 0, 0),
boost::posix_time::time_duration(16, 0, 0)));
REQUIRE (session_name (datetime::make_ptime (2015,7,17,17,51,0,0),
boost::posix_time::time_duration(17, 50, 59),
boost::posix_time::time_duration(17, 45, 59)) ==
"2015-07-17");
REQUIRE (session_name (datetime::make_ptime (2015,7,17,17,47,0,0),
boost::posix_time::time_duration(17, 50, 59),
boost::posix_time::time_duration(17, 45, 59)) ==
"");
REQUIRE(in_session(datetime::make_ptime(2015, 7, 17, 17, 0, 0, 0),
boost::posix_time::time_duration(17, 50, 59),
boost::posix_time::time_duration(17, 45, 59)));
REQUIRE(!in_session(datetime::make_ptime(2015, 7, 17, 17, 47, 0, 0),
boost::posix_time::time_duration(17, 50, 59),
boost::posix_time::time_duration(17, 45, 59)));
}
SECTION("session_name_works") {
REQUIRE(session_name(datetime::make_ptime(2015, 7, 17, 17, 0, 0, 0),
boost::gregorian::Sunday,
boost::posix_time::time_duration(16, 59, 59),
boost::gregorian::Friday,
boost::posix_time::time_duration(17, 0, 1)).as_string ==
"2015-07-12");
REQUIRE(session_name(datetime::make_ptime(2015, 7, 17, 17, 0, 0, 0),
boost::gregorian::Sunday,
boost::posix_time::time_duration(16, 59, 59),
boost::gregorian::Friday,
boost::posix_time::time_duration(16, 59, 59)).as_string == "");
REQUIRE(session_name(datetime::make_ptime(2015, 7, 17, 17, 0, 0, 0),
boost::posix_time::time_duration(16, 59, 59),
boost::posix_time::time_duration(17, 59, 59)).as_string ==
"2015-07-17");
REQUIRE(session_name(datetime::make_ptime(2015, 7, 17, 18, 0, 0, 0),
boost::posix_time::time_duration(16, 59, 59),
boost::posix_time::time_duration(17, 59, 59)).as_string == "");
REQUIRE(session_name(datetime::make_ptime(2015, 7, 17, 17, 0, 0, 0),
boost::posix_time::time_duration(17, 50, 59),
boost::posix_time::time_duration(17, 45, 59)).as_string ==
"2015-07-16");
REQUIRE(session_name(datetime::make_ptime(2015, 7, 17, 17, 51, 0, 0),
boost::posix_time::time_duration(17, 50, 59),
boost::posix_time::time_duration(17, 45, 59)).as_string ==
"2015-07-17");
REQUIRE(session_name(datetime::make_ptime(2015, 7, 17, 17, 47, 0, 0),
boost::posix_time::time_duration(17, 50, 59),
boost::posix_time::time_duration(17, 45, 59)).as_string == "");
}
}
TEST_CASE ("JetFIX FileStoreTests", "[jetfix]")
{
SECTION("empty_store_works")
{
TEST_CASE("JetFIX FileStoreTests", "[jetfix]") {
SECTION("empty_store_works") {
config::JetFIX::Connection c;
c.set_name ("test");
c.set_sender_compid ("CME");
c.set_sender_subid ("G");
c.set_target_compid ("1U7001N");
c.set_target_subid ("BLUESTREAM");
c.set_name("test");
c.set_sender_compid("CME");
c.set_sender_subid("G");
c.set_target_compid("1U7001N");
c.set_target_subid("BLUESTREAM");
c.set_start_time ("00:00:00");
c.set_end_time ("23:59:59");
c.set_start_time("00:00:00");
c.set_end_time("23:59:59");
utils::TempDir td;
FIXFileStore fs (c,td.name ());
REQUIRE (fs.session_active ());
REQUIRE (fs.replay ().size ()==0);
REQUIRE (fs.last_seq_num ()==0);
REQUIRE (fs.last_seq_num_processed ()==0);
FIXFileStore fs(c, td.name());
REQUIRE(fs.session_active());
REQUIRE(fs.replay().size() == 0);
REQUIRE(fs.last_seq_num() == 0);
REQUIRE(fs.last_seq_num_processed() == 0);
}
SECTION("in_msg_is_stored")
{
auto td = std::make_shared<utils::TempDir> ();
SECTION("in_msg_is_stored") {
auto td = std::make_shared<utils::TempDir>();
config::JetFIX::Connection c;
c.set_name ("test");
c.set_sender_compid ("CME");
c.set_sender_subid ("G");
c.set_target_compid ("1U7001N");
c.set_target_subid ("BLUESTREAM");
c.set_name("test");
c.set_sender_compid("CME");
c.set_sender_subid("G");
c.set_target_compid("1U7001N");
c.set_target_subid("BLUESTREAM");
c.set_start_time ("00:00:00");
c.set_end_time ("23:59:59");
c.set_start_time("00:00:00");
c.set_end_time("23:59:59");
FIXEncodingStream stream (c);
auto &buf = stream.new_msg (MessageTypes::Logon);
buf.add<fix::TargetLocationID> ("US,NC");
buf.add<fix::EncryptMethod> (0);
FIXEncodingStream stream(c);
auto &buf = stream.new_msg(MessageTypes::Logon);
buf.add<fix::TargetLocationID>(std::string_view{"US,NC"});
buf.add<fix::EncryptMethod>(0);
buf.add<fix::HeartBtInt>(30);
buf.add<fix::ApplicationSystemName> ("O2");
buf.add<fix::ApplicationSystemVersion> ("1.0");
buf.add<fix::ApplicationSystemVendor> ("BTS");
stream.close ();
stream.stamp (2,3,datetime::FIX_timestamp (datetime::now ()));
auto fix = stream.to_str ();
buf.add<fix::ApplicationSystemName>(std::string_view{"O2"});
buf.add<fix::ApplicationSystemVersion>(std::string_view{"1.0"});
buf.add<fix::ApplicationSystemVendor>(std::string_view{"BTS"});
stream.close();
stream.stamp(2, 3, datetime::FIX_timestamp(datetime::now()));
auto fix = stream.to_str();
{
FIXFileStore fs (c,td->name ());
fs.store (InMsgBuffer (fix));
fs.set_last_msg_processed (3);
FIXFileStore fs(c, td->name());
fs.store(InMsgBuffer(fix));
fs.set_last_msg_processed(3);
}
FIXFileStore fs (c,td->name ());
auto replay = fs.replay ();
REQUIRE (replay.size ()==1);
REQUIRE (replay[0].type ()==MessageTypes::Logon);
REQUIRE (replay[0].to_str ()==fix);
REQUIRE (fs.last_seq_num ()==0);
REQUIRE (fs.last_seq_num_processed ()==3);
FIXFileStore fs(c, td->name());
auto replay = fs.replay();
REQUIRE(replay.size() == 1);
REQUIRE(replay[0].type() == MessageTypes::Logon);
REQUIRE(replay[0].to_str() == fix);
REQUIRE(fs.last_seq_num() == 0);
REQUIRE(fs.last_seq_num_processed() == 3);
}
SECTION("out_msg_is_stored")
{
auto td = std::make_shared<utils::TempDir> ();
SECTION("out_msg_is_stored") {
auto td = std::make_shared<utils::TempDir>();
config::JetFIX::Connection c;
c.set_name ("test");
c.set_sender_compid ("CME");
c.set_sender_subid ("G");
c.set_target_compid ("1U7001N");
c.set_target_subid ("BLUESTREAM");
c.set_name("test");
c.set_sender_compid("CME");
c.set_sender_subid("G");
c.set_target_compid("1U7001N");
c.set_target_subid("BLUESTREAM");
c.set_start_time ("00:00:00");
c.set_end_time ("23:59:59");
c.set_start_time("00:00:00");
c.set_end_time("23:59:59");
auto msg_vec = std::make_shared<OutFIX>(c);
auto &stream = msg_vec->fix.back ();
auto &buf = stream.new_msg (MessageTypes::Logon);
buf.add<fix::TargetLocationID> ("US,NC");
buf.add<fix::EncryptMethod> (0);
auto &stream = msg_vec->fix.back();
auto &buf = stream.new_msg(MessageTypes::Logon);
buf.add<fix::TargetLocationID>(std::string_view{"US,NC"});
buf.add<fix::EncryptMethod>(0);
buf.add<fix::HeartBtInt>(30);
buf.add<fix::ApplicationSystemName> ("O2");
buf.add<fix::ApplicationSystemVersion> ("1.0");
buf.add<fix::ApplicationSystemVendor> ("BTS");
stream.close ();
stream.stamp (2,3,datetime::FIX_timestamp (datetime::now ()));
buf.add<fix::ApplicationSystemName>(std::string_view{"O2"});
buf.add<fix::ApplicationSystemVersion>(std::string_view{"1.0"});
buf.add<fix::ApplicationSystemVendor>(std::string_view{"BTS"});
stream.close();
stream.stamp(2, 3, datetime::FIX_timestamp(datetime::now()));
{
FIXFileStore fs (c,td->name ());
fs.store (msg_vec);
FIXFileStore fs(c, td->name());
fs.store(msg_vec);
}
FIXFileStore fs (c,td->name ());
auto replay = fs.replay ();
REQUIRE (replay.size ()==0);
REQUIRE (fs.last_seq_num ()==3);
REQUIRE (fs.last_seq_num_processed ()==0);
FIXFileStore fs(c, td->name());
auto replay = fs.replay();
REQUIRE(replay.size() == 0);
REQUIRE(fs.last_seq_num() == 3);
REQUIRE(fs.last_seq_num_processed() == 0);
}
SECTION("test_performance")
{
auto td = std::make_shared<utils::TempDir> ();
SECTION("test_performance") {
auto td = std::make_shared<utils::TempDir>();
config::JetFIX::Connection c;
c.set_name ("test");
c.set_sender_compid ("CME");
c.set_sender_subid ("G");
c.set_target_compid ("1U7001N");
c.set_target_subid ("BLUESTREAM");
c.set_name("test");
c.set_sender_compid("CME");
c.set_sender_subid("G");
c.set_target_compid("1U7001N");
c.set_target_subid("BLUESTREAM");
c.set_start_time ("00:00:00");
c.set_end_time ("23:59:59");
c.set_start_time("00:00:00");
c.set_end_time("23:59:59");
auto start = utils::TimeStamp::now ();
auto start = utils::TimeStamp::now();
{
FIXFileStore fs (c,td->name ());
for (int i = 0; i < 10000;i++) {
FIXFileStore fs(c, td->name());
for (int i = 0; i < 10000; i++) {
auto msg_vec = std::make_shared<std::vector<FIXEncodingStream>>();
msg_vec->emplace_back (c);
auto &stream = msg_vec->back ();
auto &buf = stream.new_msg (MessageTypes::Logon);
buf.add<fix::TargetLocationID> ("US,NC");
buf.add<fix::EncryptMethod> (0);
msg_vec->emplace_back(c);
auto &stream = msg_vec->back();
auto &buf = stream.new_msg(MessageTypes::Logon);
buf.add<fix::TargetLocationID>(std::string_view{"US,NC"});
buf.add<fix::EncryptMethod>(0);
buf.add<fix::HeartBtInt>(30);
buf.add<fix::ApplicationSystemName> ("O2");
buf.add<fix::ApplicationSystemVersion> ("1.0");
buf.add<fix::ApplicationSystemVendor> ("BTS");
stream.close ();
stream.stamp (i,3,datetime::FIX_timestamp (datetime::now () - boost::gregorian::days (1)));
fs.store (InMsgBuffer (stream.to_str ()));
buf.add<fix::ApplicationSystemName>(std::string_view{"O2"});
buf.add<fix::ApplicationSystemVersion>(std::string_view{"1.0"});
buf.add<fix::ApplicationSystemVendor>(std::string_view{"BTS"});
stream.close();
stream.stamp(i, 3,
datetime::FIX_timestamp(datetime::now() -
boost::gregorian::days(1)));
fs.store(InMsgBuffer(stream.to_str()));
}
}
auto stop = utils::TimeStamp::now ();
LOG(Testing,Info,"jetfix store gen: {} ms", (stop-start).milliseconds () );
start = utils::TimeStamp::now ();
FIXFileStore fs (c,td->name ());
stop = utils::TimeStamp::now ();
LOG(Testing,Info,"jetfix store load: {} ms", (stop-start).milliseconds () );
auto stop = utils::TimeStamp::now();
LOG(Testing, Info, "jetfix store gen: {} ms",
(stop - start).milliseconds());
start = utils::TimeStamp::now();
FIXFileStore fs(c, td->name());
stop = utils::TimeStamp::now();
LOG(Testing, Info, "jetfix store load: {} ms",
(stop - start).milliseconds());
}
}
}}
} // namespace jetfix
} // namespace bts
+1
View File
@@ -120,6 +120,7 @@ void configure_logging(bts::config::Logging const&cfg, bts::config::ThreadGroupC
{log::Plugin,"Plugin"},
{log::Sdk,"Sdk"},
{log::Http,"Http"},
{log::iLink3,"iLink3"},
};
real_cfg.sinks["console"] =
+1
View File
@@ -46,6 +46,7 @@ constexpr log::Subsystem DataFlow {Latency.ordinal+1 };
constexpr log::Subsystem Plugin {DataFlow.ordinal+1 };
constexpr log::Subsystem Sdk {Plugin.ordinal+1 };
constexpr log::Subsystem Http {Sdk.ordinal+1 };
constexpr log::Subsystem iLink3 {Http.ordinal+1 };
struct LoggerHandle {
LoggerHandle(LoggerHandle const&) = delete;
+5
View File
@@ -0,0 +1,5 @@
#pragma once
#include "lib/utils/ValueWrapper.h"
namespace bts {
using ExchangeSecurityKey = utils::ValueWrapper<struct ExchangeSecurityKeyTag,uint64_t>;
}
+28 -3
View File
@@ -60,10 +60,13 @@ class IntPrice {
return delta.mantissa() < 0;
}
}
bool operator <=(const IntPrice& other) {
bool operator <=(const IntPrice& other) const {
return *this == other || *this < other;
}
bool operator >=(const IntPrice& other) {
bool operator >(const IntPrice& other) const {
return other < *this;
}
bool operator >=(const IntPrice& other) const {
return !(*this < other);
}
IntPrice operator - (const IntPrice& o) const {
@@ -72,9 +75,31 @@ class IntPrice {
IntPrice operator + (const IntPrice& o) const {
return op(o, [](int64_t m1, int64_t m2) { return m1 + m2; });
}
IntPrice& operator +=(const IntPrice& o) {
auto temp = *this + o;
_mantissa = temp._mantissa;
_exponent = temp._exponent;
return *this;
}
IntPrice& operator -=(const IntPrice& o) {
auto temp = *this - o;
_mantissa = temp._mantissa;
_exponent = temp._exponent;
return *this;
}
IntPrice operator -() {
return IntPrice(-_mantissa, _exponent);
}
IntPrice operator * (const IntPrice& other) const {
return op(other, [](int64_t m1, int64_t m2) { return m1 * m2; });
}
IntPrice operator * (const int& other) const {
return IntPrice(_mantissa * other, _exponent);
}
IntPrice& operator *=(const int& other) {
_mantissa *= other;
return *this;
}
IntPrice operator / (double v) const {
return { (int64_t)(_mantissa / v), exponent() };
}
@@ -183,4 +208,4 @@ inline std::ostream& operator<<(std::ostream& os, const IntPrice& p) {
return os;
}
}}
}}
+172 -166
View File
@@ -1,71 +1,72 @@
#ifndef _WIN32
#include "MBOServiceServer.h"
#include "MBOServiceServer.h"
#include "lib/api/services/OrderService.client.pb.h"
#include "lib/utils/Dispatcher.h"
#include "lib/utils/TimeStamp.h"
#include "lib/domain_model/Security.h"
#include "MDClient.h"
#include "ExchangeOrderKey.h"
#include "MBOProcessor.h"
#include "MDClient.h"
#include "lib/api/services/OrderService.client.pb.h"
#include "lib/domain_model/Security.h"
#include "lib/orderman/orderman_utils/orderman_common.h"
#include "lib/utils/Dispatcher.h"
#include "lib/utils/TimeStamp.h"
#include "lib/utils/LockBox.h"
#include <tuple>
#include <vector>
#include <unordered_map>
#include <vector>
using namespace bts::api::orderman;
namespace bts {
namespace md {
namespace bts {
namespace md {
//
// Simple class to handle MD recovery and maintain an index of SecurityKey=>MDConsumerHandle
// Simple class to handle MD recovery and maintain an index of
// SecurityKey=>MDConsumerHandle
class HandleIndex {
MDClient& _client;
MDClient &_client;
int _sub;
std::optional<MDConsumer> _consumer;
public:
utils::LockBox<std::unordered_map<SecurityKey,MDConsumerHandle>> index;
public:
utils::LockBox<std::unordered_map<SecurityKey, MDConsumerHandle>> index;
HandleIndex(md::MDClient& client):
_client(client),
_sub(0),
_consumer(std::nullopt),
index() {
_sub = _client.on_consumer().subscribe([=](const MDConsumer& c) { on_consumer(c); });
on_consumer(_client.current_consumer());
}
HandleIndex(md::MDClient &client)
: _client(client), _sub(0), _consumer(std::nullopt), index() {
_sub = _client.on_consumer().subscribe(
[=](const MDConsumer &c) { on_consumer(c); });
on_consumer(_client.current_consumer());
}
~HandleIndex() { _client.on_consumer().unsubscribe(_sub); }
private:
void on_consumer(const MDConsumer& c) {
private:
void on_consumer(const MDConsumer &c) {
MDConsumer temp(c);
if (temp.all_heartbeats_ok()) {
auto existing = _consumer ? _consumer->id() : 0;
LOG(md,Info, "Updateing MBOService MDConsumer. Old {} new {}", existing, temp.id());
LOG(md, Info, "Updateing MBOService MDConsumer. Old {} new {}",
existing, temp.id());
// clear the index
index.lock([](auto& index) { index.clear(); });
index.lock([](auto &index) { index.clear(); });
// Swap to new consumer
if (_consumer) {
_consumer->swap(temp);
} else {
_consumer = temp;
}
else { _consumer = temp; }
// Subscribe to on_security
auto id = _consumer->id();
_consumer->on_security([this,id](const SecurityPtr&, MDConsumerHandle handle) {
if (_consumer->id() == id) {
index.lock([handle](auto& index) {
index[SecurityKey(handle.key())] = handle;
});
}
});
_consumer->on_security(
[this, id](const SecurityPtr &, MDConsumerHandle handle) {
if (_consumer->id() == id) {
index.lock([handle](auto &index) {
index[SecurityKey(handle.key())] = handle;
});
}
});
}
}
};
@@ -73,21 +74,20 @@ private:
class ActiveOrder {
SecurityKey _security;
SparkOrderKey _id;
ExchangeOrderKey _ex_id;
ExchangeOrderKey _ex_id;
int32_t _position;
public:
public:
// empty constructor needed for unordered map operator[]
// ActiveOrder(): _security(), _id(), _ex_id(), _position(-1) { bts_assert(false); }
ActiveOrder(SecurityKey sec, SparkOrderKey s, ExchangeOrderKey ex): _security(sec), _id(s), _ex_id(ex), _position(-1) {}
ActiveOrder(SecurityKey sec, SparkOrderKey s, const std::string& ex): _security(sec), _id(s), _ex_id(), _position(-1) {
_ex_id = ExchangeOrderKey(std::stoul(ex));
}
// ActiveOrder(): _security(), _id(), _ex_id(), _position(-1) {
// bts_assert(false); }
ActiveOrder(SecurityKey sec, SparkOrderKey s, ExchangeOrderKey ex)
: _security(sec), _id(s), _ex_id(ex), _position(-1) {}
SecurityKey security_id() const { return _security; }
SparkOrderKey id() const { return _id; }
ExchangeOrderKey exchange_id() const { return _ex_id; }
int32_t& position() { return _position; };
int32_t &position() { return _position; };
int32_t position() const { return _position; };
bool set_position(int32_t pos) {
if (pos != _position) {
@@ -108,23 +108,14 @@ public:
// MBOService
struct MBOServiceImpl {
MBOServiceImpl(
api::ServiceContext& ctx,
utils::Dispatcher& d,
md::MDClient& c,
MBOServiceBase::PiqPublisher&& pub
):
dispatcher(d),
client(c),
orders(ctx.locate<api::orderman::OrderService>()),
handle_index(c),
publish(pub),
timer(d, 200, [&]() {timer_tick();}),
orders_sub()
{}
MBOServiceImpl(api::ServiceContext &ctx, utils::Dispatcher &d,
md::MDClient &c, MBOServiceBase::PiqPublisher &&pub)
: dispatcher(d), client(c),
orders(ctx.locate<api::orderman::OrderService>()), handle_index(c),
publish(pub), timer(d, 200, [&]() { timer_tick(); }), orders_sub() {}
utils::Dispatcher& dispatcher;
md::MDClient& client;
utils::Dispatcher &dispatcher;
md::MDClient &client;
OrderService orders;
HandleIndex handle_index;
@@ -138,23 +129,23 @@ struct MBOServiceImpl {
void flush();
};
///////////////////////
std::optional<uint32_t> search_orders(const ExchangeOrderKey oid, const std::vector<SnapshotOrder>& orders) {
std::optional<uint32_t>
search_orders(const ExchangeOrderKey oid,
const std::vector<SnapshotOrder> &orders) {
IntPrice price = {};
size_t position = 0;
for (size_t x = 0; x < orders.size(); ++x) {
const auto& oo = orders[x];
const auto &oo = orders[x];
if (oo.id == oid) {
if (oo.price == price) return { position };
else return 0;
}
else if (price == oo.price) {
if (oo.price == price)
return {position};
else
return 0;
} else if (price == oo.price) {
position += oo.size;
}
else {
} else {
price = oo.price;
position = oo.size;
}
@@ -162,11 +153,11 @@ std::optional<uint32_t> search_orders(const ExchangeOrderKey oid, const std::vec
return std::nullopt;
}
bool update_position(const HandleIndex& handles, ActiveOrder& order) {
const auto& index = handles.index;
auto handle = index.lock([id=order.security_id()](const auto& index) {
bool update_position(const HandleIndex &handles, ActiveOrder &order) {
const auto &index = handles.index;
auto handle = index.lock([id = order.security_id()](const auto &index) {
auto iter = index.find(id);
return iter != index.end() ? iter->second : MDConsumerHandle {};
return iter != index.end() ? iter->second : MDConsumerHandle{};
});
if (handle.is_initialized()) {
@@ -185,9 +176,9 @@ bool update_position(const HandleIndex& handles, ActiveOrder& order) {
}
void MBOServiceImpl::timer_tick() {
//Dispatcher thread
// Dispatcher thread
std::vector<OrderPIQ> rows;
for (auto& pair : active) {
for (auto &pair : active) {
if (update_position(handle_index, pair.second)) {
rows.push_back(pair.second.to_piq());
}
@@ -197,29 +188,30 @@ void MBOServiceImpl::timer_tick() {
}
}
const NativeOrderState* native_order_state(const OrderRecord& record) {
switch(record.state().custom_case()) {
case OrderState::kMarket:
return &record.state().market().native_state();
case OrderState::kLimit:
return &record.state().limit().native_state();
case OrderState::kStop:
return &record.state().stop().native_state();
case OrderState::kStopLimit:
return &record.state().stop_limit().native_state();
case OrderState::kGlbxIceberg:
return &record.state().glbx_iceberg().native_state();
default:
const NativeOrderState *native_order_state(const OrderRecord &record) {
switch (record.state().custom_case()) {
case OrderState::kMarket:
return &record.state().market().native_state();
case OrderState::kLimit:
return &record.state().limit().native_state();
case OrderState::kStop:
return &record.state().stop().native_state();
case OrderState::kStopLimit:
return &record.state().stop_limit().native_state();
case OrderState::kGlbxIceberg:
return &record.state().glbx_iceberg().native_state();
default:
return nullptr;
}
}
bool is_active_exchange_order(const OrderRecord& record) {
bool is_active_exchange_order(const OrderRecord &record) {
return record.state().status() == ORDER_STATUS_WORKING &&
native_order_state(record) != nullptr;
}
void try_remove(MBOServiceImpl& me, std::set<uint32_t>& to_remove, const OrderRecord& record) {
void try_remove(MBOServiceImpl &me, std::set<uint32_t> &to_remove,
const OrderRecord &record) {
auto key = SparkOrderKey(record.key());
auto iter = me.active.find(key);
if (iter != me.active.end()) {
@@ -227,103 +219,113 @@ void try_remove(MBOServiceImpl& me, std::set<uint32_t>& to_remove, const OrderRe
to_remove.insert(key.value());
}
}
void try_add(MBOServiceImpl& me, std::vector<OrderPIQ>& to_add, const OrderRecord& record) {
void try_add(MBOServiceImpl &me, std::vector<OrderPIQ> &to_add,
const OrderRecord &record) {
if (is_active_exchange_order(record)) {
auto native = native_order_state(record);
auto ex_key = orderman::get_exchange_order_id(*native);
auto u64_key =
std::visit(overloaded{[](orderman::ExchangeOrderID const &str) {
return std::stoul(str.value().data());
},
[](orderman::ExchangeOrderID64 oid) {
return oid.value();
}},
ex_key);
bts_assert(native);
ActiveOrder order(
SecurityKey(record.def().security_key()),
SparkOrderKey(record.key()),
native->exchange_order_id());
ActiveOrder order(SecurityKey(record.def().security_key()),
SparkOrderKey(record.key()), ExchangeOrderKey{u64_key});
update_position(me.handle_index, order);
to_add.push_back(order.to_piq());
me.active.insert(std::make_pair(order.id(), order));
}
}
MBOServiceServer::MBOServiceServer(
api::ServiceContext& ctx,
utils::Dispatcher& d,
md::MDClient& c
) :
MBOServiceBase(ctx),
_me(new MBOServiceImpl(ctx,d,c,make_piq_publisher()))
{
// subscribe to mdclient ✔
// make publishers ✔
// subscribe to orders => stash active orders dict
_me->orders_sub =
_me->orders.subscribe_orders().subscribe([=](const auto& update) {
// api thread
const auto& added = update.added();
const auto& changed = update.changed();
const auto& removed = update.removed();
_me->dispatcher.post([this,added,changed,removed]() {
// dispatcher thread
std::vector<OrderPIQ> to_add, to_change;
std::set<uint32_t> to_remove;
MBOServiceServer::MBOServiceServer(api::ServiceContext &ctx,
utils::Dispatcher &d, md::MDClient &c)
: MBOServiceBase(ctx),
_me(new MBOServiceImpl(ctx, d, c, make_piq_publisher())) {
// subscribe to mdclient ✔
// make publishers ✔
// subscribe to orders => stash active orders dict
_me->orders_sub =
_me->orders.subscribe_orders().subscribe([=](const auto &update) {
// api thread
const auto &added = update.added();
const auto &changed = update.changed();
const auto &removed = update.removed();
_me->dispatcher.post([this, added, changed, removed]() {
// dispatcher thread
std::vector<OrderPIQ> to_add, to_change;
std::set<uint32_t> to_remove;
// Remove
for(const auto& r : removed) {
// If it was active remove it and add to to_remove
try_remove(*_me, to_remove, *r);
}
// Added
for(const auto& add : added) {
// If active add look up the position and add to the state
try_add(*_me, to_add, *add);
}
// Changed
for(const auto& ch : changed) {
// new order is active and old is not look up piq and add to state
bool is_active = is_active_exchange_order(*ch.current());
bool was_active = is_active_exchange_order(*ch.previous());
if (is_active && !was_active) {
try_add(*_me, to_add, *ch.current());
}
else if (was_active && !is_active) {
try_remove(*_me, to_remove, *ch.current());
}
// is_active && was_active => don't care
// !is_active && !was_active => don't care
}
// Remove
for (const auto &r : removed) {
// If it was active remove it and add to to_remove
try_remove(*_me, to_remove, *r);
}
// Added
for (const auto &add : added) {
// If active add look up the position and add to the state
try_add(*_me, to_add, *add);
}
// Changed
for (const auto &ch : changed) {
// new order is active and old is not look up piq and add to
// state
bool is_active = is_active_exchange_order(*ch.current());
bool was_active = is_active_exchange_order(*ch.previous());
if (is_active && !was_active) {
try_add(*_me, to_add, *ch.current());
} else if (was_active && !is_active) {
try_remove(*_me, to_remove, *ch.current());
}
// is_active && was_active => don't care
// !is_active && !was_active => don't care
}
_me->publish(&to_add, &to_change, &to_remove);
if (!_me->timer.running()) _me->timer.start();
});
});
}
_me->publish(&to_add, &to_change, &to_remove);
if (!_me->timer.running())
_me->timer.start();
});
});
}
MBOServiceServer::~MBOServiceServer() { }
MBOServiceServer::~MBOServiceServer() {}
void MBOServiceServer::get_orders_for_security(const MBOSecurityQuery& request, const api::Response<PositionData>& reply) {
void MBOServiceServer::get_orders_for_security(
const MBOSecurityQuery &request, const api::Response<PositionData> &reply) {
PositionData response;
response.set_security_id(request.security_id());
auto handle = _me->handle_index.index.lock([id=request.security_id()](const auto& index) {
auto iter = index.find(SecurityKey(id));
return iter != index.end() ? iter->second : MDConsumerHandle {};
});
auto handle = _me->handle_index.index.lock(
[id = request.security_id()](const auto &index) {
auto iter = index.find(SecurityKey(id));
return iter != index.end() ? iter->second : MDConsumerHandle{};
});
std::unordered_map<ExchangeOrderKey, SparkOrderKey> reverse;
for (const auto& pair : _me->active) {
for (const auto &pair : _me->active) {
reverse.insert(std::make_pair(pair.second.exchange_id(), pair.first));
}
size_t count = 0;
auto tick = handle.get_minimum_tick();
if(handle.is_initialized()) {
if (handle.is_initialized()) {
auto mbo = handle.get_mbo();
if (mbo) {
for (const auto& o : mbo->bids) {
for (const auto &o : mbo->bids) {
double price = o.price.price();
bool isnan = os::is_nan(request.price_level());
if (isnan || securities::test_price_equality(request.price_level(), price, tick)) {
if (isnan || securities::test_price_equality(request.price_level(),
price, tick)) {
++count;
if (request.limit() && count > request.limit()) break;
auto* entry = response.add_bid();
if (request.limit() && count > request.limit())
break;
auto *entry = response.add_bid();
entry->set_exchange_id(o.id.value());
if (isnan) entry->set_price(price);
if (isnan)
entry->set_price(price);
entry->set_size(o.size);
auto iter = reverse.find(o.id);
if (iter != reverse.end()) {
@@ -331,15 +333,18 @@ void MBOServiceServer::get_orders_for_security(const MBOSecurityQuery& request,
}
}
}
for (const auto& o : mbo->asks) {
for (const auto &o : mbo->asks) {
double price = o.price.price();
bool isnan = os::is_nan(request.price_level());
if (isnan || securities::test_price_equality(request.price_level(), price, tick)) {
if (isnan || securities::test_price_equality(request.price_level(),
price, tick)) {
++count;
if (request.limit() && count > request.limit()) break;
auto* entry = response.add_ask();
if (request.limit() && count > request.limit())
break;
auto *entry = response.add_ask();
entry->set_exchange_id(o.id.value());
if (isnan) entry->set_price(price);
if (isnan)
entry->set_price(price);
entry->set_size(o.size);
auto iter = reverse.find(o.id);
if (iter != reverse.end()) {
@@ -349,9 +354,10 @@ void MBOServiceServer::get_orders_for_security(const MBOSecurityQuery& request,
}
}
}
reply(response);
}
}} //namespace
} // namespace md
} // namespace bts
#endif
+1 -1
View File
@@ -92,7 +92,7 @@ MDConsumer& MDClient::current_consumer() const {
return _self->_consumer;
}
SecurityOracle MDClient::oracle() const {
return SecurityOracle(*this);
return SecurityOracle(_self->_cache);
}
std::set<UserSymbol> MDClient::symbols() const {
return _self->_symbols;
+2 -2
View File
@@ -897,7 +897,7 @@ public:
scan_securities(resolve_legs);
}
SecurityPtr get_security(SecurityKey key) const {
const Security* get_security(SecurityKey key) const {
return _cache.get_security(key);
}
@@ -979,7 +979,7 @@ size_t MDConsumer::walk_security_list(SecurityCallback cb, const string& symbol,
size_t MDConsumer::scan_securities(bool resolve_legs) { return _pimpl->scan_securities(resolve_legs); }
void MDConsumer::on_security(SecurityCallback cb, bool resolve_legs) { _pimpl->on_security(cb, resolve_legs); }
const api::Security* MDConsumer::get_security(SecurityKey key) const { return _pimpl->get_security(key).get(); }
const api::Security* MDConsumer::get_security(SecurityKey key) const { return _pimpl->get_security(key); }
std::vector<api::Security const*> MDConsumer::get_securities() const {return _pimpl->get_securities();}
MDConsumerChannel MDConsumer::get_channel(const std::string& symbol, SecurityType type, Venue exchange) const {
+66 -42
View File
@@ -1,8 +1,10 @@
#include "SecurityCache.h"
#include "SecurityDB.h"
#include "lib/utils/Dispatcher.h"
#include "lib/utils/LockBox.h"
#include <unordered_map>
#include <google/btree/btree_map.h>
#include <tbb/concurrent_queue.h>
#include <mutex>
@@ -10,8 +12,8 @@ namespace bts {
namespace md {
struct SecurityCacheImpl {
std::mutex cache_mutex;
std::unordered_map<SecurityKey, SecurityPtr> cache;
utils::LockBox<btree::btree_map<SecurityKey, SecurityPtr>> cache;
utils::LockBox<btree::btree_map<ExchangeSecurityKey, SecurityPtr>> ex_cache;
tbb::concurrent_queue<SecurityPtr> update_queue;
void enqueue_update_db(const SecurityPtr&);
void flush();
@@ -42,61 +44,83 @@ void SecurityCacheImpl::flush() {
SecurityCache::SecurityCache() : _me(std::make_shared<SecurityCacheImpl>()) {}
SecurityCache::~SecurityCache() {}
SecurityPtr SecurityCache::try_get_security(SecurityKey key) const {
std::lock_guard<std::mutex> lock(_me->cache_mutex);
auto iter = _me->cache.find(key);
if (iter != _me->cache.end()) {
return iter->second;
}
return nullptr;
const Security* SecurityCache::try_get_security(SecurityKey key) const {
return _me->cache.lock([key](btree::btree_map<SecurityKey, SecurityPtr>& t) {
auto iter = t.find(key);
if (iter != t.end()) {
return iter->second.get();
}
return (const api::Security*)nullptr;
});
}
const Security* SecurityCache::try_get_security(ExchangeSecurityKey key) const {
return _me->ex_cache.lock([key](btree::btree_map<ExchangeSecurityKey,SecurityPtr>& t) {
auto iter = t.find(key);
if (iter != t.end()) {
return iter->second.get();
}
return (const api::Security*)nullptr;
});
}
SecurityPtr SecurityCache::get_security(SecurityKey key) {
std::unique_lock<std::mutex> lock(_me->cache_mutex);
auto iter = _me->cache.find(key);
if (iter != _me->cache.end()) {
return iter->second;
}
else {
// unlock for db read
lock.unlock();
const Security* SecurityCache::get_security(SecurityKey key) {
const Security* result = try_get_security(key);
if (result == nullptr) {
// outside lock
auto security = db::find_security(key);
if (security) {
LOG(md,Info, "Loaded security from database: {} {}", security->symbol(), security->id());
return add_db_security(security);
}
return security;
return nullptr;
}
}
std::vector<api::Security const*> SecurityCache::get_securities()
{
std::vector<api::Security const*> result;
result.reserve(100000);
std::unique_lock<std::mutex> lock(_me->cache_mutex);
for (auto const&p : _me->cache)
result.push_back(p.second.get());
return result;
}
SecurityPtr SecurityCache::add_db_security(const SecurityPtr& security) {
std::vector<api::Security const*> SecurityCache::get_securities() {
std::vector<api::Security const*> result;
result.reserve(100000);
_me->cache.lock([&result](auto& cache) {
for (auto const&p : cache) {
result.push_back(p.second.get());
}
});
return result;
}
const Security* SecurityCache::add_db_security(const SecurityPtr& security) {
bts_assert(security);
std::lock_guard<std::mutex> lock(_me->cache_mutex);
SecurityKey key(security->id());
auto result = _me->cache.insert(std::make_pair(SecurityKey(security->id()), security));
// If result was inserted result.first->second is equal to security
// If the value was already present it must be a recently inserted 'live' security and we should use it
return result.first->second;
if (security->has_glbx()) {
_me->ex_cache.lock([&security](auto& cache) {
ExchangeSecurityKey key(security->id());
cache.insert(std::make_pair(key, security));
});
}
return _me->cache.lock([&security](auto& cache) {
SecurityKey key(security->id());
auto result = cache.insert(std::make_pair(key, security));
// If result was inserted result.first->second is equal to security
// If the value was already present it must be a recently inserted 'live' security and we should use it
return result.first->second.get();
});
}
bool SecurityCache::add_live_security(const SecurityPtr& security) {
bts_assert(security);
std::lock_guard<std::mutex> lock(_me->cache_mutex);
SecurityKey key(security->id());
// Insert but don't overwrite existing
auto result = _me->cache.insert(std::make_pair(SecurityKey(security->id()), security));
_me->enqueue_update_db(security);
return result.second;
if (security->has_glbx()) {
_me->ex_cache.lock([&security](auto& cache) {
ExchangeSecurityKey key(security->id());
cache.insert(std::make_pair(key, security));
});
}
return _me->cache.lock([&](auto& cache) {
SecurityKey key(security->id());
// Insert but don't overwrite existing
auto result = cache.insert(std::make_pair(SecurityKey(security->id()), security));
_me->enqueue_update_db(security);
return result.second;
});
}
void SecurityCache::flush_to_disk() { _me->flush(); }
+5 -3
View File
@@ -3,6 +3,7 @@
#include "lib/domain_model/Security.pb.h"
#include "MDConsumerHandle.h"
#include "lib/domain_model/SecurityKey.h"
#include "lib/md/ExchangeSecurityKey.h"
#include "SecurityPtr.h"
#include <memory>
@@ -20,17 +21,18 @@ public:
~SecurityCache();
//Load security from cache with fallback to database
SecurityPtr get_security(SecurityKey key);
const Security* get_security(SecurityKey key);
// Load all securities from cache, no fallback to database
std::vector<api::Security const*> get_securities();
// Get the security if it's in the cache otherwise return nullptr. Does not query security database.
SecurityPtr try_get_security(SecurityKey key) const;
const Security* try_get_security(SecurityKey key) const;
const Security* try_get_security(ExchangeSecurityKey key) const;
// returns true if the cache was updated
bool add_live_security(const SecurityPtr& security);
SecurityPtr add_db_security(const SecurityPtr& security);
const Security* add_db_security(const SecurityPtr& security);
void flush_to_disk();
-56
View File
@@ -1,56 +0,0 @@
#pragma once
#include "core/bts_core/utils/bts_no_copy.h"
#include "Shm/FixedSecurity.h"
#include "../bts_conn_exports.h"
#include <memory>
#include <functional>
#include <atomic>
namespace bts {
namespace md {
class BTS_EXPORT SecurityConstructor {
// An object "pool" optimized for the case where only one instance is ever used at a time
template <class T>
class SingleObjectPool<T> {
public:
using ptr_type = std::unique_ptr<T, std::function<void(T*)>>;
ptr_type acquire() {
T* pt = _inst ? _inst.release() : new T();
return ptr_type(pt, [this](T* ptr) {
return_inst(std::unique_ptr<T>(ptr));
});
}
private:
void return_inst(std::unique_ptr<T> t) {
if (!_inst) {
reset(t);
_inst = std::move(t);
}
}
unique_ptr<T> _inst;
};
public:
SecurityConstructor() : _future(), _option(), _spread() {}
SingleObjectPool<FixedFutureSecurity>& future() { return _future; }
SingleObjectPool<FixedOptionSecurity>& option() { return _option; }
SingleObjectPool<FixedSpreadSecurityL>& spread() { return _spread; }
private:
template <typename T>
void reset(T& t) {
new (&t)T();
}
SingleObjectPool<FixedFutureSecurity> _future;
SingleObjectPool<FixedOptionSecurity> _option;
SingleObjectPool<FixedSpreadSecurityL> _spread;
};
}}//namespace
+7 -3
View File
@@ -4,14 +4,18 @@
namespace bts {
namespace md {
SecurityOracle::SecurityOracle(const MDClient& client): _client(client) {}
SecurityOracle::SecurityOracle(SecurityCache& cache): _cache(cache) {}
SecurityOracle::~SecurityOracle() { }
const api::Security* SecurityOracle::get_security(SecurityKey key) const {
return _client.current_consumer().get_security(key);
return _cache.get_security(key);
}
const api::Security* SecurityOracle::get_security(ExchangeSecurityKey key) const {
return _cache.try_get_security(key);
}
std::vector<api::Security const*> SecurityOracle::get_securities() const {
return _client.current_consumer().get_securities();
return _cache.get_securities();
}
}}//namespace
+5 -2
View File
@@ -1,6 +1,8 @@
#pragma once
#include "lib/domain_model/SecurityKey.h"
#include "lib/md/ExchangeSecurityKey.h"
#include "lib/md/SecurityCache.h"
#include <memory>
#include <vector>
@@ -9,11 +11,12 @@ namespace api {class Security;}
namespace md {
class MDClient;
class SecurityOracle {
const MDClient& _client;
SecurityCache& _cache;
public:
SecurityOracle(const MDClient&);
SecurityOracle(SecurityCache&);
~SecurityOracle();
const api::Security* get_security(SecurityKey key) const;
const api::Security* get_security(ExchangeSecurityKey key) const;
std::vector<api::Security const*> get_securities() const;
};
+209
View File
@@ -0,0 +1,209 @@
# Overview
The core abstraction that everything else is built on is the MDProducer+MDProducerHandle / MDConsumer+MDConsumerHandle.
A new market data source constructs an MDProducer and adds securities to it. When a security is added you get back an MDProducerHandle representing that security in shared memory. The memory is fully allocated and books can be written immidiately but the security will not be visible to the MDConsumer until the security key is stored on disk (SecurityIDRecorder). The MDProducerHandle assumes a **single writer** for most operations and *may break silently if this constraint is violated*.
The MDConsumer is the dual of the MDProducer. It provides an on_security() callback which is driven by the call to scan_securities(). No threads are created by MDConsumer/MDProducer instead they get cpu time from periodic calls to scan_securities (consumer), SecurityIDRecorder::do_work (producer). When a security is discovered by scan_securities the callback is called with the Security and the MDConsumerHandle. The MDConsumerHandle is thread safe and can be safely copied and read from concurrently.
## Single Writer Principle
- https://mechanical-sympathy.blogspot.com/2011/09/single-writer-principle.html
# Core Classes
'*' = New in spark
* MBOProcessor*
* MBOServiceServer*
* MDClient*
* MDConsumer
* MDConsumerHandle
* MDProducer
* MDProducerHandle
* PlatformLoader
* PriceSize/IntPrice
* SecurityCache*
* SecurityOracle*
* SecurityDB*
* SecurityIDRecorder
* SecurityMapper*
* SettlementRecorder
* SyntheticSecurityService
* FixedSecurity.h / MDStructure.h
* SharedList.h
## MBOProcessor*
This class is an optional dependency of MDProducer. It takes the low level incremental updates to the MBO book and constructs the book on a different thread. This is optimized for the UI use case so it tries to get the mbo messages out of the critical path as soon as possible. This uses regular heap memory so is not compatible with out of process operation. It exposes a reader and writer interface.
Each update goes into a queue and gets processed by MBOProcessorImpl::process_incremental() on a single background thread. This thread drains the queue and constructs the books in place. Then it copies snapshots of the books and publishes them as immutable snapshots which are consumed by the MBOServiceServer/UI. If mbo load gets very high this function will stop publishing snapshots and only update books until the load reduces.
## MBOServiceServer*
Reads the snapshots from the MBOProcessor/MBOReader and serves them for the UI.
## MDClient*
New in spark.
This class handles constructing the MDConsumer, calling scan_securities on it and checking for bad heartbeats and re-constructing the MDConsumer if there was a failure. In edge those tasks were the responsibility of each consumer of the market data but it was mitigated because most things used streams rather than MDConsumer directly. MDConsumer is reference counted internally so when the consuming code gets an on_consumer() callback with a new MDConsumer it should keep a copy of the old MDConsumer until it has finished any asyncronous processes involving the MDConsumerHandles from that first consumer. Once an MDConsumer is destroyed the MDConsumerHandles it produced will cause an **Access Violation** on reading because the memory has been unmapped.
Spark has always used MD in process so, while based on existing functionality, this recovery process hasn't been well tested.
## MDConsumer
An MDConsumer maps the shared memory sections and produces MDConsumerHandles. These handles are only valid as long as the MDConsumer exists. It would make some sense to store the MDConsumer reference in the MDConsumerHandle to avoid this but MDConsumerHandle is designed to be *very* small for fast iteration. The MDConsumer also holds a reference to the ISyntheticSecurityService and makes synthetic securities 'look like' regular shared memory securities using polymorphism in the MDConsumerHandle.
Internally MDConsumer is a collection of MDConsumerInst which each handle a single symbol+type+exchange and the corresponding shared memory blocks.
* scan_securities - in spark this is always called by MDClient
* get_handle - Low level way to get a handle. Does a linear walk though shared memory and can return a void handle (MDConsumerHandle::is_initialize() == false).
* get_managed_handle - returns a 'managed handle' this is a shared_ptr to an MDConsumerhandle. The only difference is that this is tracked by the MDConsumer and may transition from uninitialized to initialized. Storing these in a vector and walking them is almost *50% slower* than doing the same thing with an unmanaged handle. If you use these it's probably best to store the uninitialized handles in a nursery and check them periodically. You can dereference a managed handle into a regular handle at any time.
* get_channel - Gets a MDConsumerChannel representing common data for a symbol+type+exchange. This is mostly for the trade ticker / rfq and also exposes various counters which get published to dev-stats.
On Linux MDConsumer maps the shared memory as read-only pages. On Windows it cannot do this because std::atomic uses a different instruction which is considered a write by the CPU so we map it read/write.
If we have multiple platforms that can produce md for an exchange (i.e. onixs vs emumba) the MDConsumer consults the PlatformMap config key to determine which to read from. The md replayer impersonates the producer it was recorded from.
### MDConsumerChannel
read_trades() and read_rfqs() read from a ring buffer stored in the header section of the shared memory. They read directly into a google::protobuf::RepeatedPtrField to avoid extra copies in the QueryServer but can just as easily read into a std::vector. They expose a 'cursor' representing the position in the ring buffer and this cursor is sent all the way to the client so it can request updates.
## MDConsumerHandle
An MDConsumerHandle 4 states and would proably be impemented as an std::variant<> if done today. If not for the high performance requirements it could also be a virtual class but I didn't want to pay for a vtable. It is the size of 4 pointers plus the enum tag.
* VOID - an uninitialized handle
* BBO - Legacy from Edge. This represents an option book where we only kept the top to save space. I think this has rotted by now and wont work but I kept it as an example of how to add polymorphism to the shared memory types.
* DEPTH - Used for all exchange securities in Spark.
* SYNTHETIC - A Synthetic handle. These are bigger so they get stored as a pointer.
Reading from a BBO/DEPTH handle is done with an optimistic copy. If you inrease the size of the book data this copy can fail more often and get significantly worse perf. The result of the read can be changed, unchanged or busy (timeout). In edge 'busy' happened 1-2 times per day across all reads and gets logged by QueryServer.
## MDProducer
Handle to writeable shared memory. Each market data producer "platform" (onixs_cme, onixs_ice, emumba, etc) constructs one.
## MDProducerHandle
All updates to a single security go through here. Assumes a single writer thread.
## PlatformLoader
In Edge this was the entry point to the run() for the market data producer process. It loaded the bts_md_*.so so each producer process only loaded the third party objects for a particular producer. In spark this is all in process so this is defunct. It looks like it still exists because it has some utility functions that probably belong in another file.
## PriceSize/IntPrice
Market data uses integer prices to avoid double problems.
## SecurityCache*
Created/Owned by MDClient this owns the Securities.
It has methods to get a security from the live collection or load them from the SecurityDB on disk if someone asks for one that hasn't been seen yet.
* If a security is loaded from disk it 'freezes' it so when the live security arrives it will continue to serve the old security so we don't end up with 2 copies of a single security with different properties.
* This class is also responsible for sending the live securities to the SecurityDB where we *always* overwrite stored securities with what comes from the exchange.
* Survives beyond the MDConsumer in case of crash/recovery of out of process md.
## SecurityOracle*
Interface on top of SecurityCache exposing the read get_ operations.
## SecurityDB*
* Just a collection of methods for save/load of securities to sqlite.
* Stored in binary format so we can use the protobuf versioning
## SecurityIDRecorder
* Manages security to unique 32 bit id mapping across multiple processes
* Provides a callback when ids are safely persisted.
## SecurityMapper*
* Happens *before* the security gets stored in shared memory.
* Replaces the mapping.h from edge with a more organized mechanism.
* Owned by the MDProducer each security is passed though SecurityMapper::apply() before it is first stored in shared memory
* SecurityMapper is made up of multiple ISecurityMapper instances so each mapping can be in it's own class so it's easier to read each mapping. Currently all mappings are defined in SecurityMaper.cpp
* The mappings each have an opportunity to mutate the FixedSecurity.
## SettlementRecorder
In Edge SettlementRecorder stored the settlements for a security so we could preserve yesterday/day before settlement. In Spark this broke and nobody seemed to care so it was never fixed.
## SyntheticSecurityService
Each synthetic security is a api::Security and a chunk of metadata. This service handles API requests to create/update synthetics and fires callbacks for MDConsumer which does the work of producing synethic books.
## MDStructure.h
### Hot vs Cold
## FixedSecurity.h
* variable sized structure in shared mem
* For FixedSpreadSecurity we stack allocate a lot of legs then memcpy() into shared memory only the correct number of legs
* When accessing FixedSpreadSecurity it is cast back to a large number of legs. Iterate legs based on leg_count() not the static leg count.
## SharedList.h
* Primitive list-like abstraction over shared memory
* Allows variable sized entries so always accessed in O(N)
* Data never moves so we point into it for O(1)
# Market Data Relay / Recorder
* QueryServer
- Stateless request/response
- Uses multiple threads for throughput
- Uses change timestamps per channel instead of per security
- Will over-send books that change around the time of the request
* RelayClient
- Runs in the shell (or outside)
- Replicates books from the server (QueryServer) to local shared memory
* RelayRecorder
- Records Protobufs from QueryServer to a file
# Diagnostic tools
* sparkd md dump [symbol] [type] [exchange] [hot?]
- **Primary diagnostic tool**
- ShmDumper.cpp
- Shows current contents of shared memory
- append 'hot' to the end to see hot data i.e. books
- If you get a "cannot open" check that the file exists in /run/shm/
* dumpsecurities
- Inspect security database
* sparkd md tail
- MDPeriodicDumper.cpp
- for validating new market data
- prints books at given sequence number interval
* rectool
- print recording to text / convert back to futures.coconut
- This is most useful to checking if a particular symbol is in a replay file
- Occasionally useful for testing very specific market data transitions
# Tuning / Dealing with Loss
## Packet loss
#### If a small number 1-5 is lost it's upstream
- Use tcpdump to capture
- Check the tcpdump for gaps
- Ask Pete to contact hosting provider
#### If large gap probably overflow
- make sure socket buffers sizes, rmem_max, spin_time, data_wait are all set
- check affinity is set and working
- check ethtool
#### Show ethernet ring buffer sizes
ethtool -g eth0
#### change ethernet ring buffer rx size
sudo ethtool -G [device] rx 4096
#### interface statistics
ethtool -S eth0
- particularly look at:
ethtool -S p1p1 | egrep "(bad)|(nodesc)"
### If increasing the ring buffer doesn't work you can tweak interrupt coalescing using:
- /proc/sys/net/core/dev_weight
- This is the maximum number of packets that can be dequeued in a single interrupt
- https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-network-common-queue-issues.html
- **NOTE**: We've never used this and it's unlikely it will fix anything
-83
View File
@@ -1,83 +0,0 @@
********************
classes
********************
btscmd md => PlatformLoader
PlatformLoader:
* Loads the platform specific dll
* Calls ->start_marketdata() with a full vector<security_info>
SharedList<T> / SharedListWriter
* Shared memory list object which spans across shared memory segments as necessary
* O(N) walk
* atomic add()
* Single adder / multiple readers
* no delete
* Custom header
SecurityIDRecorder
* add_security(security, lambda callback) -> int returns security id instantly
* security_persisted callback when security id is safely persisted
SecurityIndex
* Platform Implementation helper
* Maps exhange_id to (ProducerHandle + Data)
*
CacheEntry / CacheEntryDepth (CacheStructure.h)
* CacheEntry(symbol, depth, # legs) -- size determined in constructor
* Simple inheritance tree for optional depth
* Has FixedSecurity
* Fits in SharedList<T>
* CRITICAL fields are at the front cache aligned. bbo/depth. Security goes at the end
* size, is_public(), is_persisted, is_resolved (spread)
* statistics like bbo/trade counters
* is_status_ok
* bbo, depth, trade, status, open, close
FixedSecurity / FixsdSpreadSecurity
* Normalized version of our protobuf security
* Has methods to convert to pb Security
* Size known at construction w/ # legs - spread legs of type security_key
* Can use a union for option/future fields
MDProducer(symbol, type)
* manages lifetime of SharedList instances
* Hands out MDProducerHandle - entry for one security
MDProducerHandle
* pointers to shared memory
* update methods
MDConsumer
* list of securities
* find_security_id(symbol, type, etc)
* get_cache_handle(security id#)
* Hands out MDCacheHandle
- if security is null add MDCacheHandle to a list to be updated
next time securities are polled
MDCacheHandle
* If security is not present add this is "null"
* has pointers to relevant shared memory
********************
* Cache Structure
********************
2 Shared memory segment for each (SYMBOL, TYPE) - md_hot (bbo,depth,status), md_cold (others)
When a new security is discovered
1. id is allocated to the security
2. New CacheEntry is allocated in appropriate arena (is_public=false)
3. md_cache entry count is incremented
4. When id is persisted is_public=true, increment version on cache arena
+30 -19
View File
@@ -10,6 +10,7 @@
#include "lib/persistence/SqliteStore.h"
#include "lib/utils/PBUtils.h"
#include "lib/orderman/actors/SODPosition.h"
#include "lib/utils/overloaded.h"
#include <boost/date_time/posix_time/posix_time.hpp>
#include <sys/time.h>
@@ -39,6 +40,7 @@ struct FillStore::PImpl {
FillStore::FillStore(md::SecurityOracle o) : me(new PImpl(o)) {}
// returns the sod type of a non sod type fill
auto sod_fill_kind (const api::orderman::Fill& fill){
auto rv = Fill::FILL_KIND_SOD;
switch (fill.fill_kind()){
@@ -52,6 +54,7 @@ auto sod_fill_kind (const api::orderman::Fill& fill){
rv = Fill::FILL_KIND_SOD_STRIP_LEG;
break;
default:
LOG(persist, Error, "Stored fill kinds can not be SOD");
rv = Fill::FILL_KIND_SOD;
break;
}
@@ -60,24 +63,19 @@ auto sod_fill_kind (const api::orderman::Fill& fill){
auto leg_fill_kind (const api::Security& parent,
const api::Security& child){
if (parent.has_ice()){
if(securities::is_strip(parent))
return Fill::FILL_KIND_STRIP_LEG;
else{
// parent is a spread
if(securities::is_spread_of_strips(parent)){
// this has 3 levels strip spread -> strip -> outright
// first level is an ordinary leg and second is a strip leg.
// since we dont get the level passed we have to
// use the following heurestics
if(securities::is_strip(child))
return Fill::FILL_KIND_LEG;
else
return Fill::FILL_KIND_STRIP_LEG;
}
else
// regular spread
if(securities::is_strip(parent))
return Fill::FILL_KIND_STRIP_LEG;
else{
// parent is a spread
if(securities::is_spread_of_strips(parent)){
// this has 3 levels strip spread -> strip -> outright
// first level is an ordinary leg and second is a strip leg.
// since we dont get the level passed we have to
// use the following heurestics
if(securities::is_strip(child))
return Fill::FILL_KIND_LEG;
else
return Fill::FILL_KIND_STRIP_LEG;
}
}
return Fill::FILL_KIND_LEG;
@@ -167,8 +165,21 @@ void FillStore::on_execution_report(
fill.set_fill_status(api::orderman::Fill::FILL_STATUS_CORRECTED);
fill.set_fill_key(fk);
auto exchange_ts = utils::pb::to_timestamp(
datetime::parse_FIX_timestamp(to_str(er.transact_time)));
auto exchange_ts =
std::visit(overloaded {
[](const orderman::ExchangeTimeStampStr& ts) {
return utils::pb::to_timestamp(
datetime::parse_FIX_timestamp(to_str(ts)));
},
[](ExchangeTimeStamp64 ts) {
google::protobuf::Timestamp gts;
auto seconds = ts.value() / 1E9;
gts.set_seconds(seconds);
gts.set_nanos(ts.value() - (seconds * 1E9));
return gts;
}
}, er.transact_time);
*fill.mutable_timestamp() = exchange_ts;
auto it = me->known_orders.find(er.order_key);
@@ -222,7 +222,9 @@ void GLBXExecutionGateway::on_exe_report(
msg.get<fix::ManualOrderIndicator>()
? api::orderman::MANUAL_ORDER_INDICATOR_YES
: api::orderman::MANUAL_ORDER_INDICATOR_YES;
str_to(msg.get<fix::OrderID>(), &data->exchange_order_id);
str_to(msg.get<fix::OrderID>(),
& std::get<ExchangeOrderID>(
data->exchange_order_id));
if (msg.has<fix::Price>())
data->price = securities::decode_exchange_price(
@@ -261,7 +263,9 @@ void GLBXExecutionGateway::on_exe_report(
: Price(0);
if (msg.has<fix::ExecID>())
str_to(msg.get<fix::ExecID>(), &data->exchange_fill_id);
str_to(msg.get<fix::TransactTimeStr>(), &data->transact_time);
orderman::ExchangeTimeStampStr t;
str_to(msg.get<fix::TransactTimeStr>(), &t);
data->transact_time = t;
if (msg.has<fix::CustOrderHandlingInst>())
data->cust_order_handling_inst = msg.get<fix::CustOrderHandlingInst>();
@@ -280,12 +284,16 @@ void GLBXExecutionGateway::on_cancel_reject(
decode_clorid(msg.get<fix::ClOrdID>(), &ok, &sk);
std::string reject_msg;
CancelRejectReason reason = CancelRejectReason::Other;
if (msg.has<fix::Text>())
reject_msg = msg.get<fix::Text>();
else if (msg.has<fix::CXlRejReason>())
else if (msg.has<fix::CXlRejReason>()) {
if (msg.get<fix::CXlRejReason>() == 2045)
reason = CancelRejectReason::UnknownOrder;
reject_msg = fmt::format("Order cancel was rejected with error code {}",
msg.get<fix::CXlRejReason>());
me->on_cancel_reject(ok, reject_msg);
}
me->on_cancel_reject({ok, reason,reject_msg});
}
void GLBXExecutionGateway::on_connection_status_change(bool is_connected) {
@@ -18,6 +18,7 @@ class GLBXExecutionGateway : public IExecutionGateway {
void on_securities(const mercury::Update<NewSecuritiesEvent> &) override;
void on_admin_data(const mercury::Update<NewAdminDataEvent> &) override;
void send_fix(jetfix::OutFIXPtr const &fix, trace::CorID trace) override;
void send_msgs(std::vector<OrderMsg> const&, trace::CorID) override {bts_assert(false);}
OnExeReport& on_exe_report() override;
OnGwStatusChange& on_gw_status_change() override;
@@ -0,0 +1,305 @@
#include "lib/orderman/actors/GLBXiLink3ExecutionGateway.h"
#include "lib/config/config.h"
#include "lib/config/protos/orderman_cfg.pb.h"
#include "lib/domain_model/Security.h"
#include "lib/jetfix/common/OutFIX.h"
#include "lib/jetfix/store/session_name.h"
#include "lib/orderman/actors/GLBXExecutionGateway.h"
#include "lib/orderman/actors/ilink3/IL3GW.h"
#include "lib/orderman/actors/ilink3/IL3MessageFramer.h"
#include "lib/orderman/actors/ilink3/IL3Reactor.h"
#include "lib/orderman/actors/ilink3/IL3SecDB.h"
#include "lib/orderman/actors/ilink3/SessionStore.h"
#include "lib/orderman/events/NewSecuritiesEvent.h"
#include "lib/orderman/orderman_utils/fix_order_codec.h"
namespace bts {
namespace orderman {
namespace ilink3 {
enum class GLBXiLink3ExecutionGateway::State {
WAITING_FOR_SECURITIES,
PAUSED,
CONNECTING,
READY
};
struct GLBXiLink3ExecutionGateway::PImpl {
PImpl(config::JetFIX::Connection const &cfg, trace::PerfLog &log,
md::SecurityOracle o)
: state(), cfg(cfg), reactor(), start_of_pause(), on_exe_report(),
on_gw_status_change(), on_cancel_reject(), perf_log(log), db(o),
gw_name(cfg.name()), session_store(), gw(), connection(nullptr),
connection_failures(0), started(false), replay_messages(),
has_replayed(false) {
TCPEndpoint endpoint{cfg.socket_connect_host_ip(),
(ushort)cfg.socket_connect_port()};
connection = &reactor.register_connection({endpoint});
}
State state;
config::JetFIX::Connection const cfg;
IL3Reactor reactor;
IL3MessageFramer framer;
TimeStamp start_of_pause;
OnExeReport on_exe_report;
OnGwStatusChange on_gw_status_change;
orderman::OnCancelReject on_cancel_reject;
trace::PerfLog &perf_log;
IL3SecDB db;
GatewayName gw_name;
std::optional<SessionStore> session_store;
std::optional<IL3GW> gw;
IL3Connection *connection;
size_t connection_failures;
bool started;
// We need to keep this collection alive for the duration of the
// program because we will be passing pointers to ExecutionReport
// to other threads
std::vector<ExecutionReport> replay_messages;
bool has_replayed;
};
GLBXiLink3ExecutionGateway::GLBXiLink3ExecutionGateway(
config::JetFIX::Connection const &cfg, trace::PerfLog &log,
md::SecurityOracle o)
: me(new PImpl(cfg, log, o)) {
me->state = State::WAITING_FOR_SECURITIES;
}
GLBXiLink3ExecutionGateway::~GLBXiLink3ExecutionGateway() {}
void GLBXiLink3ExecutionGateway::start() {
bts_assert(!me->started);
me->started = true;
}
void GLBXiLink3ExecutionGateway::dispatch_incoming_fix_messages() {
if (me->started) {
switch (me->state) {
case State::WAITING_FOR_SECURITIES:
break;
case State::PAUSED:
on_time();
break;
case State::READY:
case State::CONNECTING:
me->reactor.do_work();
me->gw->on_time();
break;
}
}
}
void GLBXiLink3ExecutionGateway::on_time() {
bts_assert(me->state == State::PAUSED);
// If we are still paused bail early, this avoids logging too much
// or spending too much CPU
if (pause_in_effect())
return;
me->start_of_pause = Timer::now();
// If we alread exceed the maximum number of failures per run
// bail. This is a safety measure to avoid overwhelming the
// exchange in the event of a bug or operational problem that
// results in repeated connection failures.
if (max_failures_reached()) {
LOG(iLink3, Warn,
"Maximum number of connection failures reached for {}. Reboot to try "
"again",
me->gw_name.value());
return;
}
// If we reach this point we are no longer in the pause window and
// we have not exceed our connection failure limit. The only thing
// to check now is wheter we are inside the exchange's active
// hours.
auto current_session = jetfix::session_name(me->cfg);
if (current_session.empty()) {
LOG(iLink3, Info, "Outside of session in ", me->gw_name.value());
return;
}
// Yep, the stars are all aligned, lets try to connect
connect(current_session);
}
void GLBXiLink3ExecutionGateway::connect(jetfix::SessionName const &sn) {
bts_assert(me->state == State::PAUSED);
transition(State::CONNECTING);
Config c;
c.cme_session = me->cfg.sender_compid().substr(0, 3);
c.firm = me->cfg.sender_compid().substr(3);
c.failover_on_disconnect = false;
me->session_store.emplace(me->db, sn, &me->cfg);
if (!me->has_replayed) {
me->has_replayed = true;
me->replay_messages = me->session_store.value().replay();
for (auto const &er : me->replay_messages)
me->on_exe_report(er, {});
}
c.raw_message_logger =
[store = &me->session_store.value()](auto d, auto sn, auto const &msg) {
store->async_store_raw_msg(d, sn, msg);
};
me->gw.emplace(*me->connection, me->framer, &me->db,
me->session_store->starting_session_descriptor(), c,
&me->cfg);
me->gw->subscribe_cancel_reject([this](auto &cr) {
bts_assert(me->state == State::READY || me->state == State::CONNECTING);
me->on_cancel_reject(cr);
});
me->gw->subscribe_execution_reports([this](auto er) {
bts_assert(me->state == State::READY || me->state == State::CONNECTING);
me->on_exe_report(*er, {});
});
me->gw->subscribe_session(
[this](auto const &s) { me->session_store->async_update_session(s); });
me->gw->subscribe_state([this](auto gw_state) {
LOG(iLink3, Info, "Internal gateway state change in {}. State is now {}",
me->gw_name.value(),
log::CharPtrWrapper{ilink3::state_string(gw_state)});
switch (me->state) {
case State::CONNECTING:
if (gw_state == ilink3::State::READY) {
transition(State::READY);
me->on_gw_status_change(true);
} else if (gw_state == ilink3::State::ERROR) {
me->connection_failures++;
pause();
}
break;
case State::READY:
if (gw_state == ilink3::State::ERROR) {
me->connection_failures++;
pause();
me->on_gw_status_change(false);
}
break;
default:
bts_assert(false);
break;
}
});
me->gw->connect();
}
void GLBXiLink3ExecutionGateway::pause() {
me->start_of_pause = Timer::now();
me->gw.reset();
transition(State::PAUSED);
}
bool GLBXiLink3ExecutionGateway::pause_in_effect() const {
return sec_diff(Timer::now(), me->start_of_pause) <=
me->cfg.seconds_to_pause();
}
bool GLBXiLink3ExecutionGateway::max_failures_reached() const {
return me->connection_failures > me->cfg.max_connection_attempts();
}
void GLBXiLink3ExecutionGateway::on_securities(
const mercury::Update<NewSecuritiesEvent> &update) {
if (update.events) {
for (auto const &ev : *(update.events)) {
auto sec_ptr = ev.security;
me->db.add_security(*sec_ptr);
}
if (me->state == State::WAITING_FOR_SECURITIES) {
LOG(iLink3, Warn,
"Received first set of {} securities. Proceeding "
"with initialization of execution gateway {}",
update.events->size(), me->gw_name.value());
// got our security synopsis, ready to start the GW
transition(State::PAUSED);
}
}
}
void GLBXiLink3ExecutionGateway::on_admin_data(
const mercury::Update<NewAdminDataEvent> &) {}
void GLBXiLink3ExecutionGateway::send_msgs(std::vector<OrderMsg> const &msgs,
trace::CorID corid) {
if (me->state == State::READY) {
for (auto &m : msgs) {
// NewOrderSingle
auto create_ptr = std::get_if<CreateOrderMsg>(&m);
if (create_ptr) {
me->gw->send_order(*create_ptr);
continue;
}
// CancelReplace
auto cancel_replace_ptr = std::get_if<CancelReplaceOrderMsg>(&m);
if (cancel_replace_ptr) {
LOG(iLink3,Info,"Cancel Replace. CorID:{}.{} Order:{}",
log::CharPtrWrapper{corid.channel},
corid.sn,
cancel_replace_ptr->exchange_order_id);
me->gw->send_cancel_replace(*cancel_replace_ptr);
continue;
}
// Cancel
auto cancel_ptr = std::get_if<CancelOrderMsg>(&m);
if (cancel_ptr) {
me->gw->send_cancel(*cancel_ptr);
continue;
}
}
me->gw->flush();
}
}
OnExeReport &GLBXiLink3ExecutionGateway::on_exe_report() {
return me->on_exe_report;
}
OnGwStatusChange &GLBXiLink3ExecutionGateway::on_gw_status_change() {
return me->on_gw_status_change;
}
orderman::OnCancelReject &GLBXiLink3ExecutionGateway::on_cancel_reject() {
return me->on_cancel_reject;
}
log::CharPtrWrapper GLBXiLink3ExecutionGateway::state_name(State st) {
switch (st) {
case State::WAITING_FOR_SECURITIES:
return {"WAITING_FOR_SECURITIES"};
case State::PAUSED:
return {"PAUSED"};
case State::CONNECTING:
return {"CONNECTING"};
case State::READY:
return {"READY"};
}
bts_assert(false);
return {""};
}
void GLBXiLink3ExecutionGateway::transition(State new_state) {
LOG(iLink3, Info, "Transitioning {} from {} to {}", me->gw_name.value(),
state_name(me->state), state_name(new_state));
me->state = new_state;
}
} // namespace ilink3
} // namespace orderman
} // namespace bts
@@ -0,0 +1,48 @@
#pragma once
#include "IExecutionGateway.h"
#include "lib/logging/logging.h"
namespace bts {
namespace jetfix {struct SessionName;}
namespace orderman {
namespace ilink3 {
struct Session;
class GLBXiLink3ExecutionGateway : public IExecutionGateway {
public:
GLBXiLink3ExecutionGateway(config::JetFIX::Connection const &, trace::PerfLog &,
md::SecurityOracle);
virtual ~GLBXiLink3ExecutionGateway();
void start() override;
void dispatch_incoming_fix_messages() override;
void on_securities(const mercury::Update<NewSecuritiesEvent> &) override;
void on_admin_data(const mercury::Update<NewAdminDataEvent> &) override;
void send_fix(jetfix::OutFIXPtr const &, trace::CorID ) override {bts_assert(false);}
void send_msgs(std::vector<OrderMsg> const&, trace::CorID) override;
OnExeReport& on_exe_report() override;
OnGwStatusChange& on_gw_status_change() override;
OnCancelReject &on_cancel_reject() override;
private:
enum class State;
struct PImpl;
std::unique_ptr<PImpl> me;
// Time passing
void on_time();
bool pause_in_effect() const;
bool max_failures_reached()const;
static log::CharPtrWrapper state_name(State);
void transition(State);
void pause();
void connect(jetfix::SessionName const&);
};
} // namespace orderman
} // namespace bts
}
@@ -390,7 +390,9 @@ void ICEExecutionGateway::on_exe_report(const jetfix::fix::ICEExecutionReportMsg
data->manual_order_indicator = api::orderman::MANUAL_ORDER_INDICATOR_NO;
std::string oid_string = get_ice_order_id_string(msg);
str_to(oid_string, &data->exchange_order_id);
str_to(oid_string,
& std::get<ExchangeOrderID>(
data->exchange_order_id));
data->ice_cl_ord_id = 0;
if(msg.has<fix::Int_ClOrdID>())
data->ice_cl_ord_id = msg.get<fix::Int_ClOrdID>();
@@ -438,8 +440,10 @@ void ICEExecutionGateway::on_exe_report(const jetfix::fix::ICEExecutionReportMsg
if (msg.has<fix::TransactTimeStr>())
{
std::string stamp = msg.get<fix::TransactTimeStr>();
stamp.resize(sizeof(mercury::TimeStampStr));
str_to(stamp, &data->transact_time);
stamp.resize(sizeof(orderman::ExchangeTimeStampStr));
orderman::ExchangeTimeStampStr t;
str_to(stamp, &t);
data->transact_time = t;
}
data->cfg = &me->cfg;
data->msg = text_msg;
@@ -458,7 +462,7 @@ void ICEExecutionGateway::on_cancel_reject(const jetfix::fix::ICECancelRejectMsg
else if (msg.has<fix::CXlRejReason>())
reject_msg = fmt::format("Order cancel was rejected with error code {}",
msg.get<fix::CXlRejReason>());
me->on_cancel_reject(ok, reject_msg);
me->on_cancel_reject(CancelReject {ok, CancelRejectReason::Other,reject_msg});
}
@@ -17,6 +17,7 @@ class ICEExecutionGateway : public IExecutionGateway {
void on_securities(const mercury::Update<NewSecuritiesEvent> &) override;
void on_admin_data(const mercury::Update<NewAdminDataEvent> &) override;
void send_fix(const jetfix::OutFIXPtr& fix, trace::CorID trace) override;
void send_msgs(std::vector<OrderMsg> const&, trace::CorID) override {bts_assert(false);}
OnExeReport& on_exe_report() override;
OnGwStatusChange& on_gw_status_change() override;
+12 -5
View File
@@ -1,17 +1,24 @@
#include "IExecutionGateway.h"
#include "ICEExecutionGateway.h"
#include "GLBXExecutionGateway.h"
#include "ICEExecutionGateway.h"
#include "lib/utils/ValueWrapper.h"
#include "lib/orderman/actors/GLBXiLink3ExecutionGateway.h"
#include <variant>
namespace bts {
namespace orderman {
std::unique_ptr<IExecutionGateway>
ExecutionGatewayFactory::create_gateway(const config::JetFIX::Connection& conn, trace::PerfLog& pl, md::SecurityOracle o)
{
if(conn.platform() == bts::config::JetFIX_Connection_JetFIXPlatform_ICE ||
conn.platform() == bts::config::JetFIX_Connection_JetFIXPlatform_SIMICE)
ExecutionGatewayFactory::create_gateway(const config::JetFIX::Connection &conn,
trace::PerfLog &pl,
md::SecurityOracle o) {
if (conn.platform() == bts::config::JetFIX_Connection_JetFIXPlatform_ICE ||
conn.platform() == bts::config::JetFIX_Connection_JetFIXPlatform_SIMICE)
return std::make_unique<ICEExecutionGateway>(conn, pl, o);
else if (conn.use_ilink3())
return std::make_unique<ilink3::GLBXiLink3ExecutionGateway>(conn, pl, o);
else
return std::make_unique<GLBXExecutionGateway>(conn, pl, o);
}
+34 -2
View File
@@ -9,6 +9,7 @@
#include "lib/utils/bts_no_copy.h"
#include <memory>
#include <variant>
namespace bts {
namespace trace {struct CorID;class PerfLog;}
@@ -16,11 +17,42 @@ namespace orderman {
class NewSecuritiesEvent;
class NewAdminDataEvent;
class ExecutionReport;
class CancelReject;
struct CreateOrderMsg {
OrderKey key;
Price price;
Quantity quantity;
api::Security const *security;
api::Side side;
Price stop_px;
Quantity display_quantity;
api::orderman::NativeOrderType order_type;
api::orderman::TimeInForce time_in_force;
api::orderman::ManualOrderIndicator manual_order_indicator;
api::orderman::RoutingRecord const *routing_record;
};
struct CancelReplaceOrderMsg {
CreateOrderMsg create_fields;
ExchangeOrderID64 exchange_order_id;
};
struct CancelOrderMsg {
OrderKey key;
ExchangeOrderID64 exchange_order_id;
api::orderman::ManualOrderIndicator manual_order_indicator;
api::Security const *security;
api::orderman::RoutingRecord const *routing_record;
api::Side side;
};
using OrderMsg =
std::variant<CreateOrderMsg, CancelReplaceOrderMsg, CancelOrderMsg>;
using OnGwStatusChange = std::function<void(bool)>;
using OnExeReport = std::function<void(ExecutionReport const &, trace::CorID const &)>;
using OnCancelReject = std::function<void(OrderKey, std::string const &)>;
using OnCancelReject = std::function<void(CancelReject const&)>;
// fix execution gateway for a single FIX session
class IExecutionGateway {
@@ -33,7 +65,7 @@ class IExecutionGateway {
virtual void on_securities(const mercury::Update<NewSecuritiesEvent> &) = 0;
virtual void on_admin_data(const mercury::Update<NewAdminDataEvent> &) = 0;
virtual void send_fix(const jetfix::OutFIXPtr&, trace::CorID) = 0;
virtual void send_msgs(std::vector<OrderMsg> const&, trace::CorID) = 0;
virtual OnExeReport& on_exe_report() = 0;
virtual OnGwStatusChange& on_gw_status_change() = 0;
virtual OnCancelReject &on_cancel_reject() = 0;
+88 -73
View File
@@ -1,5 +1,6 @@
#pragma once
#include "lib/domain_model/Security.h"
#include "lib/logging/logging.h"
#include "lib/md/IntPrice.h"
#include "lib/md/MDConsumer.h"
@@ -10,12 +11,11 @@
#include "lib/mercury/events/TimerEvent.h"
#include "lib/orderman/events/BookUpdateEvent.h"
#include "lib/orderman/events/MDRequestEvent.h"
#include "lib/orderman/events/TickerUpdateEvent.h"
#include "lib/orderman/events/StatusUpdateEvent.h"
#include "lib/orderman/events/TickerUpdateEvent.h"
#include "lib/tracing/PerfLog.h"
#include "lib/tracing/bts_trace.h"
#include "lib/utils/TimeStamp.h"
#include "lib/domain_model/Security.h"
#include <date/date.h>
@@ -31,9 +31,9 @@ class MDSource : public mercury::Actor, public mercury::IPollableActor {
public:
MDSource(md::MDConsumer &consumer, metrics::MetricsContainer &stats,
trace::PerfLog &log)
: _handles(), _pending_handles(), _status_subs(), _md_consumer(consumer),
_book_update(), _ticker_update(), _status_update(),
_perf_log(log), _stats(stats), _loop_timer(), _counter(0) {
: _handles(), _pending_handles(), _status_subs(), _md_consumer(consumer),
_book_update(), _ticker_update(), _status_update(), _perf_log(log),
_stats(stats), _loop_timer(), _counter(0) {
_trades_buffer.Reserve(1000);
auto cme_channels = md::get_configured_cme_channels();
@@ -67,9 +67,8 @@ class MDSource : public mercury::Actor, public mercury::IPollableActor {
auto ticker_channel_cout_metric =
std::make_shared<metrics::CounterMetric<int>>(
"spark_mdsource_ticker_channel_count");
auto status_count_metric =
std::make_shared<metrics::CounterMetric<int>>(
"spark_mdsource_status_count");
auto status_count_metric = std::make_shared<metrics::CounterMetric<int>>(
"spark_mdsource_status_count");
_stats.publish(security_count_metric);
_stats.publish(synthetic_count_metric);
_stats.publish(ticker_channel_cout_metric);
@@ -78,37 +77,40 @@ class MDSource : public mercury::Actor, public mercury::IPollableActor {
_book_update = make_stream<BookUpdateEvent>();
_ticker_update = make_stream<TickerUpdateEvent>();
_status_update = make_stream<StatusUpdateEvent>();
subscribe<MDRequestEvent>(
[this, security_count_metric, synthetic_count_metric,
ticker_channel_cout_metric, status_count_metric](auto const &update) {
if (update.events) {
for (auto const &ev : *update.events) {
auto k = ev.security_key;
subscribe<MDRequestEvent>([this, security_count_metric,
synthetic_count_metric,
ticker_channel_cout_metric,
status_count_metric](auto const &update) {
if (update.events) {
for (auto const &ev : *update.events) {
auto k = ev.security_key;
switch (ev.md_type) {
case MDRequestType::BOOK:
on_book_sub(k, *synthetic_count_metric);
security_count_metric->set(_handles.size());
break;
case MDRequestType::TICKER:
on_ticker_sub(k, *ticker_channel_cout_metric);
break;
case MDRequestType::STATUS:
on_status_sub(k);
status_count_metric->set(_status_subs.size());
break;
}
}
}
});
switch (ev.md_type) {
case MDRequestType::BOOK:
on_book_sub(k, *synthetic_count_metric);
security_count_metric->set(_handles.size());
break;
case MDRequestType::TICKER:
on_ticker_sub(k, *ticker_channel_cout_metric);
break;
case MDRequestType::STATUS:
on_status_sub(k);
status_count_metric->set(_status_subs.size());
break;
}
}
}
});
security_count_metric->set(_handles.size());
}
private:
void on_poll(trace::PerfLog& log) override {
void on_poll(trace::PerfLog &log) override {
auto scoped_do_work = trace::log_scope(log, {}, "MDSource::on_poll", utils::TimeStamp::Interval::from_microseconds(2));
auto scoped_do_work =
trace::log_scope(log, {}, "MDSource::on_poll",
utils::TimeStamp::Interval::from_microseconds(2));
// Book Updates
Book book;
@@ -158,11 +160,11 @@ class MDSource : public mercury::Actor, public mercury::IPollableActor {
}
// Status updates
for (auto& s : _status_subs) {
for (auto &s : _status_subs) {
auto status = s.handle.trading_status();
if (status != s.status) {
s.status = status;
publish(_status_update, SecurityKey { s.handle.key() }, status);
publish(_status_update, SecurityKey{s.handle.key()}, status);
}
}
@@ -171,7 +173,7 @@ class MDSource : public mercury::Actor, public mercury::IPollableActor {
_loop_timer->observe((start - stop).nanoseconds());
_counter += 1;
if (_counter % (2<<20) == 0) {
if (_counter % (2 << 20) == 0) {
// Invoke periodic work every 2M calls on on_poll
// This is approximately 20-30 seconds
do_periodic_work();
@@ -180,35 +182,39 @@ class MDSource : public mercury::Actor, public mercury::IPollableActor {
void do_periodic_work() {
if (!_pending_handles.empty()) {
for (auto iter = _pending_handles.begin(); iter != _pending_handles.end(); ++iter) {
for (auto iter = _pending_handles.begin();
iter != _pending_handles.end(); ++iter) {
if (iter->handle->is_initialized()) {
auto handle = *(iter->handle);
LOG(OrderMan, Info, "Found md for {}. Pending handle removed.", handle.key());
switch(iter->type) {
case MDRequestType::BOOK:
_handles.push_back({utils::TimeStamp(), handle,
get_external_event_channel(handle.get_usersymbol()),
std::make_unique<std::string>(securities::descriptor(
handle.get_normalized_security()))});
break;
case MDRequestType::STATUS:
for (auto& h : _status_subs) {
// no need to reset status
if (h.handle.key() == handle.key()) break;
}
_status_subs.push_back(
StatusRecord{ handle, co2::TRADINGSTATUS_UNKNOWN });
case MDRequestType::TICKER:
LOG(OrderMan, Info, "Found md for {}. Pending handle removed.",
handle.key());
switch (iter->type) {
case MDRequestType::BOOK:
_handles.push_back(
{utils::TimeStamp(), handle,
get_external_event_channel(handle.get_usersymbol()),
std::make_unique<std::string>(securities::descriptor(
handle.get_normalized_security()))});
break;
case MDRequestType::STATUS:
for (auto &h : _status_subs) {
// no need to reset status
if (h.handle.key() == handle.key())
break;
}
_status_subs.push_back(
StatusRecord{handle, co2::TRADINGSTATUS_UNKNOWN});
case MDRequestType::TICKER:
// We aren't currently monitoring for pending ticker requests
break;
}
}
}
_pending_handles.erase(
std::remove_if(_pending_handles.begin(),
_pending_handles.end(),
[](const auto& h) { return h.handle->is_initialized(); }),
_pending_handles.end());
std::remove_if(
_pending_handles.begin(), _pending_handles.end(),
[](const auto &h) { return h.handle->is_initialized(); }),
_pending_handles.end());
}
}
@@ -229,18 +235,18 @@ class MDSource : public mercury::Actor, public mercury::IPollableActor {
auto mhandle = _md_consumer.get_managed_handle(k);
if (mhandle->is_initialized()) {
auto handle = *mhandle;
_handles.push_back({utils::TimeStamp(), handle,
get_external_event_channel(handle.get_usersymbol()),
std::make_unique<std::string>(securities::descriptor(
handle.get_normalized_security()))});
_handles.push_back(
{utils::TimeStamp(), handle,
get_external_event_channel(handle.get_usersymbol()),
std::make_unique<std::string>(
securities::descriptor(handle.get_normalized_security()))});
if (handle.is_synthetic())
synthetic_count_metric.increment();
}
else {
// A pending handle is a handle for security we are reasonbly sure exists
// but we haven't yet found live market data for.
} else {
// A pending handle is a handle for security we are reasonbly sure
// exists but we haven't yet found live market data for.
LOG(OrderMan, Warn, "No market data for {}. Pending handle added.", k);
_pending_handles.push_back({ mhandle, MDRequestType::BOOK });
_pending_handles.push_back({mhandle, MDRequestType::BOOK});
}
}
@@ -253,8 +259,17 @@ class MDSource : public mercury::Actor, public mercury::IPollableActor {
void on_ticker_sub(SecurityKey sk,
metrics::CounterMetric<int> &channel_count_metric) {
auto handle = _md_consumer.get_managed_handle(sk.value());
auto user_symbol = _md_consumer.get_handle(sk.value()).get_usersymbol();
if (!handle->is_initialized()) {
LOG(OrderMan, Warn,
"Received ticker subscription for security {} which does not have "
"valid market data. Ignoring",
sk.value());
return;
}
auto user_symbol = handle->get_usersymbol();
for (auto &ch : _channels) {
if (ch.user_symbol == user_symbol) {
auto it =
@@ -262,7 +277,7 @@ class MDSource : public mercury::Actor, public mercury::IPollableActor {
[sk](auto const &ev) { return ev.key() == sk; });
if (it == ch.securities.end()) {
// first time we are asked for this security, we will
// publish the trades for it next time next round
// publish the trades for it next time around
LOG(OrderMan, Warn,
"Starting to publish ticker updates on user symbol {} for "
"security key: {} (sec count for the symbol: {} )",
@@ -294,7 +309,7 @@ class MDSource : public mercury::Actor, public mercury::IPollableActor {
void on_status_sub(SecurityKey sk) {
auto k = sk.value();
for (auto& h : _status_subs) {
for (auto &h : _status_subs) {
if (h.handle.key() == k) {
// reset status so it get resent next on_poll
h.status = co2::TRADINGSTATUS_UNKNOWN;
@@ -303,11 +318,11 @@ class MDSource : public mercury::Actor, public mercury::IPollableActor {
}
auto handle = _md_consumer.get_managed_handle(k);
if (handle->is_initialized()) {
_status_subs.push_back(StatusRecord{ *handle, co2::TRADINGSTATUS_UNKNOWN });
}
else {
_status_subs.push_back(
StatusRecord{*handle, co2::TRADINGSTATUS_UNKNOWN});
} else {
LOG(OrderMan, Warn, "No market data for {}. Pending handle added.", k);
_pending_handles.push_back({ handle, MDRequestType::STATUS });
_pending_handles.push_back({handle, MDRequestType::STATUS});
}
}
@@ -337,7 +352,7 @@ class MDSource : public mercury::Actor, public mercury::IPollableActor {
md::MDConsumerHandle handle;
co2::TradingStatus status;
};
boost::container::small_vector<StatusRecord,20> _status_subs;
boost::container::small_vector<StatusRecord, 20> _status_subs;
md::MDConsumer _md_consumer;
mercury::Stream<BookUpdateEvent> _book_update;
+4
View File
@@ -101,6 +101,8 @@ void OrderStore::on_order_state_update(
for (auto const &ev : *u.events) {
auto &rec_info = me->known_orders[ev.key];
LOG(iLink3,Info, "Received order state update for {}",
ev.key.value());
auto &rec = rec_info.rec;
bts_assert(rec.has_def());
rec.set_key(ev.key.value());
@@ -123,6 +125,8 @@ void OrderStore::on_order_definition_update(
std::set<OrderKey> modified_orders;
if (u.events)
for (auto const &ev : *u.events) {
LOG(iLink3,Info, "Received order definition update for {}",
ev.key.value());
auto &rec_info = me->known_orders[ev.key];
auto &rec = rec_info.rec;
rec.set_key(ev.key.value());
@@ -0,0 +1,20 @@
#pragma once
#include "IL3ConnectionHandler.h"
#include <string>
#include <vector>
namespace bts {
namespace orderman {
namespace ilink3 {
struct IL3Connection {
virtual ~IL3Connection() = default;
virtual void connect(IL3ConnectionHandler&) = 0;
virtual void disconnect() = 0;
// Switch connection but do not connect()
virtual void failover() = 0;
virtual void send(const std::string_view) = 0;
virtual void send(const std::vector<std::string_view>&) = 0;
};
}}}//namespace
@@ -0,0 +1,10 @@
#include "IL3ConnectionHandler.h"
namespace bts {
namespace orderman {
namespace ilink3 {
IL3ConnectionHandler::~IL3ConnectionHandler() = default;
}}}
@@ -0,0 +1,17 @@
#pragma once
#include <cstddef>
#include <string>
namespace bts {
namespace orderman {
namespace ilink3 {
struct IL3ConnectionHandler {
virtual ~IL3ConnectionHandler() ;
virtual void on_connect_success() = 0;
virtual void on_disconnect() = 0;
// Send available data and length. Handler will return what it consumed
virtual size_t on_data(const std::string_view&) = 0;
};
}}}//namespace
@@ -0,0 +1,140 @@
#pragma once
#include <cstdint>
#include "IL3GeneratedFull.h"
#include "IL3MessageFramer.h"
namespace bts {
namespace orderman {
namespace ilink3 {
template <class Handler, class T>
void handle(Handler& h, const char* data, uint16_t len) {
const FramingHeader* fheader = reinterpret_cast<const FramingHeader*>(data);
iLinkBinary::MessageHeader mheader {
(char*)data, sizeof(FramingHeader), fheader->message_size,
iLinkBinary::MessageHeader::sbeSchemaVersion() };
T t {
const_cast<char*>(data),
sizeof(FramingHeader) + iLinkBinary::MessageHeader::encodedLength(),
len,
mheader.blockLength(),
mheader.version()};
h.handle(t);
}
template <class H>
void dispatch_message(H& handler, const FramingHeader* header, const char* data) {
using namespace iLinkBinary;
bts_assert_d(header->encoding_type == 0XCAFE);
MessageHeader mheader {
(char*)data, sizeof(FramingHeader), header->message_size,
MessageHeader::sbeSchemaVersion() };
switch(mheader.templateId()) {
case 501: handle<H, NegotiationResponse501>(handler, data, header->message_size); break;
case 502: handle<H, NegotiationReject502>(handler, data, header->message_size); break;
case 504: handle<H, EstablishmentAck504>(handler, data, header->message_size); break;
case 505: handle<H, EstablishmentReject505>(handler, data, header->message_size); break;
case 506: handle<H, Sequence506>(handler, data, header->message_size); break;
case 507: handle<H, Terminate507>(handler, data, header->message_size); break;
case 509: handle<H, Retransmission509>(handler, data, header->message_size); break;
case 510: handle<H, RetransmitReject510>(handler, data, header->message_size); break;
case 513: handle<H, NotApplied513>(handler, data, header->message_size); break;
case 519: handle<H, PartyDetailsDefinitionRequestAck519>(handler, data, header->message_size); break;
case 521: handle<H, BusinessReject521>(handler, data, header->message_size); break;
case 522: handle<H, ExecutionReportNew522>(handler, data, header->message_size); break;
case 523: handle<H, ExecutionReportReject523>(handler, data, header->message_size); break;
case 524: handle<H, ExecutionReportElimination524>(handler, data, header->message_size); break;
case 525: handle<H, ExecutionReportTradeOutright525>(handler, data, header->message_size); break;
case 526: handle<H, ExecutionReportTradeSpread526>(handler, data, header->message_size); break;
case 527: handle<H, ExecutionReportTradeSpreadLeg527>(handler, data, header->message_size); break;
case 531: handle<H, ExecutionReportModify531>(handler, data, header->message_size); break;
case 532: handle<H, ExecutionReportStatus532>(handler, data, header->message_size); break;
case 534: handle<H, ExecutionReportCancel534>(handler, data, header->message_size); break;
case 535: handle<H, OrderCancelReject535>(handler, data, header->message_size); break;
case 536: handle<H, OrderCancelReplaceReject536>(handler, data, header->message_size); break;
// case 538: handle<H, PartyDetailsListReport538>(handler, data, header->message_size); break;
// The Execution Acknowledgment message notifies CME that a given fill has either been accepted or rejected by a counterparty for EU bilateral trades.
// case 539: handle<H, ExecutionAck539>(handler, data, header->message_size); break;
// case 544: handle<H, NewOrderCross544>(handler, data, header->message_size); break;
// case 545: handle<H, MassQuoteAck545>(handler, data, header->message_size); break;
// case 546: handle<H, RequestForQuoteAck546>(handler, data, header->message_size); break;
// case 547: handle<H, QuoteCancelAck547>(handler, data, header->message_size); break;
case 548: handle<H, ExecutionReportTradeAddendumOutright548>(handler, data, header->message_size); break;
case 549: handle<H, ExecutionReportTradeAddendumSpread549>(handler, data, header->message_size); break;
case 550: handle<H, ExecutionReportTradeAddendumSpreadLeg550>(handler, data, header->message_size); break;
// case 558: handle<H, iLinkBinary::OrderMassActionReport558>(handler, data, header->message_size); break;
// case 561: handle<H, iLinkBinary::SecurityDefinitionResponse561>(handler, data, header->message_size); break;
}
}
template <class H>
void dispatch_execution_reports(H& handler, const FramingHeader* header, const char* data) {
using namespace iLinkBinary;
bts_assert_d(header->encoding_type == 0XCAFE);
MessageHeader mheader {
(char*)data, sizeof(FramingHeader), header->message_size,
MessageHeader::sbeSchemaVersion() };
switch(mheader.templateId()) {
case 522: handle<H, ExecutionReportNew522>(handler, data, header->message_size); break;
case 523: handle<H, ExecutionReportReject523>(handler, data, header->message_size); break;
case 524: handle<H, ExecutionReportElimination524>(handler, data, header->message_size); break;
case 525: handle<H, ExecutionReportTradeOutright525>(handler, data, header->message_size); break;
case 526: handle<H, ExecutionReportTradeSpread526>(handler, data, header->message_size); break;
case 527: handle<H, ExecutionReportTradeSpreadLeg527>(handler, data, header->message_size); break;
case 531: handle<H, ExecutionReportModify531>(handler, data, header->message_size); break;
case 532: handle<H, ExecutionReportStatus532>(handler, data, header->message_size); break;
case 534: handle<H, ExecutionReportCancel534>(handler, data, header->message_size); break;
}
}
// This handler handles exchange messages for testing
template <class H>
void dispatch_exchange_message(H& handler, const FramingHeader* header, const char* data) {
using namespace iLinkBinary;
bts_assert_d(header->encoding_type == 0XCAFE);
MessageHeader mheader {
(char*)data, sizeof(FramingHeader), header->message_size,
MessageHeader::sbeSchemaVersion() };
// Rather than trying to be exhaustive like the client
// we will only add handlers for the messages we use for testing
switch(mheader.templateId()) {
case 500: handle<H, Negotiate500>(handler, data, header->message_size); break;
case 503: handle<H, Establish503>(handler, data, header->message_size); break;
case 506: handle<H, Sequence506>(handler, data, header->message_size); break;
case 508: handle<H, RetransmitRequest508>(handler, data, header->message_size); break;
case 518: handle<H, PartyDetailsDefinitionRequest518>(handler, data, header->message_size); break;
case 514: handle<H, NewOrderSingle514>(handler, data, header->message_size); break;
case 515: handle<H, OrderCancelReplaceRequest515>(handler, data, header->message_size); break;
case 516: handle<H, OrderCancelRequest516>(handler, data, header->message_size); break;
}
}
template <class Handler, class Dispatcher>
size_t consume_messages(
Handler& handler, const char* data, size_t len, Dispatcher dispatch) {
// void(*dispatch)(Handler&, const FramingHeader*, const char*)) {
size_t read = 0;
while(len > sizeof(FramingHeader)) {
// read the FramingHeader
const FramingHeader* header = reinterpret_cast<const FramingHeader*>(data);
// Check if we can consume the message
auto msize = header->message_size;
if (msize <= len) {
read += msize;
dispatch(handler, header, data);
data += msize;
len -= msize;
}
else {
return read;
}
}
return read;
}
}}}//namespace
+662
View File
@@ -0,0 +1,662 @@
#include "IL3GW.h"
#include "IL3Dispatcher.h"
#include "IL3GeneratedFull.h"
#include "IL3MessageFramer.h"
#include "ILink3Utils.h"
#include "lib/orderman/orderman_utils/fix_order_codec.h"
#include "lib/utils/SecureLogon.h"
#include "lib/utils/bts_assert.h"
#include "lib/utils/datetime.h"
/*
# Some notes about session binding and initialization: #
## Official documentation:
https://www.cmegroup.com/confluence/display/EPICSANDBOX/iLink+3+Binary+Order+Entry+-+Session+Layer#iLink3BinaryOrderEntry-SessionLayer-WeeklyReset
## Begining of week (new session?)
- config.session = X
- config.outgoing_sequence = 1
-> Negotiate session=X
<- NegotiateResponse session=X, oldsession=Y, oldSeq=N
if oldSeq > 0 -> ReplayRequest
-> Establish seq=1
<- EstablishAck oldsession=Y, oldSeq=N
## Middle of week (same session)
- config.session = X
- config.outgoing_sequence > 1
-> Establish seq=N
*/
namespace bts {
namespace orderman {
namespace ilink3 {
log::CharPtrWrapper wrap(const char *str) { return log::CharPtrWrapper{str}; }
IL3GW::IL3GW(IL3Connection &con, IL3MessageFramer &buffer, IL3SecDB *db,
const Session &session, const Config &conf,
config::JetFIX::Connection const *cfg)
: _config(conf), _x_state(State::INIT), _buffer(buffer),
_on_state_change([](auto) {}), _on_exe_report([](auto *) {}),
_on_session_change([](auto) {}), _connection(con), _session(session),
_msg_count(0), _time(), _pool(), _exe_parser(cfg, db), _retransmit() {
_clorid_buffer.resize(clordid_sz());
}
void IL3GW::connect() {
bts_assert(state() == State::INIT || state() == State::ERROR);
set_state(State::CONNECTING);
_connection.connect(*this); // -> on_connect_success
}
// This determines the session uuid
void IL3GW::send_negotiate() {
// add 2 bytes for the empty 'credentials' (39008) length which must be
// present
auto msg = _buffer.frame<iLinkBinary::Negotiate500>(
iLinkBinary::Negotiate500::credentialsHeaderLength());
msg.uUID(_session.uuid);
msg.requestTimestamp(_time.now());
msg.putSession(_config.cme_session);
// We need this call to set the length of the credential block to 0
// This behavior is specific to "data" in the xml template
msg.putCredentials("", 0);
msg.putFirm(_config.firm);
const auto ld = glbx::calculate_hmac(
_config.cme_session, std::to_string(msg.requestTimestamp()),
std::to_string(msg.uUID()), _config.cme_session, _config.firm);
msg.putAccessKeyID(ld.access_key);
msg.putHMACSignature(ld.hmac);
std::string temp = ld.canonical;
for (size_t x = 0; x < temp.size(); ++x) {
if (temp[x] == '\n')
temp[x] = '$';
}
flush(false);
// clear HMAC for json logging
msg.putHMACSignature(std::string_view{nullptr, 0});
log_out(msg, 0);
// expect -> NegotiationResponse501 or NegotiationReject502
}
void IL3GW::send_establish() {
// add 2 bytes for the empty 'credentials' (39008) length which must be
// present
auto msg = _buffer.frame<iLinkBinary::Establish503>(2);
msg.uUID(_session.uuid);
msg.requestTimestamp(_time.now());
msg.putSession(_config.cme_session);
msg.putCredentials("", 0);
msg.putFirm(_config.firm);
msg.putTradingSystemName(std::string_view("SPARK"));
msg.putTradingSystemVersion(std::string_view("1"));
msg.putTradingSystemVendor(std::string_view("BTS"));
// On primary connection, the expected message sequence number of the next
// business message from customer to CME
msg.nextSeqNo(_session.to_cme);
msg.keepAliveInterval(30000);
const auto ld =
glbx::calculate_hmac(_config.cme_session,
// RequestTimestamp
std::to_string(msg.requestTimestamp()),
std::to_string(msg.uUID()), _config.cme_session,
// Firm ID
_config.firm,
// TradingSystemName
msg.getTradingSystemNameAsString(),
// TradingSystemVersion
msg.getTradingSystemVersionAsString(),
// TradingSystemVendor
msg.getTradingSystemVendorAsString(),
// NextSeqNo
std::to_string(msg.nextSeqNo()),
// KeepAliveInterval
std::to_string(msg.keepAliveInterval()));
msg.putAccessKeyID(ld.access_key);
msg.putHMACSignature(ld.hmac);
flush(false);
// clear HMAC for json logging
msg.putHMACSignature(std::string_view{nullptr, 0});
log_out(msg, 0);
// expect -> EstablishmentAck504 or EstablishmentReject505
}
void IL3GW::send_sequence(bool lapsed) {
auto msg = _buffer.frame<iLinkBinary::Sequence506>();
msg.uUID(_session.uuid);
msg.nextSeqNo(_session.to_cme);
msg.faultToleranceIndicator(iLinkBinary::FTI::NULL_VALUE);
msg.keepAliveIntervalLapsed(lapsed
? iLinkBinary::KeepAliveLapsed::Lapsed
: iLinkBinary::KeepAliveLapsed::NotLapsed);
log_out(msg, 0);
}
void IL3GW::send_retransmit_request(uint64_t uuid, uint32_t from,
size_t count) {
bts_assert(from > 0);
using namespace iLinkBinary;
auto msg = _buffer.frame<RetransmitRequest508>();
msg.uUID(_session.uuid);
msg.lastUUID(uuid);
msg.requestTimestamp(_time.now());
msg.fromSeqNo(from);
msg.msgCount(count);
log_out(msg, 0);
}
uint64_t IL3GW::send_party_details(const api::orderman::RoutingRecord &route) {
using namespace iLinkBinary;
auto msg = _buffer.frame<PartyDetailsDefinitionRequest518>(
// 2 repeating group header.
// 1 for party details and 1 for trade reg
// This seems unsafe but the api doesn't seem to want you to know how to
// size buffers ahead of time
2 * PartyDetailsDefinitionRequest518::NoPartyDetails::sbeHeaderSize() +
// N * size of each block
3 * PartyDetailsDefinitionRequest518::NoPartyDetails::sbeBlockLength() +
0 * PartyDetailsDefinitionRequest518::NoTrdRegPublications::
sbeBlockLength());
msg.partyDetailsListReqID(0);
msg.sendingTimeEpoch(_time.now());
msg.listUpdateAction(ListUpdAct::Add);
msg.seqNum(next_seq());
// msg.putMemo("") -- not supported
if (route.smp_id().empty()) {
msg.selfMatchPreventionID(
PartyDetailsDefinitionRequest518::selfMatchPreventionIDNullValue());
} else {
auto smp_id = std::atoi(route.smp_id().c_str());
msg.selfMatchPreventionID(smp_id);
}
msg.cmtaGiveupCD(CmtaGiveUpCD::NULL_VALUE);
msg.custOrderCapacity(map_cust_cti(route.cti_code()));
msg.clearingAccountType(map_customer_or_firm(route.customer_or_firm()));
msg.selfMatchPreventionInstruction(
map_smp_instruction(route.smp_instruction()));
msg.avgPxIndicator(AvgPxInd::NULL_VALUE);
msg.clearingTradePriceType(SLEDS::NULL_VALUE);
msg.custOrderHandlingInst(static_cast<iLinkBinary::CustOrdHandlInst::Value>(
route.cust_order_handing_inst()[0]));
msg.executor(PartyDetailsDefinitionRequest518::executorNullValue());
msg.iDMShortCode(PartyDetailsDefinitionRequest518::iDMShortCodeNullValue());
auto details = msg.noPartyDetailsCount(3);
details.next();
details.partyDetailRole(PartyDetailRole::CustomerAccount);
details.putPartyDetailID(route.account());
details.next();
details.partyDetailRole(PartyDetailRole::ExecutingFirm);
// 004 in cert
details.putPartyDetailID(
std::string_view(route.sender_compid().c_str() + 3, 3));
details.next();
details.partyDetailRole(PartyDetailRole::Operator);
details.putPartyDetailID(std::string_view(route.sender_compid().c_str(), 3));
msg.noTrdRegPublicationsCount(0);
// In the future we may check for existing party details definitions and
// return that id
log_out(msg, msg.seqNum());
return 0;
}
bool needs_stop_px(api::orderman::NativeOrderType order_type) {
return order_type ==
api::orderman::NativeOrderType::NATIVE_ORDER_TYPE_STOP ||
order_type ==
api::orderman::NativeOrderType::NATIVE_ORDER_TYPE_STOP_LIMIT;
}
bool need_px(api::orderman::NativeOrderType order_type) {
return order_type !=
api::orderman::NativeOrderType::NATIVE_ORDER_TYPE_MARKET &&
order_type !=
api::orderman::NativeOrderType::NATIVE_ORDER_TYPE_MARKET_LIMIT &&
order_type != api::orderman::NativeOrderType::NATIVE_ORDER_TYPE_STOP;
}
void IL3GW::send_order(const CreateOrderMsg &create) {
bts_assert_d(create.routing_record);
bts_assert_d(create.security);
bts_assert_d(create.security->has_glbx());
auto party = send_party_details(*create.routing_record);
auto msg = _buffer.frame<iLinkBinary::NewOrderSingle514>();
if (need_px(create.order_type)) {
msg.price().mantissa(to_price9(create.price));
} else {
msg.price().mantissa(iLinkBinary::PRICENULL9::mantissaNullValue());
}
msg.orderQty(create.quantity.value());
msg.securityID(create.security->glbx().exchange_id());
msg.side(map_side(create.side));
msg.seqNum(next_seq());
msg.putSenderID(create.routing_record->operator_id());
encode_clorid(create.key, SecurityKey(create.security->id()),
&_clorid_buffer);
msg.putClOrdID(_clorid_buffer);
msg.partyDetailsListReqID(party);
msg.orderRequestID(++_msg_count);
msg.sendingTimeEpoch(_time.now());
if (needs_stop_px(create.order_type)) {
msg.stopPx().mantissa(to_price9(create.stop_px));
} else {
msg.stopPx().mantissa(iLinkBinary::PRICENULL9::mantissaNullValue());
}
msg.putLocation(create.routing_record->location());
msg.minQty(iLinkBinary::NewOrderSingle514::minQtyNullValue());
if (create.time_in_force != api::orderman::TimeInForce::TIME_IN_FORCE_IOC) {
msg.displayQty(create.display_quantity.value());
} else {
msg.displayQty(iLinkBinary::NewOrderSingle514::displayQtyNullValue());
}
if (create.time_in_force == api::orderman::TimeInForce::TIME_IN_FORCE_DATE) {
// TODO: # of days since jan 1 1970
msg.expireDate(18424);
} else {
msg.expireDate(iLinkBinary::NewOrderSingle514::expireDateNullValue());
}
msg.ordType(map_order_type(create.order_type));
msg.timeInForce(map_time_in_force(create.time_in_force));
msg.manualOrderIndicator(
map_manual_order_indicator(create.manual_order_indicator));
msg.execInst().clear();
msg.executionMode(iLinkBinary::ExecMode::NULL_VALUE);
msg.liquidityFlag(iLinkBinary::BooleanNULL::NULL_VALUE);
msg.managedOrder(iLinkBinary::BooleanNULL::NULL_VALUE);
msg.shortSaleType(iLinkBinary::ShortSaleType::NULL_VALUE);
log_out(msg, msg.seqNum());
}
void IL3GW::send_cancel_replace(const CancelReplaceOrderMsg &replace) {
const auto &create = replace.create_fields;
bts_assert_d(create.routing_record);
bts_assert_d(create.security);
bts_assert_d(create.security->has_glbx());
auto party = send_party_details(*create.routing_record);
auto msg = _buffer.frame<iLinkBinary::OrderCancelReplaceRequest515>();
msg.orderID(replace.exchange_order_id.value());
if (need_px(create.order_type)) {
msg.price().mantissa(to_price9(create.price));
} else {
msg.price().mantissa(iLinkBinary::PRICENULL9::mantissaNullValue());
}
msg.orderQty(create.quantity.value());
msg.securityID(create.security->glbx().exchange_id());
msg.side(map_side(create.side));
msg.seqNum(next_seq());
msg.putSenderID(create.routing_record->operator_id());
encode_clorid(create.key, SecurityKey(create.security->id()),
&_clorid_buffer);
msg.putClOrdID(_clorid_buffer);
msg.partyDetailsListReqID(party);
msg.orderRequestID(++_msg_count);
msg.sendingTimeEpoch(_time.now());
if (needs_stop_px(create.order_type)) {
msg.stopPx().mantissa(to_price9(create.stop_px));
} else {
msg.stopPx().mantissa(iLinkBinary::PRICENULL9::mantissaNullValue());
}
msg.putLocation(create.routing_record->location());
msg.minQty(iLinkBinary::NewOrderSingle514::minQtyNullValue());
msg.displayQty(create.display_quantity.value());
msg.expireDate(iLinkBinary::NewOrderSingle514::expireDateNullValue());
msg.ordType(map_order_type(create.order_type));
msg.timeInForce(map_time_in_force(create.time_in_force));
msg.manualOrderIndicator(
map_manual_order_indicator(create.manual_order_indicator));
msg.oFMOverride(iLinkBinary::OFMOverrideReq::Enabled);
msg.execInst().clear();
msg.executionMode(iLinkBinary::ExecMode::NULL_VALUE);
msg.liquidityFlag(iLinkBinary::BooleanNULL::NULL_VALUE);
msg.managedOrder(iLinkBinary::BooleanNULL::NULL_VALUE);
msg.shortSaleType(iLinkBinary::ShortSaleType::NULL_VALUE);
log_out(msg, msg.seqNum());
}
void IL3GW::send_cancel(const CancelOrderMsg &cancel) {
bts_assert_d(cancel.routing_record);
bts_assert_d(cancel.security);
bts_assert_d(cancel.security->has_glbx());
auto party = send_party_details(*cancel.routing_record);
auto msg = _buffer.frame<iLinkBinary::OrderCancelRequest516>();
msg.orderID(cancel.exchange_order_id.value());
msg.partyDetailsListReqID(party);
msg.manualOrderIndicator(
map_manual_order_indicator(cancel.manual_order_indicator));
msg.seqNum(next_seq());
msg.putSenderID(cancel.routing_record->operator_id());
encode_clorid(cancel.key, SecurityKey(cancel.security->id()),
&_clorid_buffer);
msg.putClOrdID(_clorid_buffer);
msg.orderRequestID(++_msg_count);
msg.sendingTimeEpoch(_time.now());
msg.putLocation(cancel.routing_record->location());
msg.securityID(cancel.security->glbx().exchange_id());
msg.side(map_side(cancel.side));
msg.liquidityFlag(iLinkBinary::BooleanNULL::NULL_VALUE);
log_out(msg, msg.seqNum());
}
void IL3GW::send_terminate(uint16_t code) {
LOG(iLink3, Info, "Terminate");
auto msg = _buffer.frame<iLinkBinary::Terminate507>();
msg.uUID(_session.uuid);
msg.requestTimestamp(_time.now());
msg.errorCodes(code);
log_out(msg, 0);
}
void IL3GW::flush(bool publish) {
_buffer.flush(_connection);
if (publish)
publish_session_update();
}
const char *state_string(State s) {
switch (s) {
case State::INIT:
return "INIT";
case State::CONNECTING:
return "CONNECTING";
case State::NEGOTIATING:
return "NEGOTIATING";
case State::ESTABLISHING:
return "ESTABLISHING";
case State::RETRANSMITTING:
return "RETRANSMITTING";
case State::READY:
return "READY";
case State::ERROR:
return "ERROR";
}
return "UNKNOWN";
}
log::CharPtrWrapper wstate_string(State s) { return wrap(state_string(s)); }
void IL3GW::set_state(State s) {
if (s != _x_state) {
LOG(iLink3, Info, "ILinkGW {}: {}->{}", _session.uuid,
wstate_string(_x_state), wstate_string(s));
_x_state = s;
if (_on_state_change)
_on_state_change(s);
}
}
//
// IConnectionHandler
void IL3GW::on_connect_success() {
if (!_session) {
// If we have a 0 cme_seq this is a new session and we need to negotiate
// it
LOG(iLink3, Info, "Negotiating new iLink session. {}", _session.uuid);
set_state(State::NEGOTIATING);
send_negotiate();
} else {
// We only need to establish the current sequence number
LOG(iLink3, Info, "re-establishing existing iLink session. {}",
_session.uuid);
_session.to_cme = std::max(_session.to_cme, 1u);
set_state(State::ESTABLISHING);
send_establish();
}
}
void IL3GW::on_disconnect() {
LOG(iLink3, Error, "Disconnected.");
if (_config.failover_on_disconnect) {
LOG(iLink3, Error, "Attempting failover");
_connection.failover();
connect();
} else {
set_state(State::ERROR);
}
}
void IL3GW::enqueue_retransmit(uint32_t from, uint32_t to, uint64_t uuid) {
LOG(iLink3, Warn, "Prepare retransmit from: {} to: {}", from, to);
bts_assert(to >= from);
do {
// +1 because to/from are using an inclusive range
auto count = std::min(to - from + 1, Retransmit::MAX_RECOVERY);
_retransmit.push(Retransmit{uuid, from, from + count - 1});
from += count;
} while (from < to);
}
void IL3GW::begin_retransmit() {
if (_retransmit.empty())
return;
auto &working = _retransmit.front();
// Don't pop the queue because we need to know when we're done
auto count = working.to - working.from + 1;
LOG(iLink3, Warn,
"Sending retransmit for missing packets: from: {} to: {} count: {}",
working.from, working.to, count);
bts_assert_d(count <= Retransmit::MAX_RECOVERY);
send_retransmit_request(working.uuid, working.from, count);
}
void IL3GW::on_retransmit_complete() {
// Replay is complete. Advance to go. Collect $200
_retransmit.pop();
LOG(iLink3, Warn, "Retransmit complete");
if (!_retransmit.empty()) {
// More replay to do...
begin_retransmit();
flush();
} else {
// Replay is finished, transition to ready state.
// Since we're already established we dont' need to send anything
set_state(State::READY);
}
}
size_t IL3GW::on_data(const std::string_view &data) {
return consume_messages(*this, data.data(), data.size(),
dispatch_message<IL3GW>);
}
void IL3GW::handle(const iLinkBinary::NegotiationResponse501 &msg) {
// If we're receiving a NegotiateResponse we just created a session
log_in(msg, 0);
// NegotiationResponse indicates the session was created. CME sequence will
// be 1
_session.from_cme = 1;
publish_session_update();
// Even though the CME sends a previous uuid+sequence we can't act on
// that data right now since we aren't established. We don't have to remember
// it, though, because exactly the same data comes on the EstablishmentAck
// msg.previousUUID(), msg.previousSeqNo();
set_state(State::ESTABLISHING);
send_establish();
}
void IL3GW::handle(const iLinkBinary::NegotiationReject502 &msg) {
LOG(iLink3, Error, "Negotiate Reject: {} '{}'.", msg.errorCodes(),
std::string(translate_negotiation_reject(msg.errorCodes())));
log_in(msg, 0);
_connection.disconnect();
set_state(State::ERROR);
}
void IL3GW::handle(const iLinkBinary::EstablishmentAck504 &msg) {
// previous uuid+sequence
auto cme_prev_uuid = msg.previousUUID();
auto cme_prev_seq = msg.previousSeqNo();
if (cme_prev_seq != 0 && cme_prev_uuid != _session.uuid) {
// There was a previous session and we need to replay messages from it
// We *intend* to only have a single session each week except
// in the case of catastrophic failure. So If we log in and
// there is a previous UUID we can just replay it entirely.
// The other possibility is a session was created automatically
// to handle a fill from a GTC order from the previous week
enqueue_retransmit(1, cme_prev_seq, cme_prev_uuid);
}
// next sequence number from the current session may also be ahead
auto cme_current_seq = msg.nextSeqNo() - 1;
if (_session.from_cme < cme_current_seq) {
// If we're behind it indicates we missed messages while disconnected
enqueue_retransmit(_session.from_cme + 1, cme_current_seq, msg.uUID());
}
_session.from_cme = cme_current_seq;
log_in(msg, 0);
if (!_retransmit.empty()) {
send_sequence();
begin_retransmit();
flush();
set_state(State::RETRANSMITTING);
} else {
send_sequence();
flush();
set_state(State::READY);
}
}
void IL3GW::handle(const iLinkBinary::EstablishmentReject505 &msg) {
LOG(iLink3, Error, "Establishment Reject: {} '{}'. NextSeq: {}.",
msg.errorCodes(),
std::string(translate_establish_reject(msg.errorCodes())),
msg.nextSeqNo());
_connection.disconnect();
log_in(msg, 0);
set_state(State::ERROR);
}
void IL3GW::handle(const iLinkBinary::Sequence506 &msg) {
log_in(msg, 0);
_session.from_cme = msg.nextSeqNo() - 1;
publish_session_update();
send_sequence();
flush();
}
void IL3GW::handle(const iLinkBinary::Terminate507 &msg) {
// https://www.cmegroup.com/confluence/display/EPICSANDBOX/Terminate
LOG(iLink3, Error, "Terminated by exchange. Code: {} Reason: {} Text: {}",
msg.errorCodes(), wrap(msg.reason()),
wrap(translate_terminate(msg.errorCodes())));
if (msg.errorCodes() == 11 || msg.errorCodes() == 10) {
// This indicates our sequence number is too low. We will jump
// forward one at a time. That should eventually fix it.
LOG(iLink3, Error,
"Will attempt to recover by advancing our sequence number");
_session.to_cme++;
publish_session_update();
}
log_in(msg, 0);
_connection.disconnect();
set_state(State::ERROR);
}
void IL3GW::handle(const iLinkBinary::Retransmission509 &msg) {
// Accepted Retransmit
log_in(msg, 0);
}
void IL3GW::handle(const iLinkBinary::RetransmitReject510 &msg) {
LOG(iLink3, Error, "Retransmit requeste rejected. {}",
msg.getReasonAsString());
log_in(msg, 0);
set_state(State::ERROR);
}
void IL3GW::handle(const iLinkBinary::NotApplied513 &msg) {
log_in(msg, 0);
send_sequence();
}
void IL3GW::handle(
const iLinkBinary::PartyDetailsDefinitionRequestAck519 &msg) {
_exe_parser.set_account(msg);
handle_common(msg);
}
void IL3GW::handle(const iLinkBinary::BusinessReject521 &msg) {
LOG(iLink3, Error, "Business Reject: {} '{}'.", msg.businessRejectReason(),
wrap(translate_business_reject_reason(msg.businessRejectReason())));
handle_common(msg);
// A business reject can de-sync the sequence number because it rejects
// the sequence number of the rejected message. Send a sequence message
// to tell the cme to advance.
send_sequence();
flush();
}
void IL3GW::handle(const iLinkBinary::ExecutionReportNew522 &msg) {
handle_exe_report(msg);
}
void IL3GW::handle(const iLinkBinary::ExecutionReportReject523 &msg) {
LOG(iLink3, Error, "Execution report reject: {}", wrap(msg.text()));
handle_exe_report(msg);
}
void IL3GW::handle(const iLinkBinary::ExecutionReportElimination524 &msg) {
handle_exe_report(msg);
}
void IL3GW::handle(const iLinkBinary::ExecutionReportTradeOutright525 &msg) {
handle_exe_report(msg);
}
void IL3GW::handle(const iLinkBinary::ExecutionReportTradeSpread526 &msg) {
handle_exe_report(msg);
}
void IL3GW::handle(const iLinkBinary::ExecutionReportTradeSpreadLeg527 &msg) {
handle_exe_report(msg);
}
void IL3GW::handle(const iLinkBinary::ExecutionReportModify531 &msg) {
handle_exe_report(msg);
}
void IL3GW::handle(const iLinkBinary::ExecutionReportStatus532 &msg) {
handle_exe_report(msg);
}
void IL3GW::handle(const iLinkBinary::ExecutionReportCancel534 &msg) {
handle_exe_report(msg);
}
void IL3GW::handle(const iLinkBinary::OrderCancelReject535 &msg) {
CancelReject cr;
bool result = _exe_parser.parse(msg, &cr);
if (result) {
_on_cancel_reject(cr);
} else {
LOG(iLink3, Error, "Failed to parse cancel reject.");
}
handle_common(msg);
}
void IL3GW::handle(const iLinkBinary::OrderCancelReplaceReject536 &msg) {
handle_common(msg);
}
void IL3GW::handle(
const iLinkBinary::ExecutionReportTradeAddendumOutright548 &msg) {
// Ignore trade correction/addendum messages for now.
// See GLBXExecutionGateway.cpp:139 for further details
handle_common(msg);
}
void IL3GW::handle(
const iLinkBinary::ExecutionReportTradeAddendumSpread549 &msg) {
// Ignore trade correction/addendum messages for now.
// See GLBXExecutionGateway.cpp:139 for further details
handle_common(msg);
}
void IL3GW::handle(
const iLinkBinary::ExecutionReportTradeAddendumSpreadLeg550 &msg) {
// Ignore trade correction/addendum messages for now.
// See GLBXExecutionGateway.cpp:139 for further details
handle_common(msg);
}
} // namespace ilink3
} // namespace orderman
} // namespace bts
+235
View File
@@ -0,0 +1,235 @@
#pragma once
#include <cstdint>
#include <functional>
#include <queue>
#include <string>
#include <vector>
#include "../IExecutionGateway.h"
#include "IL3Connection.h"
#include "IL3ConnectionHandler.h"
#include "TimeKeeper.h"
#include "IL3GeneratedFwd.h"
#include "IL3Parser.h"
#include "generated/MessageHeader.h"
#include "lib/config/protos/jetfix.pb.h"
#include "lib/orderman/actors/ilink3/RawMsg.h"
#include "lib/orderman/orderman_utils/CancelReject.h"
#define MID_SESSION_REPLAY
namespace bts {
namespace orderman {
class ExecutionReport;
namespace ilink3 {
class IL3SecDB;
struct Session {
uint64_t uuid;
// bts->cme
uint32_t to_cme;
// cme->bts
uint32_t from_cme;
operator bool const() { return to_cme || from_cme; }
};
struct Config {
std::string cme_session; // NGG
std::string firm; // 004
bool failover_on_disconnect;
std::function<void(RawMessageDirection, size_t sn, RawMsg const &)>
raw_message_logger;
};
enum class State {
INIT,
CONNECTING,
NEGOTIATING,
ESTABLISHING,
RETRANSMITTING,
READY,
ERROR
};
const char *state_string(State s);
typedef std::function<void(State)> OnStateChange;
typedef std::function<void(const ExecutionReport *)> OnExecutionReport;
typedef std::function<void(const Session &s)> OnSessionChange;
typedef std::function<void(CancelReject &)> OnCancelReject;
struct Retransmit {
static constexpr uint32_t MAX_RECOVERY = 2500;
uint64_t uuid;
uint32_t from;
uint32_t to;
};
class IL3MessageFramer;
// Does not allow re-entrance
class IL3GW : IL3ConnectionHandler {
const Config _config;
State _x_state;
IL3MessageFramer &_buffer;
OnStateChange _on_state_change;
OnExecutionReport _on_exe_report;
OnSessionChange _on_session_change;
OnCancelReject _on_cancel_reject;
IL3Connection &_connection;
Session _session;
std::string _clorid_buffer;
uint32_t _msg_count;
TimeKeeper _time;
boost::object_pool<ExecutionReport> _pool;
IL3Parser _exe_parser;
std::queue<Retransmit> _retransmit;
public:
IL3GW(IL3Connection &, IL3MessageFramer &, IL3SecDB *, const Session &,
const Config &, config::JetFIX::Connection const *);
void connect();
void send_order(const CreateOrderMsg &);
void send_cancel_replace(const CancelReplaceOrderMsg &);
void send_cancel(const CancelOrderMsg &);
void send_terminate(uint16_t code = 0);
void flush(bool publish = true);
void on_time(){
// Currently we just reply to a heartbeat with another heartbeat
// Probably better to track time and send them on demand
};
// state change
State state() { return _x_state; }
void subscribe_state(OnStateChange o) { _on_state_change = o; }
void subscribe_execution_reports(OnExecutionReport sub) {
_on_exe_report = sub;
}
void subscribe_session(OnSessionChange cb) { _on_session_change = cb; }
void subscribe_cancel_reject(OnCancelReject cb) { _on_cancel_reject = cb; }
// Only used for certification
void send_sequence(bool lapsed = false);
private:
void send_negotiate();
void send_establish();
void send_retransmit_request(uint64_t uuid, uint32_t from, size_t count);
uint64_t send_party_details(const api::orderman::RoutingRecord &);
private:
void set_state(State);
uint32_t next_seq() {
_session.to_cme += 1;
return _session.to_cme - 1;
}
void publish_session_update() { _on_session_change(_session); }
void on_connect_success() override;
void on_disconnect() override;
void enqueue_retransmit(uint32_t from, uint32_t to, uint64_t uuid);
void begin_retransmit();
void on_retransmit_complete();
template <class T> static auto serialize(const T &msg) -> auto {
return [=]() -> auto {
RawMessageData result;
std::stringstream buffer;
buffer << msg;
result.json = buffer.str();
result.binary = std::string(
msg.buffer() + msg.offset() -
iLinkBinary::MessageHeader::encodedLength(),
msg.decodeLength() + iLinkBinary::MessageHeader::encodedLength());
return result;
};
}
template <class T> void log_in(const T &msg, size_t sn) {
_config.raw_message_logger(RawMessageDirection::INCOMING, sn,
serialize(msg));
}
template <class T> void log_out(const T &msg, size_t sn) {
_config.raw_message_logger(RawMessageDirection::OUTGOING, sn,
serialize(msg));
}
//
// Message handlers
size_t on_data(const std::string_view &) override;
template <class T> void handle_common(const T &msg) {
auto uuid = msg.uUID();
auto seq = msg.seqNum();
log_in(msg, seq);
// Test if replay is copmlete
if (!_retransmit.empty() && _retransmit.front().to == seq) {
on_retransmit_complete();
return;
}
// Handle non-replay packet
if (uuid == _session.uuid) {
#ifdef MID_SESSION_REPLAY
// Gap detected during normal operation
auto expected = _session.from_cme + 1;
if (seq > expected) {
bool busy = !_retransmit.empty();
enqueue_retransmit(expected, seq - 1, _session.uuid);
if (!busy)
begin_retransmit();
flush();
}
#endif
// msg may be a replay message with a lower sequence number
if (msg.seqNum() > _session.from_cme) {
_session.from_cme = msg.seqNum();
publish_session_update();
}
}
}
template <class T> void handle_exe_report(const T &msg) {
auto *report = _pool.construct();
bool result = _exe_parser.parse(msg, report);
if (result) {
_on_exe_report(report);
} else {
LOG(iLink3, Error, "Failed to parse error report.");
}
handle_common(msg);
}
public:
void handle(const iLinkBinary::NegotiationResponse501 &);
void handle(const iLinkBinary::NegotiationReject502 &);
void handle(const iLinkBinary::EstablishmentAck504 &);
void handle(const iLinkBinary::EstablishmentReject505 &);
void handle(const iLinkBinary::Sequence506 &);
void handle(const iLinkBinary::Terminate507 &);
void handle(const iLinkBinary::Retransmission509 &);
void handle(const iLinkBinary::RetransmitReject510 &);
void handle(const iLinkBinary::NotApplied513 &);
void handle(const iLinkBinary::PartyDetailsDefinitionRequestAck519 &);
void handle(const iLinkBinary::BusinessReject521 &);
void handle(const iLinkBinary::ExecutionReportNew522 &);
void handle(const iLinkBinary::ExecutionReportReject523 &);
void handle(const iLinkBinary::ExecutionReportElimination524 &);
void handle(const iLinkBinary::ExecutionReportTradeOutright525 &);
void handle(const iLinkBinary::ExecutionReportTradeSpread526 &);
void handle(const iLinkBinary::ExecutionReportTradeSpreadLeg527 &);
void handle(const iLinkBinary::ExecutionReportModify531 &);
void handle(const iLinkBinary::ExecutionReportStatus532 &);
void handle(const iLinkBinary::ExecutionReportCancel534 &);
void handle(const iLinkBinary::OrderCancelReject535 &);
void handle(const iLinkBinary::OrderCancelReplaceReject536 &);
void handle(const iLinkBinary::ExecutionReportTradeAddendumOutright548 &);
void handle(const iLinkBinary::ExecutionReportTradeAddendumSpread549 &);
void handle(const iLinkBinary::ExecutionReportTradeAddendumSpreadLeg550 &);
};
} // namespace ilink3
} // namespace orderman
} // namespace bts
@@ -0,0 +1,56 @@
#pragma once
//
// Message templates
// BTS -> CME
#include "generated/Negotiate500.h"
#include "generated/Establish503.h"
#include "generated/RetransmitRequest508.h"
#include "generated/NewOrderSingle514.h"
#include "generated/OrderCancelReplaceRequest515.h"
#include "generated/OrderCancelRequest516.h"
// #include "generated/MassQuote517.h"
#include "generated/PartyDetailsDefinitionRequest518.h"
// #include "generated/QuoteCancel528.h"
// #include "generated/OrderMassActionRequest529.h"
// #include "generated/OrderMassStatusRequest530.h"
// #include "generated/OrderStatusRequest533.h"
// #include "generated/PartyDetailsListRequest537.h"
// #include "generated/RequestForQuote543.h"
// #include "generated/SecurityDefinitionRequest560.h"
// CME -> BTS
#include "generated/NegotiationResponse501.h"
#include "generated/NegotiationReject502.h"
#include "generated/EstablishmentAck504.h"
#include "generated/EstablishmentReject505.h"
#include "generated/Sequence506.h"
#include "generated/Terminate507.h"
#include "generated/Retransmission509.h"
#include "generated/RetransmitReject510.h"
#include "generated/NotApplied513.h"
#include "generated/PartyDetailsDefinitionRequestAck519.h"
#include "generated/BusinessReject521.h"
#include "generated/ExecutionReportNew522.h"
#include "generated/ExecutionReportReject523.h"
#include "generated/ExecutionReportElimination524.h"
#include "generated/ExecutionReportTradeOutright525.h"
#include "generated/ExecutionReportTradeSpread526.h"
#include "generated/ExecutionReportTradeSpreadLeg527.h"
#include "generated/ExecutionReportModify531.h"
#include "generated/ExecutionReportStatus532.h"
#include "generated/ExecutionReportCancel534.h"
#include "generated/OrderCancelReject535.h"
#include "generated/OrderCancelReplaceReject536.h"
// #include "generated/PartyDetailsListReport538.h"
#include "generated/ExecutionAck539.h"
// #include "generated/NewOrderCross544.h"
// #include "generated/MassQuoteAck545.h"
// #include "generated/RequestForQuoteAck546.h"
// #include "generated/QuoteCancelAck547.h"
#include "generated/ExecutionReportTradeAddendumOutright548.h"
#include "generated/ExecutionReportTradeAddendumSpread549.h"
#include "generated/ExecutionReportTradeAddendumSpreadLeg550.h"
// #include "generated/OrderMassActionReport558.h"
// #include "generated/SecurityDefinitionResponse561.h"
@@ -0,0 +1,37 @@
#pragma once
namespace iLinkBinary {
class Negotiate500;
class Establish503;
class RetransmitRequest508;
class NewOrderSingle514;
class OrderCancelReplaceRequest515;
class OrderCancelRequest516;
class PartyDetailsDefinitionRequest518;
class NegotiationResponse501;
class NegotiationReject502;
class EstablishmentAck504;
class EstablishmentReject505;
class Sequence506;
class Terminate507;
class Retransmission509;
class RetransmitReject510;
class NotApplied513;
class PartyDetailsDefinitionRequestAck519;
class BusinessReject521;
class ExecutionReportNew522;
class ExecutionReportReject523;
class ExecutionReportElimination524;
class ExecutionReportTradeOutright525;
class ExecutionReportTradeSpread526;
class ExecutionReportTradeSpreadLeg527;
class ExecutionReportModify531;
class ExecutionReportStatus532;
class ExecutionReportCancel534;
class OrderCancelReject535;
class OrderCancelReplaceReject536;
class ExecutionAck539;
class ExecutionReportTradeAddendumOutright548;
class ExecutionReportTradeAddendumSpread549;
class ExecutionReportTradeAddendumSpreadLeg550;
}
@@ -0,0 +1,58 @@
#pragma once
#include <memory>
#include <cstring>
#include "IL3Connection.h"
#include "IL3ReceiveBuffer.h"
#include "generated/MessageHeader.h"
namespace bts {
namespace orderman {
namespace ilink3 {
struct FramingHeader {
uint16_t message_size;
uint16_t encoding_type;
};
class IL3MessageFramer {
std::vector<std::string_view> _buffer_ptrs;
IL3ReceiveBuffer _memory;
public:
IL3MessageFramer():
_buffer_ptrs(), _memory()
{
_buffer_ptrs.reserve(512);
}
~IL3MessageFramer() = default;
//
// Framing
template <class Msg>
Msg frame(size_t extra=0) {
auto size =
sizeof(FramingHeader) +
iLinkBinary::MessageHeader::encodedLength() +
Msg::sbeBlockLength() +
extra;
auto buffer = _memory.allocate(size);
_memory.advance(size);
FramingHeader* fhead = (FramingHeader*)(buffer.data());
fhead->message_size = size;
fhead->encoding_type = 0xCAFE;
Msg msg;
msg.wrapAndApplyHeader(const_cast<char*>(buffer.data()), sizeof(FramingHeader), size+sizeof(FramingHeader));
_buffer_ptrs.push_back(buffer);
return msg;
}
void flush(IL3Connection& connection) {
if (!_buffer_ptrs.empty()) {
connection.send(_buffer_ptrs);
_buffer_ptrs.clear();
}
}
};
}}} //namespace
@@ -0,0 +1,361 @@
#include "IL3Parser.h"
#include "IL3Dispatcher.h"
#include "IL3GeneratedFull.h"
#include "ILink3Utils.h"
#include "lib/domain_model/Security.h"
#include "lib/md/IntPrice.h"
#include "lib/orderman/actors/ilink3/IL3SecDB.h"
#include "lib/orderman/orderman_utils/CancelReject.h"
#include "lib/orderman/orderman_utils/fix_order_codec.h"
#include "lib/orderman/orderman_utils/orderman_common.h"
namespace bts {
namespace orderman {
namespace ilink3 {
IL3Parser::IL3Parser(config::JetFIX::Connection const *cfg, IL3SecDB *db)
: most_recent_account(), cfg(cfg), db(db) {}
const api::Security *IL3Parser::get_security(SecurityKey id) {
if (db)
return db->get_security(id);
else
return nullptr;
}
const api::Security *IL3Parser::get_security(ExchangeSecurityKey id) {
if (db)
return db->get_security(id);
else
return nullptr;
}
template <class T>
bool IL3Parser::parse_common(const T &msg, ExecutionReport *data) {
data->account_name = most_recent_account;
static_assert(sizeof(OperatorID) == T::senderIDLength());
memcpy(&data->operator_id.value(), msg.senderID(), msg.senderIDLength());
data->exchange = api::VENUE_GLBX;
data->exchange_order_id = ExchangeOrderID64{msg.orderID()};
data->leg_security = nullptr;
data->manual_order_indicator =
map_manual_order_indicator(msg.manualOrderIndicator());
data->msg_seq_num = msg.seqNum();
SecurityKey sk;
decode_clorid(msg.clOrdID(), &data->order_key, &sk);
auto sec = get_security(sk);
if (!sec) {
LOG(iLink3, Error,
"Cannot resolve security key {} in order {}. Clorid: {} Exchange "
"order id: {}",
sk.value(), data->order_key.value(), std::string(msg.clOrdID()),
msg.orderID());
return false;
}
data->security = sec;
data->order_type = map_order_type(msg.ordType());
auto &price = const_cast<T &>(msg).price();
data->price = securities::decode_exchange_price(
*sec, Price{price.mantissa() * md::pow10(price.exponent())});
data->quantity = Quantity{msg.orderQty()};
data->side = map_side(msg.side());
auto &stop_price = const_cast<T &>(msg).stopPx();
data->stop_price = securities::decode_exchange_price(
*sec, Price{stop_price.mantissa() * md::pow10(stop_price.exponent())});
data->time_in_force = map_time_in_force(msg.timeInForce());
data->transact_time = ExchangeTimeStamp64{msg.transactTime()};
data->ice_cl_ord_id = 0;
data->ice_company_id = 0;
data->cfg = cfg;
return true;
}
template <class T>
void IL3Parser::parse_trade(const T &msg, ExecutionReport *data) {
orderman::to_array(msg.getExecIDAsStringView(),
&data->exchange_fill_id.value());
switch (msg.ordStatus()) {
case iLinkBinary::OrdStatusTrd::Filled:
data->order_status =
api::orderman::NativeOrderStatus::NATIVE_ORDER_STATUS_FILLED;
data->exec_type = api::orderman::EXEC_TYPE_FILL;
break;
case iLinkBinary::OrdStatusTrd::PartiallyFilled:
data->order_status = api::orderman::NativeOrderStatus::
NATIVE_ORDER_STATUS_PARTIALLY_FILLED;
data->exec_type = api::orderman::EXEC_TYPE_PARTIAL_FILL;
break;
default:
data->order_status =
api::orderman::NativeOrderStatus::NATIVE_ORDER_STATUS_UNDEFINED;
}
data->last_shares = Quantity{msg.lastQty()};
auto &price = const_cast<T &>(msg).lastPx();
if (data->leg_security) {
data->last_price = securities::decode_exchange_price(
*data->leg_security,
Price{price.mantissa() * md::pow10(price.exponent())});
} else {
data->last_price = securities::decode_exchange_price(
*data->security,
Price{price.mantissa() * md::pow10(price.exponent())});
}
data->filled_quantity = Quantity{msg.cumQty()};
}
bool IL3Parser::has_account() const {
return strlen(most_recent_account.value().data());
}
bool IL3Parser::parse(const iLinkBinary::ExecutionReportNew522 &msg,
ExecutionReport *data) {
if (parse_common(msg, data)) {
data->avg_price = Price{0.0};
data->cust_order_handling_inst = "";
data->exchange_fill_id = ExchangeFillID{};
data->exec_type = api::orderman::EXEC_TYPE_NEW;
data->filled_quantity = Quantity{0};
data->last_price = Price{0.0};
data->last_shares = Quantity{0};
data->max_show = Quantity{0};
data->memo = "";
data->msg = "";
data->order_status = api::orderman::NativeOrderStatus(*msg.ordStatus());
return true;
} else
return false;
}
template <class ER>
bool try_parse_exe_report_helper(IL3Parser &parser, std::string_view const &msg,
ExecutionReport *out) {
iLinkBinary::MessageHeader mheader{
(char *)msg.data(), 0, msg.size(),
iLinkBinary::MessageHeader::sbeSchemaVersion()};
if (mheader.templateId() == ER::sbeTemplateId()) {
return parser.parse(
ER{(char *)msg.data(), iLinkBinary::MessageHeader::encodedLength(),
msg.size(), mheader.blockLength(), mheader.version()},
out);
}
return false;
}
std::optional<ExecutionReport>
IL3Parser::try_parse_exe_report(std::string_view const &msg) {
using namespace iLinkBinary;
if (has_account()) {
ExecutionReport report;
if (try_parse_exe_report_helper<ExecutionReportNew522>(*this, msg,
&report))
return {report};
if (try_parse_exe_report_helper<ExecutionReportReject523>(*this, msg,
&report))
return {report};
if (try_parse_exe_report_helper<ExecutionReportElimination524>(*this, msg,
&report))
return {report};
if (try_parse_exe_report_helper<ExecutionReportTradeOutright525>(
*this, msg, &report))
return {report};
if (try_parse_exe_report_helper<ExecutionReportTradeSpread526>(*this, msg,
&report))
return {report};
if (try_parse_exe_report_helper<ExecutionReportTradeSpreadLeg527>(
*this, msg, &report))
return {report};
if (try_parse_exe_report_helper<ExecutionReportModify531>(*this, msg,
&report))
return {report};
if (try_parse_exe_report_helper<ExecutionReportStatus532>(*this, msg,
&report))
return {report};
if (try_parse_exe_report_helper<ExecutionReportCancel534>(*this, msg,
&report))
return {report};
}
iLinkBinary::MessageHeader mheader{
(char *)msg.data(), 0, msg.size(),
iLinkBinary::MessageHeader::sbeSchemaVersion()};
if (mheader.templateId() ==
PartyDetailsDefinitionRequestAck519::sbeTemplateId())
set_account(PartyDetailsDefinitionRequestAck519{
(char *)msg.data(), iLinkBinary::MessageHeader::encodedLength(),
msg.size(), mheader.blockLength(), mheader.version()});
return {};
}
bool IL3Parser::parse(const iLinkBinary::ExecutionReportReject523 &msg,
ExecutionReport *data) {
if (parse_common(msg, data)) {
data->order_status = api::orderman::NativeOrderStatus(*msg.ordStatus());
data->msg = msg.getTextAsStringView();
return true;
} else
return false;
}
bool IL3Parser::parse(const iLinkBinary::ExecutionReportElimination524 &msg,
ExecutionReport *data) {
if (parse_common(msg, data)) {
data->order_status = api::orderman::NativeOrderStatus(*msg.ordStatus());
return true;
} else
return false;
}
bool IL3Parser::parse(const iLinkBinary::ExecutionReportTradeOutright525 &msg,
ExecutionReport *data) {
if (parse_common(msg, data)) {
parse_trade(msg, data);
return true;
} else
return false;
}
bool IL3Parser::parse(const iLinkBinary::ExecutionReportTradeSpread526 &msg,
ExecutionReport *data) {
if (parse_common(msg, data)) {
parse_trade(msg, data);
return true;
} else
return false;
}
bool IL3Parser::parse(const iLinkBinary::ExecutionReportTradeSpreadLeg527 &msg,
ExecutionReport *data) {
data->account_name = most_recent_account;
data->avg_price = Price{0.0};
static_assert(sizeof(OperatorID) ==
iLinkBinary::ExecutionReportTradeSpread526::senderIDLength());
memcpy(&data->operator_id.value(), msg.senderID(), msg.senderIDLength());
data->exchange = api::VENUE_GLBX;
data->exchange_order_id = ExchangeOrderID64{msg.orderID()};
data->msg_seq_num = msg.seqNum();
SecurityKey sk;
decode_clorid(msg.clOrdID(), &data->order_key, &sk);
auto sec = get_security(sk);
if (!sec) {
LOG(iLink3, Error,
"Cannot resolve security key {} in order {}. Clorid: {} Exchange "
"order id: {}",
sk.value(), data->order_key.value(), std::string(msg.clOrdID()),
msg.orderID());
return false;
}
data->security = sec;
data->leg_security =
get_security(ExchangeSecurityKey{(uint64_t)msg.securityID()});
if (data->leg_security == nullptr) {
LOG(iLink3, Error,
"Received execution report for an unknown spread leg"
"security {}. Will not be "
" able to process this report. This is likely a "
"bug that could lead to"
" lost leg fills. Please investigate. Spread sec_key: {} leg "
"exchange "
"security id: {}",
sk.value(), msg.securityID());
return false;
}
data->order_type = map_order_type(msg.ordType());
auto &price =
const_cast<iLinkBinary::ExecutionReportTradeSpreadLeg527 &>(msg)
.lastPx();
data->price = securities::decode_exchange_price(
*data->leg_security,
Price{price.mantissa() * md::pow10(price.exponent())});
data->quantity = Quantity{0};
data->side = map_side(msg.side());
data->stop_price = Price{0.0};
data->time_in_force = api::orderman::TimeInForce::TIME_IN_FORCE_UNDEFINED;
data->transact_time = ExchangeTimeStamp64{msg.transactTime()};
data->ice_cl_ord_id = 0;
data->ice_company_id = 0;
data->cfg = cfg;
parse_trade(msg, data);
return true;
}
bool IL3Parser::parse(const iLinkBinary::ExecutionReportModify531 &msg,
ExecutionReport *data) {
if (parse_common(msg, data)) {
data->order_status = api::orderman::NativeOrderStatus(*msg.ordStatus());
return true;
} else
return false;
}
bool IL3Parser::parse(const iLinkBinary::ExecutionReportStatus532 &msg,
ExecutionReport *data) {
return parse_common(msg, data);
}
bool IL3Parser::parse(const iLinkBinary::ExecutionReportCancel534 &msg,
ExecutionReport *data) {
if (parse_common(msg, data)) {
data->order_status = api::orderman::NativeOrderStatus(*msg.ordStatus());
return true;
} else
return false;
}
bool IL3Parser::parse(const iLinkBinary::OrderCancelReject535 &msg,
CancelReject *cr) {
SecurityKey sk;
OrderKey ok;
decode_clorid(msg.clOrdID(), &ok, &sk);
cr->order_key = ok;
cr->reject_msg = msg.getTextAsStringView();
// https://www.cmegroup.com/confluence/display/EPICSANDBOX/iLink+Reject+Codes
if (msg.cxlRejReason() == 2045)
cr->reason = CancelRejectReason::UnknownOrder;
else
cr->reason = CancelRejectReason::Other;
return true;
}
void IL3Parser::set_account(
const iLinkBinary::PartyDetailsDefinitionRequestAck519 &msg) {
// When using on-demand format the PartyDetailsDefinitionRequestAck519
// is sent back immediately preceding the execution report on the wire.
auto details =
const_cast<iLinkBinary::PartyDetailsDefinitionRequestAck519 &>(msg)
.noPartyDetails();
while (details.hasNext()) {
details.next();
if (details.partyDetailRole() ==
iLinkBinary::PartyDetailRole::CustomerAccount) {
const char *p_account = details.partyDetailID();
strncpy(most_recent_account.value().data(), p_account,
most_recent_account.value().size());
break;
}
}
}
struct ParseHandler {
IL3Parser &parser;
ExecutionReport *result;
bool success = false;
template <class T> void handle(const T &t) {
success = parser.parse(t, result);
}
void handle(iLinkBinary::PartyDetailsDefinitionRequestAck519 &t) {
parser.set_account(t);
}
};
bool parse_report(const std::string &data, IL3Parser &parser,
ExecutionReport *result) {
ParseHandler handler{parser, result};
consume_messages(handler, data.c_str(), data.size(),
dispatch_execution_reports<ParseHandler>);
return handler.success;
}
} // namespace ilink3
} // namespace orderman
} // namespace bts
@@ -0,0 +1,46 @@
#pragma once
#include "lib/orderman/orderman_utils/ExecutionReport.h"
#include "IL3GeneratedFwd.h"
#include "lib/md/SecurityOracle.h"
#include "lib/md/ExchangeSecurityKey.h"
#include <boost/pool/object_pool.hpp>
#include <google/btree/btree_map.h>
namespace bts {
namespace orderman {
class CancelReject;
namespace ilink3 {
class IL3SecDB;
class IL3Parser {
AccountName most_recent_account;
config::JetFIX::Connection const *cfg;
IL3SecDB *db;
const api::Security* get_security(SecurityKey);
const api::Security* get_security(ExchangeSecurityKey);
public:
IL3Parser(config::JetFIX::Connection const *, IL3SecDB *sec_db);
bool parse(const iLinkBinary::ExecutionReportNew522&, ExecutionReport*);
bool parse(const iLinkBinary::ExecutionReportReject523&, ExecutionReport*);
bool parse(const iLinkBinary::ExecutionReportElimination524&, ExecutionReport*);
bool parse(const iLinkBinary::ExecutionReportTradeOutright525&, ExecutionReport*);
bool parse(const iLinkBinary::ExecutionReportTradeSpread526&, ExecutionReport*);
bool parse(const iLinkBinary::ExecutionReportTradeSpreadLeg527&, ExecutionReport*);
bool parse(const iLinkBinary::ExecutionReportModify531&, ExecutionReport*);
bool parse(const iLinkBinary::ExecutionReportStatus532&, ExecutionReport*);
bool parse(const iLinkBinary::ExecutionReportCancel534&, ExecutionReport*);
bool parse(const iLinkBinary::OrderCancelReject535&, CancelReject*);
void set_account(const iLinkBinary::PartyDetailsDefinitionRequestAck519&);
bool has_account() const;
std::optional<ExecutionReport> try_parse_exe_report(std::string_view const&bytes);
private:
template <class T> bool parse_common(const T&, ExecutionReport*);
template <class T> void parse_trade(const T&, ExecutionReport*);
};
bool parse_report(const std::string&, IL3Parser&, ExecutionReport*);
}}}//namespace
@@ -0,0 +1,221 @@
#include "IL3Reactor.h"
#include "lib/utils/bts_assert.h"
#include "IL3ReceiveBuffer.h"
#include <arpa/inet.h>
#include <fcntl.h>
#include <fstream>
#include <netinet/tcp.h>
#include <stdlib.h>
#include <sys/epoll.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <unistd.h>
#include <array>
#include <queue>
namespace bts {
namespace orderman {
namespace ilink3 {
struct IL3Reactor::Connection : public IL3Connection {
int epoll_fd;
int fd;
bool is_connected;
IL3ConnectionHandler* handler;
std::queue<TCPEndpoint> end_points;
IL3ReceiveBuffer rc_buffer;
Connection(int epoll_fd, const std::vector<TCPEndpoint>& end_points):
epoll_fd(epoll_fd), fd(0), is_connected(false), handler(nullptr),
end_points(std::deque<TCPEndpoint> { end_points.begin(), end_points.end() }),
rc_buffer()
{ }
~Connection() {
if (is_connected) disconnect();
}
void connect(IL3ConnectionHandler& h) override {
bts_assert(!is_connected);
handler = &h;
fd = socket(AF_INET, SOCK_STREAM, 0);
int flag = 1;
auto result = ::setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, (char*)&flag, sizeof(int));
bts_assert(result == 0);
bts_assert(fd >= 0);
struct sockaddr_in serv_addr;
memset(&serv_addr, '0', sizeof(serv_addr));
auto end_point = current_end_point();
auto& host = end_point.host;
auto port = end_point.port;
serv_addr.sin_family = AF_INET;
serv_addr.sin_port = htons(port);
result = inet_pton(AF_INET, host.c_str(), &serv_addr.sin_addr);
bts_assert(result > 0);
result = ::connect(fd, (struct sockaddr *)&serv_addr, sizeof(serv_addr));
if (result < 0) {
auto er = errno;
char msg[256];
LOG(iLink3, Error, "Failed to connect: {} {}:{}",
std::string(strerror_r(er, msg, sizeof(msg))),host,port);
disconnect();
}
else {
int flag = 1;
result = ::setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, (char *)&flag,
sizeof(int));
bts_assert(result == 0);
epoll_event ev;
ev.events = EPOLLIN | EPOLLHUP | EPOLLERR | EPOLLRDHUP;
ev.data.ptr = this;
auto result = epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev);
bts_assert_msg(result == 0, "epoll");
is_connected = true;
handler->on_connect_success();
}
}
void disconnect() override {
if (fd) close(fd);
fd = 0;
is_connected = false;
handler->on_disconnect();
}
void send(const std::string_view data) override {
if (!is_connected) {
LOG(iLink3, Error, "Attempted to send on closed connection. {}:{}", current_end_point().host, current_end_point().port);
return;
}
auto result = write(fd, data.data(), data.size());
if (result < 0) {
auto er = errno;
char msg[256];
auto host = current_end_point().host;
auto port = current_end_point().port;
LOG(iLink3, Warn, "Could not write to reactor connection. {}:{}",
std::string(strerror_r(er, msg, sizeof(msg))),host,port);
disconnect();
}
}
void send(const std::vector<std::string_view>& dt) override {
if (!is_connected) {
LOG(iLink3, Error, "Attempted to send on closed connection. {}:{}", current_end_point().host, current_end_point().port);
return;
}
iovec *iv = (iovec*)alloca(sizeof(iovec)*dt.size());
for (size_t i = 0; i < dt.size();i++) {
iv[i].iov_base = (void*)dt[i].data();
iv[i].iov_len = dt[i].size();
}
auto result = ::writev(fd,iv,dt.size());
if (result < 0) {
auto er = errno;
char msg[256];
auto host = current_end_point().host;
auto port = current_end_point().port;
LOG(iLink3, Warn, "Could not write to reactor connection. {}:{}",
std::string(strerror_r(er, msg, sizeof(msg))),host,port);
disconnect();
}
}
void on_data(const std::string_view data) {
bts_assert_d(is_connected);
// Copy input data into a permanent buffer
rc_buffer.append(data);
bts_assert_d(rc_buffer.unread() > 0);
auto read = handler->on_data(rc_buffer.data());
rc_buffer.advance(read);
}
void failover() override {
bts_assert(!is_connected);
next_end_point();
}
TCPEndpoint current_end_point() {
return end_points.front();
}
void next_end_point() {
auto next = end_points.front();
end_points.pop();
end_points.push(next);
}
};
struct IL3Reactor::PImpl {
int epoll_fd = -1;
std::vector<std::unique_ptr<Connection>> connections;
};
IL3Reactor::IL3Reactor() : me(new PImpl()) {
me->epoll_fd = epoll_create(100);
bts_assert(me->epoll_fd >= 0);
}
IL3Reactor::~IL3Reactor() {
me->connections.clear();
close(me->epoll_fd);
me->epoll_fd = -1;
}
IL3Connection& IL3Reactor::register_connection(const std::vector<TCPEndpoint>& end_points) {
me->connections.emplace_back(
std::make_unique<Connection>(me->epoll_fd, end_points));
return *me->connections.back();
}
[[gnu::noinline]]
void non_inline_assert() {
bts_assert(false);
}
constexpr bool has_error(int events) {
return events & (EPOLLHUP | EPOLLERR | EPOLLRDHUP);
}
void IL3Reactor::do_work() {
std::array<epoll_event, 100> ev_list;
auto ready = epoll_wait(me->epoll_fd, ev_list.data(), ev_list.size(), 0);
if (ready == -1) {
if (errno == EINTR) return;
else non_inline_assert();
}
for (int i = 0; i < ready; ++i) {
auto& ev = ev_list[i];
auto handler = (Connection*)(ev.data.ptr);
if (ev.events & EPOLLIN) {
char buffer[4096];
auto sz = read(handler->fd, buffer, sizeof(buffer));
if (sz > 0) {
bts_assert_d(handler->is_connected);
handler->on_data(std::string_view(buffer, sz));
}
else if (has_error(ev.events)) {
handler->disconnect();
}
}
else if (has_error(ev.events)) {
// if EPOLLIN was set along with
// EPOLLHUP/EPOLLERR/EPOLLRDHUP, we pretend it is a normal
// read so that we drain the bytes in the socket. The error
// or hup condition would remain and we would catch in the
// next iteration.
handler->disconnect();
}
}
}
}}} //namespace
@@ -0,0 +1,33 @@
#pragma once
#include "IL3Connection.h"
#include "IL3ConnectionHandler.h"
#include <vector>
#include <memory>
namespace bts {
namespace orderman {
namespace ilink3 {
struct TCPEndpoint {
std::string host;
uint16_t port;
};
class IL3Reactor {
struct PImpl;
struct Connection;
std::unique_ptr<PImpl> me;
public:
IL3Reactor();
~IL3Reactor();
IL3Connection& register_connection(const std::vector<TCPEndpoint>&);
void do_work();
};
}}} //namespace
@@ -0,0 +1,77 @@
#pragma once
#include "lib/utils/bts_assert.h"
#include <string>
#include <vector>
#include <list>
namespace bts {
namespace orderman {
namespace ilink3 {
class IL3ReceiveBuffer {
std::list<std::vector<char>> buffers;
size_t read_pos, write_pos;
size_t buffer_size;
public:
IL3ReceiveBuffer(size_t size = 1024*1024) : buffers(), read_pos(0), write_pos(0), buffer_size(size) {
add_buffer(size);
}
void append(const std::string_view& data) {
auto dest = allocate(data.size());
memcpy(const_cast<char*>(dest.data()), data.data(), data.size());
}
std::string_view allocate(size_t size) {
// If we have enough space append the data
if (size <= available_write_space()) {
char* write_ptr = buffers.back().data() + write_pos;
write_pos += size;
return std::string_view(write_ptr, size);
}
else {
// If we don't have enough space...
// Compute the minimum size of the new buffer
auto min_size = unread() + size;
// allocate a new buffer
const auto& prev_buffer = buffers.back();
auto prev_read = read_pos;
auto prev_write = write_pos;
add_buffer(std::max(buffer_size, min_size));
// copy unread portion of old buffer + the new data into the new buffer
auto unread_sz = prev_write - prev_read;
memcpy(buffers.back().data(), prev_buffer.data()+prev_read, unread_sz);
write_pos = unread_sz;
bts_assert_d(size <= available_write_space());
return allocate(size);
}
}
void advance(size_t consumed) {
read_pos += consumed;
bts_assert_d(read_pos <= write_pos);
}
size_t unread() const {
return write_pos - read_pos;
}
const std::string_view data() const {
return std::string_view(buffers.back().data()+read_pos, unread());
}
private:
size_t available_write_space() const {
return buffers.back().size() - write_pos;
}
void add_buffer(size_t size) {
buffers.emplace_back(std::vector<char>());
buffers.back().resize(size);
read_pos = write_pos = 0;
}
};
}}} //namespace
+40
View File
@@ -0,0 +1,40 @@
#pragma once
#include "lib/domain_model/Security.pb.h"
#include "lib/domain_model/SecurityKey.h"
#include "lib/md/ExchangeSecurityKey.h"
#include "lib/md/SecurityOracle.h"
namespace bts {
namespace orderman {
namespace ilink3 {
class IL3SecDB {
public:
IL3SecDB(md::SecurityOracle& oracle) : oracle(oracle) {}
void add_security(const api::Security &sec) {
auto id = SecurityKey{sec.id()};
security_by_key[id] = &sec;
auto ex_id = ExchangeSecurityKey{sec.glbx().exchange_id()};
security_by_exchange_key[ex_id] = &sec;
}
const api::Security *get_security(SecurityKey sk) {
auto &sec = security_by_key[sk];
if (!sec)
sec = oracle.get_security(sk);
return sec;
}
const api::Security *get_security(ExchangeSecurityKey k) {
auto &sec = security_by_exchange_key[k];
if (!sec)
sec = oracle.get_security(k);
return sec;
}
private:
md::SecurityOracle oracle;
std::unordered_map<SecurityKey, const api::Security*> security_by_key;
std::unordered_map<ExchangeSecurityKey, const api::Security*> security_by_exchange_key;
};
}}}
@@ -0,0 +1,528 @@
#include "ILink3Utils.h"
#include "lib/orderman/orderman_utils/fix_order_codec.h"
#include "lib/utils/base62.h"
namespace bts {
namespace orderman {
namespace ilink3 {
iLinkBinary::SideReq::Value map_side(api::Side side) {
switch (side) {
case api::Side::SIDE_BUY:
return iLinkBinary::SideReq::Buy;
case api::Side::SIDE_SELL:
return iLinkBinary::SideReq::Sell;
default:
return iLinkBinary::SideReq::NULL_VALUE;
}
}
api::Side map_side(iLinkBinary::SideReq::Value side) {
switch (side) {
case iLinkBinary::SideReq::Buy:
return api::Side::SIDE_BUY;
case iLinkBinary::SideReq::Sell:
return api::Side::SIDE_SELL;
default:
return api::Side::SIDE_UNDEFINED;
}
}
iLinkBinary::OrderTypeReq::Value
map_order_type(api::orderman::NativeOrderType ot) {
using namespace api::orderman;
using namespace iLinkBinary;
switch (ot) {
case NativeOrderType::NATIVE_ORDER_TYPE_MARKET:
return OrderTypeReq::MarketwithProtection;
case NativeOrderType::NATIVE_ORDER_TYPE_LIMIT:
return OrderTypeReq::Limit;
case NativeOrderType::NATIVE_ORDER_TYPE_STOP:
return OrderTypeReq::StopwithProtection;
case NativeOrderType::NATIVE_ORDER_TYPE_STOP_LIMIT:
return OrderTypeReq::StopLimit;
case NativeOrderType::NATIVE_ORDER_TYPE_MARKET_LIMIT:
return OrderTypeReq::MarketWithLeftoverAsLimit;
default:
return iLinkBinary::OrderTypeReq::NULL_VALUE;
}
}
api::orderman::NativeOrderType
map_order_type(iLinkBinary::OrderType::Value ot) {
using namespace api::orderman;
using namespace iLinkBinary;
switch (ot) {
case OrderType::MarketWithProtection:
return NativeOrderType::NATIVE_ORDER_TYPE_MARKET;
case OrderType::Limit:
return NativeOrderType::NATIVE_ORDER_TYPE_LIMIT;
case OrderType::StopLimit:
return NativeOrderType::NATIVE_ORDER_TYPE_STOP_LIMIT;
case OrderType::MarketWithLeftoverAsLimit:
return NativeOrderType::NATIVE_ORDER_TYPE_MARKET_LIMIT;
case OrderType::NULL_VALUE:
return NativeOrderType::NATIVE_ORDER_TYPE_UNDEFINED;
}
return NativeOrderType::NATIVE_ORDER_TYPE_UNDEFINED;
}
iLinkBinary::TimeInForce::Value
map_time_in_force(api::orderman::TimeInForce tif) {
using namespace api::orderman;
switch (tif) {
case TimeInForce::TIME_IN_FORCE_DATE:
return iLinkBinary::TimeInForce::GoodTillDate;
case TimeInForce::TIME_IN_FORCE_DAY:
return iLinkBinary::TimeInForce::Day;
case TimeInForce::TIME_IN_FORCE_GTC:
return iLinkBinary::TimeInForce::GoodTillCancel;
case TimeInForce::TIME_IN_FORCE_IOC:
return iLinkBinary::TimeInForce::FillAndKill;
default:
return iLinkBinary::TimeInForce::NULL_VALUE;
}
}
api::orderman::TimeInForce
map_time_in_force(iLinkBinary::TimeInForce::Value tif) {
using namespace api::orderman;
switch (tif) {
case iLinkBinary::TimeInForce::Day:
return TimeInForce::TIME_IN_FORCE_DAY;
case iLinkBinary::TimeInForce::GoodTillCancel:
return TimeInForce::TIME_IN_FORCE_GTC;
case iLinkBinary::TimeInForce::FillOrKill:
return TimeInForce::TIME_IN_FORCE_IOC;
default:
return TimeInForce::TIME_IN_FORCE_UNDEFINED;
}
}
iLinkBinary::ManualOrdIndReq::Value
map_manual_order_indicator(api::orderman::ManualOrderIndicator moi) {
using namespace api::orderman;
using namespace iLinkBinary;
switch (moi) {
case ManualOrderIndicator::MANUAL_ORDER_INDICATOR_NO:
return ManualOrdIndReq::Automated;
case ManualOrderIndicator::MANUAL_ORDER_INDICATOR_YES:
return ManualOrdIndReq::Manual;
default:
return ManualOrdIndReq::NULL_VALUE;
}
}
api::orderman::ManualOrderIndicator
map_manual_order_indicator(iLinkBinary::ManualOrdIndReq::Value moi) {
using namespace api::orderman;
using namespace iLinkBinary;
switch (moi) {
case ManualOrdIndReq::Automated:
return ManualOrderIndicator::MANUAL_ORDER_INDICATOR_NO;
case ManualOrdIndReq::Manual:
return ManualOrderIndicator::MANUAL_ORDER_INDICATOR_YES;
default:
return ManualOrderIndicator::MANUAL_ORDER_INDICATOR_NO;
}
}
iLinkBinary::ClearingAcctType::Value
map_customer_or_firm(api::orderman::CustomerOrFirm c) {
using namespace api::orderman;
using namespace iLinkBinary;
switch (c) {
case CustomerOrFirm::CUSTOMER_OR_FIRM_CUSTOMER:
return ClearingAcctType::Customer;
case CustomerOrFirm::CUSTOMER_OR_FIRM_FIRM:
return ClearingAcctType::Firm;
default:
return ClearingAcctType::NULL_VALUE;
}
}
iLinkBinary::SMPI::Value
map_smp_instruction(api::orderman::SelfMatchPreventionInstruction smpi) {
using namespace api::orderman;
using namespace iLinkBinary;
switch (smpi) {
case SELF_MATCH_PREVENTION_INSTRUCTION_CANCEL_RESTING:
return SMPI::CancelOldest;
case SELF_MATCH_PREVENTION_INSTRUCTION_CANCEL_AGRESSING:
return SMPI::CancelNewest;
default:
return SMPI::NULL_VALUE;
}
}
iLinkBinary::CustOrderCapacity::Value
map_cust_cti(api::orderman::CTICode code) {
using namespace api::orderman;
using namespace iLinkBinary;
switch (code) {
case CTI_CODE_CTI_CODE1:
return CustOrderCapacity::Membertradingfortheirownaccount;
case CTI_CODE_CTI_CODE2:
return CustOrderCapacity::Clearingfirmtradingforitsproprietaryaccount;
case CTI_CODE_CTI_CODE3:
return CustOrderCapacity::Membertradingforanothermember;
case CTI_CODE_CTI_CODE4:
return CustOrderCapacity::Allother;
default:
case CTI_CODE_UNDEFINED:
return CustOrderCapacity::NULL_VALUE;
}
}
const char *translate_business_reject_reason(int reason) {
switch (reason) {
case 0:
return "Other";
case 1:
return "Unknown ID (unknown PartyDetailsListReqID being referenced in "
"business message)";
case 2:
return "Unknown Security";
case 3:
return "Unsupported Message Type (for example using messages specific to "
"cash markets for futures)";
case 5:
return "Required Field Missing";
case 6:
return "Not Authorized";
case 8:
return "Throttle Limit Exceeded (volume controls)";
case 100:
return "Value is incorrect (out of range) for this tag (for example "
"using a value outside defined range of integer) ";
case 101:
return "Incorrect data format for value (for example using ASCII "
"character outside 32-126 in string field)";
case 102:
return "Rejected due to Kill Switch";
case 103:
return "Rejected due to Risk Management API";
case 104:
return "Rejected due to Globex Credit Controls";
case 105:
return "Not Authorized to trade Deliverable Swap Futures";
case 106:
return "Not Authorized to trade Interest Rate Swap Futures";
case 107:
return "Rejected due to Inline Credit Controls";
case 108:
return "Invalid PartyDetailsListReqID (reusing already existing "
"PartyDetailsListReqID while creating Party Details Definition "
"Request)";
case 109:
return "Incoming message could not be decoded. (Check schema version)";
case 110:
return "Same repeating group entry appears more than once";
case 111:
return "Exceeded maximum number of allowable Party Details Definition "
"Requests";
case 112:
return "Technical Error in Registering Party Details";
case 113:
return "Rejected due to Cross Venue Risk";
case 114:
return "Order Status Not Available";
case 115:
return "Enum Not Supported";
case 116:
return "Order Status Not Found";
case 117:
return "Mass Order Status Cannot be Completed";
case 118:
return "Exceeded Maximum Number of Allowable RequestingPartyID's in "
"Party Details List Request";
case 119:
return "Duplicate Order Threshold Exceeded";
case 120:
return "On-Demand Message Rejected Due to Preceding Party Details "
"Definition Request Being Rejected";
case 121:
return "Message Rejected Since PartyDetailsListReqID Does Not Match with "
"Immediately Preceding Party Details Definition Request sent to "
"MSGW when using On-demand administrative information";
case 122:
return "Party Details Definition Request sent to MSGW Being Rejected Due "
"to Corresponding Business Message Being Rejected";
case 123:
return "Another Message Sent in Between Party Details Definition Request "
"and Business Message when using On-demand administrative "
"information";
case 124:
return "Cannot Have More Than One In-Flight Mass Order Status Request in "
"Progress";
case 125:
return "Exceeded Maximum Number of In-Flight Order Status Requests";
case 126:
return "Cannot Have More Than One In-Flight Party Details List Request "
"in Progress";
case 127:
return "Party Details List Request is Missing Requesting Party ID and "
"Party ID";
case 128:
return "Party Details List Request cannot contain both RequestingPartyID "
"and PartyID";
case 129:
return "Party Details Definition Request Being Rejected Since Another "
"Message was sent in Between On-Demand Message";
case 130:
return "Buy Side Firm ID Does Not Match Sell Side Firm ID in New Order "
"Cross";
case 131:
return "Message Type Not Supported on Backup Instance";
case 132:
return "New Order Cross Does Not Contain Buy Side Followed by Sell Side";
}
return "Unknown or invalid business reject reason.";
}
const char *translate_negotiation_reject(int reason) {
// https://www.cmegroup.com/confluence/display/EPICSANDBOX/Negotiation+Reject
// 01/13/2020
switch (reason) {
case 0:
return "HMACNotAuthenticated: failed authentication because identity is "
"not recognized";
case 1:
return "HMACNotAvailable: HMAC component is not responding (5sec)";
case 2:
return "InvalidUUID: UUID is not greater than the one last used or value "
"sent by the client is out of acceptable range (MIN, MAX)";
case 3:
return "InvalidTimestamp: Timestamp is not greater than the one last "
"used or value sent by the client is out of acceptable range "
"(MIN, MAX)";
case 4:
return "RequiredHMACSignatureMissing: empty bytes in HMACSignature field";
case 5:
return "RequiredAccessKeyIDMissing: empty bytes in AccessKeyID field";
case 6:
return "RequiredSessionMissing: empty bytes in Session field";
case 7:
return "RequiredFirmMissing: empty bytes in Firm field";
case 8:
return "RequiredUUIDMisssing: null value in UUID field";
case 9:
return "RequiredRequestTimestampMissing: null value in RequestTimestamp "
"field";
case 10:
return "SessionBlocked: session and firm are not authorized for this "
"port";
case 12:
return "InvalidAccessKeyID: contains non-printable ASCII character";
case 13:
return "InvalidSession: contains non-printable ASCII character";
case 14:
return "InvalidFirm: contains non-printable ASCII character";
case 15:
return "Volume Controls - exceeding TPS limit as defined for volume "
"controls (reject action)";
case 16:
return "SplitMessageRejected - Messages queued due to split message "
"penalty being rejected because of logout or disconnect";
case 17:
return "SplitMessageQueue - Reached threshold of messages queued due to "
"split message penalty";
case 27:
return "DesignatedBackup - Using Designated backup before designated "
"primary not allowed";
}
return "Unknown or invalid Negotiate reject reason.";
}
const char *translate_establish_reject(int reason) {
switch (reason) {
case 0:
return "HMACNotAuthenticated: failed authentication because identity is "
"not recognized";
case 1:
return "HMACNotAvailable: HMAC component is not responding (5sec)";
case 2:
return "InvalidUUID: UUID is not greater than the one last used or value "
"sent by the client is out of acceptable range (MIN, MAX)";
case 3:
return "InvalidTimestamp: Timestamp is not greater than the one last "
"used or value sent by the client is out of acceptable range "
"(MIN, MAX)";
case 4:
return "RequiredHMACSignatureMissing: empty bytes in HMACSignature field";
case 5:
return "RequiredAccessKeyIDMissing: empty bytes in AccessKeyID field";
case 6:
return "RequiredSessionMissing: empty bytes in Session field";
case 7:
return "RequiredFirmMissing: empty bytes in Firm field";
case 8:
return "RequiredUUIDMisssing: null value in UUID field";
case 9:
return "RequiredRequestTimestampMissing: null value in RequestTimestamp "
"field";
case 10:
return "SessionBlocked: session and firm are not authorized for this "
"port";
case 11:
return "InvalidKeepAliveInterval: value is out of acceptable range (MIN, "
"MAX)";
case 12:
return "InvalidAccessKeyID: contains non-printable ASCII character";
case 13:
return "InvalidSession: contains non-printable ASCII character";
case 14:
return "InvalidFirm: contains non-printable ASCII character";
case 15:
return "Volume Controls - exceeding TPS limit as defined for volume "
"controls (reject action)";
case 16:
return "SplitMessageRejected - Messages queued due to split message "
"penalty being rejected because of logout or disconnect";
case 17:
return "SplitMessageQueue - Reached threshold of messages queued due to "
"split message penalty";
case 18:
return "RequiredTradingSystemNameMissing: empty bytes in "
"TradingSystemName";
case 19:
return "RequiredTradingSystemVersionMissing: empty bytes in "
"TradingSystemVersion";
case 20:
return "RequiredTradingSystemVendorMissing: empty bytes in "
"TradingSystemVendor";
case 21:
return "RequiredKeepAliveIntervalMissing: null value in "
"KeepAliveInterval field";
case 22:
return "RequiredNextSeqNoMissing: empty bytes in NextSeqNo field";
case 23:
return "InvalidTradingSystemName: contains non-prinatable ASCII "
"character";
case 24:
return "InvalidTradingSystemVersion: contains non-prinatable ASCII "
"character";
case 25:
return "InvalidTradingSystemVendor: contains non-prinatable ASCII "
"character";
case 26:
return "DesignatedBackup - Using Designated backup before designated "
"primary not allowed";
case 27:
return "DesignatedBackup - Using Designated backup before designated "
"primary not allowed";
}
return "Unknown or invalid Establish reject reason.";
}
const char *translate_terminate(int reason) {
switch (reason) {
case 0:
return "Finished: session is being terminated without finalization";
case 2:
return "Unnegotiated: Sending any message when session has not been "
"Negotiated ";
case 3:
return "NotEstablished: Sending any message (except Negotiate) when "
"session has not been established";
case 4:
return "AlreadyNegotiated: NegotiationResponse was already sent; "
"Negotiate was redundant";
case 5:
return "NegotiationInProgress: Previous Negotiate still being processed; "
"Wait for NegotiationResponse or timeout";
case 6:
return "AlreadyEstablished: EstablishmentAck was already sent; Establish "
"was redundant";
case 7:
return "EstablishInProgress: Previous Establish still being processed; "
"Wait for EstablishmentAck or timeout";
case 8:
return "CMEAdministeredPortClosure: due to invalid number of "
"Negotiate/Establish attempts being exceeded";
case 9:
return "Volume Controls - exceeding TPS limit as defined for volume "
"controls (logout action)";
case 10:
return "InvalidNextSeqNo - value is not greater than the one last used "
"for same UUID or value sent by the client is out of acceptable "
"range (MIN, MAX)";
case 11:
return "InvalidMsgSeqNo - value is lower than the last one used for the "
"same UUID or value is not initialized to 1 at the beginning of "
"the week or value sent by the client is out of acceptable range "
"(MIN, MAX)";
case 12:
return "InvalidLastSeqNo - value is lower than the last one used for the "
"same UUID or value sent by the client is out of acceptable range "
"(MIN, MAX)";
case 13:
return "InvalidUUID: UUID value does not match current UUID or value "
"sent by the client is out of acceptable range (MIN, MAX)";
case 14:
return "InvalidTimestamp: Timestamp value does not match with "
"RequestTimestamp sent by CME or value sent by the client is out "
"of acceptable range (MIN, MAX)";
case 15:
return "RequiredUUIDMisssing: null value in UUID field";
case 16:
return "RequiredRequestTimestampMissing: null value in RequestTimestamp "
"field";
case 17:
return "RequiredCodeMisssing: null value in Code field";
case 18:
return "InvalidSOFH: Invalid message length or invalid encoding type "
"specified";
case 19:
return "DecodingError: Incoming message could not be decoded";
case 20:
return "KeepAliveIntervalLapsed: KeepAliveInterval has lapsed without "
"any response so terminating session";
case 21:
return "RequiredNextSeqNoMissing: null value in NextSeqNo field";
case 22:
return "RequiredKeepAliveIntervalLapsedMissing: null value in "
"KeepAliveIntervalLapsed field";
case 23:
return "Non-Negotiate/Establish message received when not "
"Negotiated/Established";
case 24:
return "TerminateInProgress: Previous Terminate still being processed; "
"Wait for Terminate or timeout";
case 25:
return "Other: any other error condition not mentioned above";
case 26:
return "DisconnectFromPrimary: Backup session will be terminated as "
"well}";
default:
return "Unknown error";
}
}
} // namespace ilink3
} // namespace orderman
} // namespace bts
@@ -0,0 +1,51 @@
#pragma once
#include "lib/domain_model/common.h"
#include "lib/domain_model/Security.pb.h"
#include "lib/api/services/OrderService.pb.h"
#include "lib/orderman/orderman_utils/orderman_common.h"
#include "generated/SideReq.h"
#include "generated/OrderTypeReq.h"
#include "generated/OrderType.h"
#include "generated/TimeInForce.h"
#include "generated/ManualOrdInd.h"
#include "generated/ManualOrdIndReq.h"
#include "generated/ClearingAcctType.h"
#include "generated/SMPI.h"
#include "generated/CustOrderCapacity.h"
#include <tuple>
namespace bts {
namespace orderman {
namespace ilink3 {
iLinkBinary::SideReq::Value map_side(api::Side side);
api::Side map_side(iLinkBinary::SideReq::Value side);
inline int64_t to_price9(Price p) {
return std::round(p.value() * 1E9);
}
iLinkBinary::OrderTypeReq::Value map_order_type(api::orderman::NativeOrderType ot);
api::orderman::NativeOrderType map_order_type(iLinkBinary::OrderType::Value);
iLinkBinary::TimeInForce::Value map_time_in_force(api::orderman::TimeInForce);
api::orderman::TimeInForce map_time_in_force(iLinkBinary::TimeInForce::Value);
iLinkBinary::ManualOrdIndReq::Value map_manual_order_indicator(api::orderman::ManualOrderIndicator moi);
api::orderman::ManualOrderIndicator map_manual_order_indicator(iLinkBinary::ManualOrdIndReq::Value moi);
iLinkBinary::ClearingAcctType::Value map_customer_or_firm(api::orderman::CustomerOrFirm c);
iLinkBinary::SMPI::Value map_smp_instruction(api::orderman::SelfMatchPreventionInstruction smpi);
iLinkBinary::CustOrderCapacity::Value map_cust_cti(api::orderman::CTICode code);
const char* translate_business_reject_reason(int reason) ;
const char* translate_negotiation_reject(int reason) ;
const char* translate_establish_reject(int reason) ;
const char* translate_terminate(int reason) ;
}}}//namespace
+15
View File
@@ -0,0 +1,15 @@
#pragma once
#include <functional>
#include <string>
namespace bts {
namespace orderman {
namespace ilink3 {
enum class RawMessageDirection {INCOMING, OUTGOING};
struct RawMessageData {
std::string json;
std::string binary;
};
using RawMsg = std::function<RawMessageData(void)>;
}}}
@@ -0,0 +1,188 @@
#include "SessionStore.h"
#include "lib/jetfix/store/session_name.h"
#include "lib/orderman/actors/ilink3/IL3GW.h"
#include "lib/orderman/actors/ilink3/IL3SecDB.h"
#include "lib/orderman/orderman_utils/ExecutionReport.h"
#include "lib/utils/base64.hpp"
#include "lib/utils/bts_chrono.h"
#include "lib/utils/file_utils.h"
#include "lib/utils/thread_manager.h"
#include <boost/filesystem.hpp>
#include <nlohmann/json.hpp>
#include <tbb/concurrent_queue.h>
#include <thread>
namespace bts {
namespace orderman {
namespace ilink3 {
using Action = std::function<void()>;
static std::string base_store_path(std::string base_path,
std::string const &connection_name,
std::string const &session_name) {
if (base_path.empty())
base_path = os::bts_data_path();
boost::filesystem::path result = utils::normalize_data_path(base_path);
result /= "ilink3";
result /= connection_name;
result /= session_name;
if (!boost::filesystem::exists(result))
bts_assert(boost::filesystem::create_directories(result));
return result.string();
}
static std::string raw_msg_log_path(std::string const &store_path) {
boost::filesystem::path result = store_path;
result /= "log";
return result.string();
}
static std::string session_path(std::string const &store_path) {
boost::filesystem::path result = store_path;
result /= "seqnum";
return result.string();
}
static std::vector<ExecutionReport>
load_replay(IL3SecDB &db, config::JetFIX::Connection const *cfg,
std::string const &path) {
IL3Parser parser(cfg, &db);
std::vector<ExecutionReport> result;
if (boost::filesystem::exists(path)) {
std::ifstream f(path);
nlohmann::json j;
for (;;) {
while (!f.eof() && f.get() != '{')
;
if (f.eof())
break;
else {
f.unget();
f >> j;
if (j["direction"] == "<") {
auto ts = datetime::parse_FIX_timestamp(j["time"]);
if (!datetime::is_at_least_secs_old(ts,
cfg->seconds_to_replay())) {
auto msg_bytes = websocketpp::base64_decode(j["msg_sbe"]);
if (msg_bytes.size()) {
auto maybe_ex_rep = parser.try_parse_exe_report(msg_bytes);
if (maybe_ex_rep)
result.push_back(maybe_ex_rep.value());
}
}
}
}
}
}
return result;
}
static void save_raw_msg(std::ofstream &out, RawMessageDirection d, size_t sn,
RawMessageData const &msg) {
nlohmann::json j;
j["direction"] = d == RawMessageDirection::INCOMING ? "<" : ">";
j["sn"] = sn;
j["time"] = datetime::FIX_timestamp(datetime::now());
j["msg_sbe"] = websocketpp::base64_encode(msg.binary);
try {
j["msg_json"] = nlohmann::json::parse(msg.json);
} catch (...) {
LOG(iLink3, Error, "Skipping troublesome json encoded message: {}",
msg.json);
j["msg_json"] = nlohmann::json{};
}
out << j << std::endl;
out.flush();
}
void save_session(std::string const &path, Session const &session) {
std::ofstream f(path);
f << session.uuid << " " << session.to_cme << " " << session.from_cme
<< std::endl;
}
Session load_session(jetfix::SessionName const &sn, std::string const &path) {
auto result = Session{sn.as_uint64_t, 0, 0};
if (boost::filesystem::exists(path)) {
std::ifstream f(path);
uint64_t dummy;
f >> dummy >> result.to_cme >> result.from_cme;
} else
save_session(path, result);
return result;
}
struct SessionStore::PImpl {
PImpl(IL3SecDB &db, jetfix::SessionName const &session_name,
config::JetFIX::Connection const *cfg, std::string const &base_path)
: cfg(cfg), store_path(base_store_path(base_path, cfg->name(),
session_name.as_string)),
replay_messages(load_replay(db, cfg, raw_msg_log_path(store_path))),
starting_session_descriptor(
load_session(session_name, session_path(store_path))),
raw_msg_store() {
raw_msg_store.open(raw_msg_log_path(store_path), std::ofstream::app);
}
const config::JetFIX::Connection *cfg;
const std::string store_path;
const std::vector<ExecutionReport> replay_messages;
const Session starting_session_descriptor;
std::atomic<bool> done = false;
tbb::concurrent_queue<Action> event_queue;
std::thread storage_thread;
std::ofstream raw_msg_store;
};
SessionStore::SessionStore(IL3SecDB &db,
jetfix::SessionName const &session_name,
config::JetFIX::Connection const *cfg,
std::string base_path)
: me(new PImpl(db, session_name, cfg, base_path)) {
me->storage_thread = threading::make_thread_from_function(
"il3_store", [this]() { do_work(); });
}
std::vector<ExecutionReport> const &SessionStore::replay() const {
return me->replay_messages;
}
Session SessionStore::starting_session_descriptor() const {
return me->starting_session_descriptor;
}
void SessionStore::async_update_session(Session const &s) {
me->event_queue.push(
Action{[this, s]() { save_session(session_path(me->store_path), s); }});
}
void SessionStore::async_store_raw_msg(RawMessageDirection d, size_t sn,
RawMsg const &ms) {
me->event_queue.push(Action{
[this, d, sn, ms]() { save_raw_msg(me->raw_msg_store, d, sn, ms()); }});
}
void SessionStore::do_work() {
while (!me->done) {
Action a;
if (me->event_queue.try_pop(a))
a();
else
std::this_thread::sleep_for(100ms);
}
}
SessionStore::~SessionStore() {
me->done = true;
me->storage_thread.join();
}
} // namespace ilink3
} // namespace orderman
} // namespace bts
@@ -0,0 +1,41 @@
#pragma once
#include <memory>
#include <vector>
#include <functional>
#include "lib/config/protos/jetfix.pb.h"
#include "RawMsg.h"
// fwds
namespace bts {
namespace jetfix {struct SessionName;}
namespace orderman {class ExecutionReport;}
}
namespace bts {
namespace orderman {
namespace ilink3 {
struct Session;
class IL3SecDB;
class SessionStore {
public:
SessionStore(IL3SecDB &,
jetfix::SessionName const&session_name,
config::JetFIX::Connection const*,
std::string base_path="");
~SessionStore() ;
Session starting_session_descriptor() const;
std::vector<ExecutionReport> const &replay() const;
void async_update_session(Session const&) ;
void async_store_raw_msg(RawMessageDirection, size_t sn,RawMsg const&);
private:
void do_work();
struct PImpl;
std::unique_ptr<PImpl> me;
};
}}}
@@ -0,0 +1,25 @@
#pragma once
#include "lib/utils/datetime.h"
#include "lib/utils/TimeStamp.h"
namespace bts {
namespace orderman {
namespace ilink3 {
struct TimeKeeper {
uint64_t unix_time;
utils::TimeStamp start;
TimeKeeper() :
unix_time(datetime::unix_nano_now()),
start(utils::TimeStamp::now())
{}
uint64_t now() const {
auto now = utils::TimeStamp::now();
auto delta = now - start;
return unix_time + delta.nanoseconds();
}
};
}}} //namespace
@@ -0,0 +1,144 @@
/* Generated SBE (Simple Binary Encoding) message codec */
#ifndef _ILINKBINARY_AVGPXIND_H_
#define _ILINKBINARY_AVGPXIND_H_
#if defined(SBE_HAVE_CMATH)
/* cmath needed for std::numeric_limits<double>::quiet_NaN() */
# include <cmath>
# define SBE_FLOAT_NAN std::numeric_limits<float>::quiet_NaN()
# define SBE_DOUBLE_NAN std::numeric_limits<double>::quiet_NaN()
#else
/* math.h needed for NAN */
# include <math.h>
# define SBE_FLOAT_NAN NAN
# define SBE_DOUBLE_NAN NAN
#endif
#if __cplusplus >= 201103L
# define SBE_CONSTEXPR constexpr
# define SBE_NOEXCEPT noexcept
#else
# define SBE_CONSTEXPR
# define SBE_NOEXCEPT
#endif
#if __cplusplus >= 201402L
# define SBE_CONSTEXPR_14 constexpr
#else
# define SBE_CONSTEXPR_14
#endif
#if __cplusplus >= 201703L
# include <string_view>
# define SBE_NODISCARD [[nodiscard]]
#else
# define SBE_NODISCARD
#endif
#if !defined(__STDC_LIMIT_MACROS)
# define __STDC_LIMIT_MACROS 1
#endif
#include <cstdint>
#include <cstring>
#include <iomanip>
#include <limits>
#include <ostream>
#include <stdexcept>
#include <sstream>
#include <string>
#include <vector>
#if defined(WIN32) || defined(_WIN32)
# define SBE_BIG_ENDIAN_ENCODE_16(v) _byteswap_ushort(v)
# define SBE_BIG_ENDIAN_ENCODE_32(v) _byteswap_ulong(v)
# define SBE_BIG_ENDIAN_ENCODE_64(v) _byteswap_uint64(v)
# define SBE_LITTLE_ENDIAN_ENCODE_16(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_32(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_64(v) (v)
#elif __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
# define SBE_BIG_ENDIAN_ENCODE_16(v) __builtin_bswap16(v)
# define SBE_BIG_ENDIAN_ENCODE_32(v) __builtin_bswap32(v)
# define SBE_BIG_ENDIAN_ENCODE_64(v) __builtin_bswap64(v)
# define SBE_LITTLE_ENDIAN_ENCODE_16(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_32(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_64(v) (v)
#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
# define SBE_LITTLE_ENDIAN_ENCODE_16(v) __builtin_bswap16(v)
# define SBE_LITTLE_ENDIAN_ENCODE_32(v) __builtin_bswap32(v)
# define SBE_LITTLE_ENDIAN_ENCODE_64(v) __builtin_bswap64(v)
# define SBE_BIG_ENDIAN_ENCODE_16(v) (v)
# define SBE_BIG_ENDIAN_ENCODE_32(v) (v)
# define SBE_BIG_ENDIAN_ENCODE_64(v) (v)
#else
# error "Byte Ordering of platform not determined. Set __BYTE_ORDER__ manually before including this file."
#endif
#if defined(SBE_NO_BOUNDS_CHECK)
# define SBE_BOUNDS_CHECK_EXPECT(exp,c) (false)
#elif defined(_MSC_VER)
# define SBE_BOUNDS_CHECK_EXPECT(exp,c) (exp)
#else
# define SBE_BOUNDS_CHECK_EXPECT(exp,c) (__builtin_expect(exp,c))
#endif
#define SBE_NULLVALUE_INT8 (std::numeric_limits<std::int8_t>::min)()
#define SBE_NULLVALUE_INT16 (std::numeric_limits<std::int16_t>::min)()
#define SBE_NULLVALUE_INT32 (std::numeric_limits<std::int32_t>::min)()
#define SBE_NULLVALUE_INT64 (std::numeric_limits<std::int64_t>::min)()
#define SBE_NULLVALUE_UINT8 (std::numeric_limits<std::uint8_t>::max)()
#define SBE_NULLVALUE_UINT16 (std::numeric_limits<std::uint16_t>::max)()
#define SBE_NULLVALUE_UINT32 (std::numeric_limits<std::uint32_t>::max)()
#define SBE_NULLVALUE_UINT64 (std::numeric_limits<std::uint64_t>::max)()
namespace iLinkBinary {
class AvgPxInd
{
public:
enum Value
{
NoAveragePricing = (std::uint8_t)0,
TradeispartofanAveragePriceGroupIdentifiedbytheAvgPxGrpID = (std::uint8_t)1,
TradeispartofaNotionalValueAveragePriceGroup = (std::uint8_t)3,
NULL_VALUE = (std::uint8_t)255
};
static AvgPxInd::Value get(const std::uint8_t value)
{
switch (value)
{
case 0: return NoAveragePricing;
case 1: return TradeispartofanAveragePriceGroupIdentifiedbytheAvgPxGrpID;
case 3: return TradeispartofaNotionalValueAveragePriceGroup;
case 255: return NULL_VALUE;
}
throw std::runtime_error("unknown value for enum AvgPxInd [E103]");
}
static const char* c_str(const AvgPxInd::Value value)
{
switch (value)
{
case NoAveragePricing: return "NoAveragePricing";
case TradeispartofanAveragePriceGroupIdentifiedbytheAvgPxGrpID: return "TradeispartofanAveragePriceGroupIdentifiedbytheAvgPxGrpID";
case TradeispartofaNotionalValueAveragePriceGroup: return "TradeispartofaNotionalValueAveragePriceGroup";
case NULL_VALUE: return "NULL_VALUE";
}
throw std::runtime_error("unknown value for enum AvgPxInd [E103]:");
}
template<typename CharT, typename Traits>
friend std::basic_ostream<CharT, Traits>& operator << (
std::basic_ostream<CharT, Traits>& os, AvgPxInd::Value m)
{
return os << AvgPxInd::c_str(m);
}
};
}
#endif
@@ -0,0 +1,141 @@
/* Generated SBE (Simple Binary Encoding) message codec */
#ifndef _ILINKBINARY_BOOLEANFLAG_H_
#define _ILINKBINARY_BOOLEANFLAG_H_
#if defined(SBE_HAVE_CMATH)
/* cmath needed for std::numeric_limits<double>::quiet_NaN() */
# include <cmath>
# define SBE_FLOAT_NAN std::numeric_limits<float>::quiet_NaN()
# define SBE_DOUBLE_NAN std::numeric_limits<double>::quiet_NaN()
#else
/* math.h needed for NAN */
# include <math.h>
# define SBE_FLOAT_NAN NAN
# define SBE_DOUBLE_NAN NAN
#endif
#if __cplusplus >= 201103L
# define SBE_CONSTEXPR constexpr
# define SBE_NOEXCEPT noexcept
#else
# define SBE_CONSTEXPR
# define SBE_NOEXCEPT
#endif
#if __cplusplus >= 201402L
# define SBE_CONSTEXPR_14 constexpr
#else
# define SBE_CONSTEXPR_14
#endif
#if __cplusplus >= 201703L
# include <string_view>
# define SBE_NODISCARD [[nodiscard]]
#else
# define SBE_NODISCARD
#endif
#if !defined(__STDC_LIMIT_MACROS)
# define __STDC_LIMIT_MACROS 1
#endif
#include <cstdint>
#include <cstring>
#include <iomanip>
#include <limits>
#include <ostream>
#include <stdexcept>
#include <sstream>
#include <string>
#include <vector>
#if defined(WIN32) || defined(_WIN32)
# define SBE_BIG_ENDIAN_ENCODE_16(v) _byteswap_ushort(v)
# define SBE_BIG_ENDIAN_ENCODE_32(v) _byteswap_ulong(v)
# define SBE_BIG_ENDIAN_ENCODE_64(v) _byteswap_uint64(v)
# define SBE_LITTLE_ENDIAN_ENCODE_16(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_32(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_64(v) (v)
#elif __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
# define SBE_BIG_ENDIAN_ENCODE_16(v) __builtin_bswap16(v)
# define SBE_BIG_ENDIAN_ENCODE_32(v) __builtin_bswap32(v)
# define SBE_BIG_ENDIAN_ENCODE_64(v) __builtin_bswap64(v)
# define SBE_LITTLE_ENDIAN_ENCODE_16(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_32(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_64(v) (v)
#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
# define SBE_LITTLE_ENDIAN_ENCODE_16(v) __builtin_bswap16(v)
# define SBE_LITTLE_ENDIAN_ENCODE_32(v) __builtin_bswap32(v)
# define SBE_LITTLE_ENDIAN_ENCODE_64(v) __builtin_bswap64(v)
# define SBE_BIG_ENDIAN_ENCODE_16(v) (v)
# define SBE_BIG_ENDIAN_ENCODE_32(v) (v)
# define SBE_BIG_ENDIAN_ENCODE_64(v) (v)
#else
# error "Byte Ordering of platform not determined. Set __BYTE_ORDER__ manually before including this file."
#endif
#if defined(SBE_NO_BOUNDS_CHECK)
# define SBE_BOUNDS_CHECK_EXPECT(exp,c) (false)
#elif defined(_MSC_VER)
# define SBE_BOUNDS_CHECK_EXPECT(exp,c) (exp)
#else
# define SBE_BOUNDS_CHECK_EXPECT(exp,c) (__builtin_expect(exp,c))
#endif
#define SBE_NULLVALUE_INT8 (std::numeric_limits<std::int8_t>::min)()
#define SBE_NULLVALUE_INT16 (std::numeric_limits<std::int16_t>::min)()
#define SBE_NULLVALUE_INT32 (std::numeric_limits<std::int32_t>::min)()
#define SBE_NULLVALUE_INT64 (std::numeric_limits<std::int64_t>::min)()
#define SBE_NULLVALUE_UINT8 (std::numeric_limits<std::uint8_t>::max)()
#define SBE_NULLVALUE_UINT16 (std::numeric_limits<std::uint16_t>::max)()
#define SBE_NULLVALUE_UINT32 (std::numeric_limits<std::uint32_t>::max)()
#define SBE_NULLVALUE_UINT64 (std::numeric_limits<std::uint64_t>::max)()
namespace iLinkBinary {
class BooleanFlag
{
public:
enum Value
{
False = (std::uint8_t)0,
True = (std::uint8_t)1,
NULL_VALUE = (std::uint8_t)255
};
static BooleanFlag::Value get(const std::uint8_t value)
{
switch (value)
{
case 0: return False;
case 1: return True;
case 255: return NULL_VALUE;
}
throw std::runtime_error("unknown value for enum BooleanFlag [E103]");
}
static const char* c_str(const BooleanFlag::Value value)
{
switch (value)
{
case False: return "False";
case True: return "True";
case NULL_VALUE: return "NULL_VALUE";
}
throw std::runtime_error("unknown value for enum BooleanFlag [E103]:");
}
template<typename CharT, typename Traits>
friend std::basic_ostream<CharT, Traits>& operator << (
std::basic_ostream<CharT, Traits>& os, BooleanFlag::Value m)
{
return os << BooleanFlag::c_str(m);
}
};
}
#endif
@@ -0,0 +1,141 @@
/* Generated SBE (Simple Binary Encoding) message codec */
#ifndef _ILINKBINARY_BOOLEANNULL_H_
#define _ILINKBINARY_BOOLEANNULL_H_
#if defined(SBE_HAVE_CMATH)
/* cmath needed for std::numeric_limits<double>::quiet_NaN() */
# include <cmath>
# define SBE_FLOAT_NAN std::numeric_limits<float>::quiet_NaN()
# define SBE_DOUBLE_NAN std::numeric_limits<double>::quiet_NaN()
#else
/* math.h needed for NAN */
# include <math.h>
# define SBE_FLOAT_NAN NAN
# define SBE_DOUBLE_NAN NAN
#endif
#if __cplusplus >= 201103L
# define SBE_CONSTEXPR constexpr
# define SBE_NOEXCEPT noexcept
#else
# define SBE_CONSTEXPR
# define SBE_NOEXCEPT
#endif
#if __cplusplus >= 201402L
# define SBE_CONSTEXPR_14 constexpr
#else
# define SBE_CONSTEXPR_14
#endif
#if __cplusplus >= 201703L
# include <string_view>
# define SBE_NODISCARD [[nodiscard]]
#else
# define SBE_NODISCARD
#endif
#if !defined(__STDC_LIMIT_MACROS)
# define __STDC_LIMIT_MACROS 1
#endif
#include <cstdint>
#include <cstring>
#include <iomanip>
#include <limits>
#include <ostream>
#include <stdexcept>
#include <sstream>
#include <string>
#include <vector>
#if defined(WIN32) || defined(_WIN32)
# define SBE_BIG_ENDIAN_ENCODE_16(v) _byteswap_ushort(v)
# define SBE_BIG_ENDIAN_ENCODE_32(v) _byteswap_ulong(v)
# define SBE_BIG_ENDIAN_ENCODE_64(v) _byteswap_uint64(v)
# define SBE_LITTLE_ENDIAN_ENCODE_16(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_32(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_64(v) (v)
#elif __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
# define SBE_BIG_ENDIAN_ENCODE_16(v) __builtin_bswap16(v)
# define SBE_BIG_ENDIAN_ENCODE_32(v) __builtin_bswap32(v)
# define SBE_BIG_ENDIAN_ENCODE_64(v) __builtin_bswap64(v)
# define SBE_LITTLE_ENDIAN_ENCODE_16(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_32(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_64(v) (v)
#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
# define SBE_LITTLE_ENDIAN_ENCODE_16(v) __builtin_bswap16(v)
# define SBE_LITTLE_ENDIAN_ENCODE_32(v) __builtin_bswap32(v)
# define SBE_LITTLE_ENDIAN_ENCODE_64(v) __builtin_bswap64(v)
# define SBE_BIG_ENDIAN_ENCODE_16(v) (v)
# define SBE_BIG_ENDIAN_ENCODE_32(v) (v)
# define SBE_BIG_ENDIAN_ENCODE_64(v) (v)
#else
# error "Byte Ordering of platform not determined. Set __BYTE_ORDER__ manually before including this file."
#endif
#if defined(SBE_NO_BOUNDS_CHECK)
# define SBE_BOUNDS_CHECK_EXPECT(exp,c) (false)
#elif defined(_MSC_VER)
# define SBE_BOUNDS_CHECK_EXPECT(exp,c) (exp)
#else
# define SBE_BOUNDS_CHECK_EXPECT(exp,c) (__builtin_expect(exp,c))
#endif
#define SBE_NULLVALUE_INT8 (std::numeric_limits<std::int8_t>::min)()
#define SBE_NULLVALUE_INT16 (std::numeric_limits<std::int16_t>::min)()
#define SBE_NULLVALUE_INT32 (std::numeric_limits<std::int32_t>::min)()
#define SBE_NULLVALUE_INT64 (std::numeric_limits<std::int64_t>::min)()
#define SBE_NULLVALUE_UINT8 (std::numeric_limits<std::uint8_t>::max)()
#define SBE_NULLVALUE_UINT16 (std::numeric_limits<std::uint16_t>::max)()
#define SBE_NULLVALUE_UINT32 (std::numeric_limits<std::uint32_t>::max)()
#define SBE_NULLVALUE_UINT64 (std::numeric_limits<std::uint64_t>::max)()
namespace iLinkBinary {
class BooleanNULL
{
public:
enum Value
{
False = (std::uint8_t)0,
True = (std::uint8_t)1,
NULL_VALUE = (std::uint8_t)255
};
static BooleanNULL::Value get(const std::uint8_t value)
{
switch (value)
{
case 0: return False;
case 1: return True;
case 255: return NULL_VALUE;
}
throw std::runtime_error("unknown value for enum BooleanNULL [E103]");
}
static const char* c_str(const BooleanNULL::Value value)
{
switch (value)
{
case False: return "False";
case True: return "True";
case NULL_VALUE: return "NULL_VALUE";
}
throw std::runtime_error("unknown value for enum BooleanNULL [E103]:");
}
template<typename CharT, typename Traits>
friend std::basic_ostream<CharT, Traits>& operator << (
std::basic_ostream<CharT, Traits>& os, BooleanNULL::Value m)
{
return os << BooleanNULL::c_str(m);
}
};
}
#endif
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,141 @@
/* Generated SBE (Simple Binary Encoding) message codec */
#ifndef _ILINKBINARY_CLEARINGACCTTYPE_H_
#define _ILINKBINARY_CLEARINGACCTTYPE_H_
#if defined(SBE_HAVE_CMATH)
/* cmath needed for std::numeric_limits<double>::quiet_NaN() */
# include <cmath>
# define SBE_FLOAT_NAN std::numeric_limits<float>::quiet_NaN()
# define SBE_DOUBLE_NAN std::numeric_limits<double>::quiet_NaN()
#else
/* math.h needed for NAN */
# include <math.h>
# define SBE_FLOAT_NAN NAN
# define SBE_DOUBLE_NAN NAN
#endif
#if __cplusplus >= 201103L
# define SBE_CONSTEXPR constexpr
# define SBE_NOEXCEPT noexcept
#else
# define SBE_CONSTEXPR
# define SBE_NOEXCEPT
#endif
#if __cplusplus >= 201402L
# define SBE_CONSTEXPR_14 constexpr
#else
# define SBE_CONSTEXPR_14
#endif
#if __cplusplus >= 201703L
# include <string_view>
# define SBE_NODISCARD [[nodiscard]]
#else
# define SBE_NODISCARD
#endif
#if !defined(__STDC_LIMIT_MACROS)
# define __STDC_LIMIT_MACROS 1
#endif
#include <cstdint>
#include <cstring>
#include <iomanip>
#include <limits>
#include <ostream>
#include <stdexcept>
#include <sstream>
#include <string>
#include <vector>
#if defined(WIN32) || defined(_WIN32)
# define SBE_BIG_ENDIAN_ENCODE_16(v) _byteswap_ushort(v)
# define SBE_BIG_ENDIAN_ENCODE_32(v) _byteswap_ulong(v)
# define SBE_BIG_ENDIAN_ENCODE_64(v) _byteswap_uint64(v)
# define SBE_LITTLE_ENDIAN_ENCODE_16(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_32(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_64(v) (v)
#elif __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
# define SBE_BIG_ENDIAN_ENCODE_16(v) __builtin_bswap16(v)
# define SBE_BIG_ENDIAN_ENCODE_32(v) __builtin_bswap32(v)
# define SBE_BIG_ENDIAN_ENCODE_64(v) __builtin_bswap64(v)
# define SBE_LITTLE_ENDIAN_ENCODE_16(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_32(v) (v)
# define SBE_LITTLE_ENDIAN_ENCODE_64(v) (v)
#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
# define SBE_LITTLE_ENDIAN_ENCODE_16(v) __builtin_bswap16(v)
# define SBE_LITTLE_ENDIAN_ENCODE_32(v) __builtin_bswap32(v)
# define SBE_LITTLE_ENDIAN_ENCODE_64(v) __builtin_bswap64(v)
# define SBE_BIG_ENDIAN_ENCODE_16(v) (v)
# define SBE_BIG_ENDIAN_ENCODE_32(v) (v)
# define SBE_BIG_ENDIAN_ENCODE_64(v) (v)
#else
# error "Byte Ordering of platform not determined. Set __BYTE_ORDER__ manually before including this file."
#endif
#if defined(SBE_NO_BOUNDS_CHECK)
# define SBE_BOUNDS_CHECK_EXPECT(exp,c) (false)
#elif defined(_MSC_VER)
# define SBE_BOUNDS_CHECK_EXPECT(exp,c) (exp)
#else
# define SBE_BOUNDS_CHECK_EXPECT(exp,c) (__builtin_expect(exp,c))
#endif
#define SBE_NULLVALUE_INT8 (std::numeric_limits<std::int8_t>::min)()
#define SBE_NULLVALUE_INT16 (std::numeric_limits<std::int16_t>::min)()
#define SBE_NULLVALUE_INT32 (std::numeric_limits<std::int32_t>::min)()
#define SBE_NULLVALUE_INT64 (std::numeric_limits<std::int64_t>::min)()
#define SBE_NULLVALUE_UINT8 (std::numeric_limits<std::uint8_t>::max)()
#define SBE_NULLVALUE_UINT16 (std::numeric_limits<std::uint16_t>::max)()
#define SBE_NULLVALUE_UINT32 (std::numeric_limits<std::uint32_t>::max)()
#define SBE_NULLVALUE_UINT64 (std::numeric_limits<std::uint64_t>::max)()
namespace iLinkBinary {
class ClearingAcctType
{
public:
enum Value
{
Customer = (std::uint8_t)0,
Firm = (std::uint8_t)1,
NULL_VALUE = (std::uint8_t)255
};
static ClearingAcctType::Value get(const std::uint8_t value)
{
switch (value)
{
case 0: return Customer;
case 1: return Firm;
case 255: return NULL_VALUE;
}
throw std::runtime_error("unknown value for enum ClearingAcctType [E103]");
}
static const char* c_str(const ClearingAcctType::Value value)
{
switch (value)
{
case Customer: return "Customer";
case Firm: return "Firm";
case NULL_VALUE: return "NULL_VALUE";
}
throw std::runtime_error("unknown value for enum ClearingAcctType [E103]:");
}
template<typename CharT, typename Traits>
friend std::basic_ostream<CharT, Traits>& operator << (
std::basic_ostream<CharT, Traits>& os, ClearingAcctType::Value m)
{
return os << ClearingAcctType::c_str(m);
}
};
}
#endif
Loaded 100 of 272 files, more files were not shown because too many files have changed in this diff. Show more