Compare commits

..
Author SHA1 Message Date
John Colvin da62a82518 chore: Get bybit md plugin symbols from config 2025-06-10 14:56:33 +00:00
John Colvin 6170367b51 fix build: Don't export symbols with pybind11 hidden types
Release build was emitting werrors about symbols being declared with
greater visibility than pybind11::object
2025-06-10 14:54:47 +00:00
John Colvin c5b73eaace diag: Add log level config to py plugins 2025-06-10 14:54:47 +00:00
John Colvin 584ec6f7e8 fix: Init shared_ptr w/ default obj (not null)
Prob a typo
2025-06-10 06:53:58 +00:00
John Colvin c21f5d4bdb chore: Fix path in .env file (server/sparkd -> app_sparkd) 2025-06-10 06:49:24 +00:00
John Colvin ccd8ea4056 fix build: Remove include of non-existent header
I created the file locally, but ended up not using it and forgot to remove an
include before I pushed.
2025-06-10 05:41:44 +00:00
John Colvin 606526a1d9 fix build: Always return value in non-void fn 2025-06-10 05:34:25 +00:00
John Colvin 0a5f1eb825 fix: Don't compare size_t to -1 in SmallString num parsers 2025-06-10 05:33:12 +00:00
John Colvin 968d33adb6 refactor: Manage all py plugins (md and exe) in one class
All calls to/from python are performed on one thread via a simple
message queue (necessary because of the global interpreter lock).
2025-06-09 06:33:29 +00:00
John Colvin 178ea7066f chore: Add pybit to py venv 2025-06-09 06:06:50 +00:00
John Colvin 000ac9bb00 diag: Log incoming books at Info level in bybit_md_provider 2025-06-09 06:06:50 +00:00
John Colvin c56246a59f chore: Add bybit execution config to bybit_dev profile 2025-06-09 06:06:50 +00:00
John Colvin 5e66857fde Hardcode bybit_md_provider to use testnet
TODO: Add bybit md config for this and other params (api credentials,
symbols to subscribe, etc.)
2025-06-09 06:06:50 +00:00
John Colvin 04cf1261f8 chore: Add numpy to py venv
This is needed for md plugins
2025-06-09 06:06:50 +00:00
John Colvin 90f2a80568 fix: Include fill id in format str for ExecutionReport 2025-06-09 06:06:50 +00:00
John Colvin 453d516acf dev qol: Bind PySecurity __format__ to fmt::formatter impl 2025-06-09 06:06:50 +00:00
John Colvin 8f5338d672 refactor: Change floats to doubles in py bridge types
Getting rounding errors with floats
2025-06-09 06:06:50 +00:00
John Colvin 8774e55a40 refactor: Use SmallString instead of std::array in ExchangeFillID 2025-06-09 06:06:50 +00:00
John Colvin c1ba8c177e Add py bybit exe config to bybit_dev profile 2025-06-09 06:06:50 +00:00
John Colvin 2174162081 refactor: Consolidate all py interop into single lib
It was just too hard trying to make it work with separate libs. The
main issue was getting the python md and execution modules to
recognize PySecurityList and other types defined in the core module.
2025-06-09 06:06:50 +00:00
John Colvin 44948ab4db refactor: Move tick fns into gen-purpose tick_utils.h 2025-06-09 06:02:35 +00:00
John Colvin 276c80309f Change bybit_dev symbols.conf to list bybit futs (not spots) 2025-06-09 06:02:35 +00:00
John Colvin a5bc7ecde0 fix: Actually use configured logger in bybit_md_provider.py 2025-06-09 06:02:35 +00:00
John Colvin d55b5a434e dev qol: Reduce noisy logs in PositionServicePNLHelper 2025-06-09 06:02:35 +00:00
John Colvin 9204695a8c build: Add cmake script to sym link py module and plugins
We have several of these (python_core, python_md, python_execution),
so the repetative boilerplate code was starting to pile up.
2025-06-09 06:02:35 +00:00
John Colvin 4bfded94a4 deploy: Install/activate py venv in sparkd container 2025-06-09 06:02:35 +00:00
John Colvin 0ef0200f10 dev qol: Add pyrightconfig.json for type hints in py/c++ bridge 2025-06-09 06:02:35 +00:00
John Colvin d4c370729d Tweak py to get rid of false positive type hint warns 2025-06-09 06:02:34 +00:00
John Colvin d54d1b7860 feat: Support python execution plugins 2025-06-09 06:02:34 +00:00
John Colvin 680090fde7 refactor: Move py security (and related types) to python_core lib
So the execution gw plugins can use it (as well as the md plugins).
2025-06-09 06:02:34 +00:00
John Colvin c89b27d0c0 fix: Add missing switch cases for bybit & gateio 2025-06-09 05:58:37 +00:00
John Colvin 5e876b3b4d fix: Initialize bybit secs as perp futs (not spots) 2025-06-09 05:58:37 +00:00
John Colvin 0837584301 dev qol: Add wait/notify functionality to ConcurrentQueue 2025-06-09 05:58:37 +00:00
John Colvin cf8ec637f8 chore: Add CANCEL_ORDER op to binance.env 2025-06-09 05:58:37 +00:00
John Colvin 7b926ceb76 dev qol: Add ConcurrentValueStore, with notify/wait functionality
Uses std::condition_variable
2025-06-09 05:58:37 +00:00
John Colvin dd966967df fix: Don't raise ex if var isn't defined in req path 2025-06-09 05:58:37 +00:00
John Colvin e42d814c84 dev qol: Add gen-purpose validation fns for execution gws 2025-06-09 05:58:37 +00:00
John Colvin 933444ffc7 build: Use new gen-purpose python cmake fns in py md proj
Significantly simplifies CMakeLists.txt
2025-06-09 05:58:37 +00:00
John Colvin 1237de2a60 build: Add gen-purpose InitPythonModule, InitPythonPlugin cmake scripts 2025-06-09 05:58:37 +00:00
John Colvin 2dec5812f6 dev qol: Add fn to make Stopwatch and start it at once 2025-06-09 05:58:37 +00:00
John Colvin 237f525263 chore: Use same log format as sparkd in py md plugins 2025-06-09 05:58:37 +00:00
John Colvin 8a5fcfbe04 improvement: Set PyBook data in single call to reduce interop overhead 2025-06-09 05:58:37 +00:00
John Colvin ccc8faf353 Reduce log noise in py md providers 2025-06-09 05:58:37 +00:00
John Colvin 4c983c6ec1 doc: Improve py md readme
* Put examples in a table
* Show command to create symlinks for intellisense
2025-06-09 05:58:37 +00:00
John Colvin e7ee1e02dd refactor: Redesign py md so it doesn't lock the global interpreter
We can only embed one interpreter in the sparkd process, so all py
plugins need to share it.

* Move core py interop code to new lib_python_core. We're about to add a
  new lib for py execution, so it needs to be accesible from both
  execution and md libs
* Fix typo for type hint in py_md_utils.py
* Run py md providers on background thread
* Rename MdSink -> MdBridge
2025-06-09 05:54:10 +00:00
John Colvin de8edca010 build: Add lib_spark subdir first (all others depend on it) 2025-06-06 17:41:53 +00:00
John Colvin 63d5e70add dev qol: Add to_chrono fns to TimeSpan 2025-06-06 17:41:53 +00:00
John Colvin 5a77102bd2 dev qol: Add -> operators to ValueWrapper for convenience
So we can type "foo->" instead of "foo.value()" (much shorter!)
2025-06-06 17:41:53 +00:00
John Colvin ca1799fa68 build: Link against system installed ssl to support py interop
Unless we want to build python packages ourselves, the underlying clibs
they use link to the system installed ssl. Notably, the requests module.
2025-06-06 17:41:53 +00:00
John Colvin caaac014ba chore: Add pybind11, pybind11-stubgen to py venv 2025-06-06 17:41:53 +00:00
John Colvin 716be146b9 improvement: Generate py stubs for md providers
Provides intellisense for the python_md_module
2025-06-06 17:41:53 +00:00
John Colvin a57d38951e cleanup: Get rid of unecessary namespace prefixes 2025-06-06 17:41:53 +00:00
John Colvin 99dccbf6d1 cleanup: Get rid of "PyList" classes, bind to stl vector instead
I wasn't aware this was possible until recently.
2025-06-06 17:41:53 +00:00
John Colvin 5f16e88bcf refactor: Move SimpleOrderKeyMap from binance_execution to lib_spark
It provides basic functionality that will be useful in other execution gateways.
2025-06-06 17:41:53 +00:00
John Colvin 092ac21fce chore: Add smart parse and pretty name fns for proto enums
Mainly to ease the pain of representing enums as strings in python md
and execution plugins
2025-06-06 17:41:53 +00:00
John Colvin 2cd75209d2 refactor: Move ErrorList from binance_execution to lib_spark
It provides basic functionality that will be useful elsewhere
2025-06-06 17:41:53 +00:00
John Colvin 9dc5de4bdb refactor: Use SmallString std::array in AccountName 2025-06-06 17:41:53 +00:00
John Colvin 34e7e1e01c refactor: Use SmallString instead of std::array in ExchangeOrderID 2025-06-06 17:41:53 +00:00
John Colvin f3d32dd918 Add our own "SmallString" class for fixed cap strings
This will replace the std::array<char, N> we use in ExecutionReport, etc.
among other things
2025-06-06 17:41:53 +00:00
John Colvin 83e3780964 refactor: Move ExecutionReportStore to lib_spark
From lib_binance_execution
2025-06-04 04:37:02 +00:00
Adeel Khan 5d9c7502c9 CLEANUP: BYBIT Symbols.conf. 2025-06-04 04:37:02 +00:00
Adeel Khan d47958e268 ADD: BYBIT MD Working, NaN's also Added. 2025-06-04 04:37:02 +00:00
Adeel Khan 11bb764a47 ADD: BYBIT Securities showing up on frontend. Default Contract Spec 2025-06-04 04:37:02 +00:00
Adeel Khan 92f8167477 ADD: GATEIO md_provider with dynamic contract specs using Batch REST call 2025-06-04 04:37:02 +00:00
Adeel Khan 99f89ce965 ADD: GATEIO md_provider with dynamic contract specs 1 REST call per Contract 2025-06-04 04:37:02 +00:00
Adeel Khan 598078ce06 ADD: GATEIO md_provider with default contract specs 2025-06-04 04:37:02 +00:00
John Colvin d5ea5a9778 Add gateio_dev config profile for Adeel 2025-06-04 04:37:02 +00:00
John Colvin c3cf785f10 cleanup: Remove unused spark.conf file
Originally, this spark.conf was hardcoded into compose.yaml, and copied
into the sparkd container on each deploy. It hasn't been used since
config profiles were introduced.
2025-06-04 04:37:02 +00:00
John Colvin 569e0ed974 Add py_dummy_md config profile to test py md plugin 2025-06-04 04:37:02 +00:00
John Colvin c9eaa8af5d deploy: Move configurations to new stutter dir "app_sparkd" 2025-06-04 04:37:02 +00:00
John Colvin 679ea671c9 Add protos for gateio 2025-06-04 04:37:02 +00:00
John Colvin a245bc8610 Add protos for bybit 2025-06-04 04:37:02 +00:00
John Colvin c2362e2cd6 build: Move onix fix adapter to its own lib
We should keep order execution dependencies out of lib_spark.
2025-06-04 04:37:02 +00:00
John Colvin ddf99576af fix build: Move OnixsUtils to from lib_spark to app_sparkd
They're only needed MDP3Main.cpp and ICEMain.cpp. We're one step closer
to removing lib_spark's dependency on onixs.
2025-06-04 04:37:02 +00:00
John Colvin b8ba48c905 cleanup: Remove unnecessary onixs include from TrafixOrderStateHandler.h 2025-06-04 04:37:02 +00:00
John Colvin 0a3b8ca793 feat: Support python md plugins 2025-06-04 04:37:02 +00:00
John Colvin 82b8224416 Add "clear" fn to Book & BookSide
Prelude to python market data bindings
2025-06-04 04:37:02 +00:00
John Colvin 1849d875b1 Add "smart" parse fns for a couple proto enums
Prelude to python market data bindings. Unlike the normal parse fns, the
"smart" variants aren't case sensitive, and can match strings with or
without TYPE_NAME_ prefixes.
2025-06-04 04:37:02 +00:00
John Colvin c03f0fcf0e build: Move okx execution to its own lib 2025-06-04 04:37:02 +00:00
John Colvin 66718ed77e cleanup: Remove unnecessary onixs include from IDGenerator.h 2025-06-04 04:37:02 +00:00
John Colvin 6952821753 cleanup: Remove unnecessary includes from MDP3Listener.h 2025-06-04 04:37:01 +00:00
John Colvin ecbad48ea4 refactor: Organize libspark proj w/ stuttering include
This new structure provides several benefits:

* Allows us to install multiple CMake projects in the same destination
  dir (without include path conflicts).
  - This makes shipping the trading algo sdk a trivial excercise. Simply
    copy the lib and include dirs from the install dest. That's it!
* Puts us in a position to break up the monolithic libspark into smaller
  modules, which are easier to manage.

Windows specific notes:

* Get rid of unnecessary script vs_filters.cmake
* Define WebsocketConnectionGateway directly in cs (instead of proto)
  - The proto would no longer be in the include path passed to protoc.
    Moving it to c# because there wasn't a compelling reason for it to
    be a proto in the first place.
* Set spark output prefix to "lib" on windows
  - So we get "libspark.dll" instead of "spark.dll", which conflicts
    with the csharp lib, "Spark.dll".
* Copy libspark.dll to net8.0 & net8.0-windows in a single cmake build
  target

fix build: Copy libspark to both dirs net8.0, net8.0-windows
2025-06-04 04:37:01 +00:00
John Colvin ff6ed83e80 cleanup: Move TradeSummarySignalDispatcher to libspark
It is needed by TradeSummarySignal, which is in libspark. This prevents a
circular dependency from libspark -> sparkd.
2025-06-04 04:37:01 +00:00
John Colvin 116e4d3800 build: Clean up protobuf cmake scripts
Make them more flexible, reusable. This is a precursor to porting over
python md provider code from another branch.
2025-06-04 04:37:01 +00:00
John Colvin a6b6c1168d build: Rename external path vars w/ "EXTERNAL_" prefix in cmake
The old "SPARK_" prefix felt like a misnomer.
2025-06-04 04:37:01 +00:00
John Colvin 41be74d690 Add a few arithmetic ops to TimeSpan 2025-06-04 04:37:01 +00:00
John Colvin 661977a3ba diag: Log time it takes to start md sources
So we can detect bottlenecks
2025-06-04 04:37:01 +00:00
John Colvin ffc7dc132e refactor: Split MarketDataGateway up into providers/sources
MarketDataGateway still exists, but is greatly simplified. It creates
several `IMarketDataProvider`s, each of which generates a list of
`IMarketDataSource`s. The market data sources provide market data to an
`IMarketDataListener`.

* EmumbaMarketDataProvider
* CCApiMarketDataProvider
* PyMarketDataProvider
* OnixsMarketDataProvider
* ReplayMarketDataProvider

As a side effect, individual py market data sources are now explicitly
limited to a single venue (it probably should have been this way all
along, since the shared memory writers do not expect updates to involve
multiple venues).
2025-06-04 04:37:01 +00:00
John Colvin 248dc2e5b6 cleanup: Remove dependency on quickfix for string to_lower
We shouldn't depend on external libs for trivial things like converting
strings to lower case.
2025-06-04 04:36:53 +00:00
130 changed files with 3146 additions and 5940 deletions

No files matched your search

-1
View File
@@ -44,6 +44,5 @@ SPARKD__CONFIG_PORT=8000
SPARKD__QUICKBUS_PORT=8001
SPARKD__SIM_FIX_PORT=8002
SPARKD__SHM_SIZE=4gb
SPARKD__PYTHON_REQUIREMENTS=requirements.txt
# TODO: add config for other services (pubsub, md-publisher, webapp, api-server, etc.)
+17 -17
View File
@@ -363,7 +363,7 @@
"${workspaceFolder}/src/DataStreamSuite/DataStreamQueryAnalyzer/Configurations/DataStreamQueryAnalyzer.json",
],
"options": {
"cwd": "${workspaceFolder}/build/${input:buildConfiguration}/bin/net8.0-windows"
"cwd": "${workspaceFolder}/build/${input:buildConfiguration}/bin/net7.0-windows"
},
},
"icon": {
@@ -383,7 +383,7 @@
"${workspaceFolder}/src/DataStreamSuite/DataStreamQueryAnalyzer/Configurations/DataStreamQueryAnalyzer.json",
],
"options": {
"cwd": "${workspaceFolder}/build/Debug/bin/net8.0-windows"
"cwd": "${workspaceFolder}/build/Debug/bin/net7.0-windows"
},
},
"icon": {
@@ -403,7 +403,7 @@
"${workspaceFolder}/src/DataStreamSuite/DataStreamQueryAnalyzer/Configurations/DataStreamQueryAnalyzer.json",
],
"options": {
"cwd": "${workspaceFolder}/build/Release/bin/net8.0-windows"
"cwd": "${workspaceFolder}/build/Release/bin/net7.0-windows"
},
},
"icon": {
@@ -427,7 +427,7 @@
"${workspaceFolder}/src/DataStreamSuite/DataStreamPublisher/Configurations/${input:dataStreamPublisherConfig}.json",
],
"options": {
"cwd": "${workspaceFolder}/build/${input:buildConfiguration}/bin/net8.0"
"cwd": "${workspaceFolder}/build/${input:buildConfiguration}/bin/net7.0"
},
"icon": {
"id": "run",
@@ -450,7 +450,7 @@
"${workspaceFolder}/src/DataStreamSuite/DataStreamPublisher/Configurations/${input:dataStreamPublisherConfig}.json",
],
"options": {
"cwd": "${workspaceFolder}/build/Debug/bin/net8.0"
"cwd": "${workspaceFolder}/build/Debug/bin/net7.0"
},
"icon": {
"id": "run",
@@ -473,7 +473,7 @@
"${workspaceFolder}/src/DataStreamSuite/DataStreamPublisher/Configurations/${input:dataStreamPublisherConfig}.json",
],
"options": {
"cwd": "${workspaceFolder}/build/Release/bin/net8.0"
"cwd": "${workspaceFolder}/build/Release/bin/net7.0"
},
"icon": {
"id": "run",
@@ -496,7 +496,7 @@
"${workspaceFolder}/src/DataStreamSuite/PubSubServer/Configurations/${input:pubSubServerConfig}.json",
],
"options": {
"cwd": "${workspaceFolder}/build/${input:buildConfiguration}/bin/net8.0"
"cwd": "${workspaceFolder}/build/${input:buildConfiguration}/bin/net7.0"
},
"icon": {
"id": "run",
@@ -519,7 +519,7 @@
"${workspaceFolder}/src/DataStreamSuite/PubSubServer/Configurations/${input:pubSubServerConfig}.json",
],
"options": {
"cwd": "${workspaceFolder}/build/Debug/bin/net8.0"
"cwd": "${workspaceFolder}/build/Debug/bin/net7.0"
},
"icon": {
"id": "run",
@@ -541,7 +541,7 @@
"${workspaceFolder}/src/DataStreamSuite/PubSubServer/Configurations/${input:pubSubServerConfig}.json",
],
"options": {
"cwd": "${workspaceFolder}/build/Debug/bin/net8.0"
"cwd": "${workspaceFolder}/build/Debug/bin/net7.0"
},
"icon": {
"id": "run",
@@ -564,7 +564,7 @@
"${workspaceFolder}/src/DataStreamSuite/PubSubServer/Configurations/${input:pubSubServerConfig}.json",
],
"options": {
"cwd": "${workspaceFolder}/build/Release/bin/net8.0"
"cwd": "${workspaceFolder}/build/Release/bin/net7.0"
},
"icon": {
"id": "run",
@@ -586,7 +586,7 @@
"${workspaceFolder}/src/DataStreamSuite/PubSubServer/Configurations/${input:pubSubServerConfig}.json",
],
"options": {
"cwd": "${workspaceFolder}/build/Release/bin/net8.0"
"cwd": "${workspaceFolder}/build/Release/bin/net7.0"
},
"icon": {
"id": "run",
@@ -601,7 +601,7 @@
"windows": {
"command": "Spark.exe",
"options": {
"cwd": "${workspaceFolder}/build/${input:buildConfiguration}/bin/net8.0-windows"
"cwd": "${workspaceFolder}/build/${input:buildConfiguration}/bin/net7.0-windows"
},
},
"icon": {
@@ -617,7 +617,7 @@
"windows": {
"command": "Spark.exe",
"options": {
"cwd": "${workspaceFolder}/build/Debug/bin/net8.0-windows"
"cwd": "${workspaceFolder}/build/Debug/bin/net7.0-windows"
},
},
"icon": {
@@ -633,7 +633,7 @@
"windows": {
"command": "Spark.exe",
"options": {
"cwd": "${workspaceFolder}/build/Release/bin/net8.0-windows"
"cwd": "${workspaceFolder}/build/Release/bin/net7.0-windows"
},
},
"icon": {
@@ -648,7 +648,7 @@
"windows": {
"command": "Spark.exe",
"options": {
"cwd": "${workspaceFolder}/build/${input:buildConfiguration}/bin/net8.0-windows"
"cwd": "${workspaceFolder}/build/${input:buildConfiguration}/bin/net7.0-windows"
},
},
"icon": {
@@ -673,7 +673,7 @@
"${input:sparkConfigQuery}",
],
"options": {
"cwd": "${workspaceFolder}/build/${input:buildConfiguration}/bin/net8.0"
"cwd": "${workspaceFolder}/build/${input:buildConfiguration}/bin/net7.0"
},
"icon": {
"id": "run",
@@ -754,7 +754,7 @@
"${input:sparkTest}"
],
"options": {
"cwd": "${workspaceFolder}/build/Debug/bin/net8.0",
"cwd": "${workspaceFolder}/build/Debug/bin/net7.0",
}
},
},
+14 -2
View File
@@ -8,7 +8,6 @@ services:
- IMAGE=${SPARKD__IMAGE}
- SOURCE_DIR=${SPARKD__SOURCE_DIR}
- CONFIG_DIR=${SPARKD__CONFIG_DIR}
- PYTHON_REQUIREMENTS=${SPARKD__PYTHON_REQUIREMENTS}
shm_size: "${SPARKD__SHM_SIZE:-512mb}"
command: >
sh -c "
@@ -16,6 +15,13 @@ services:
if [ -e "/home/bts/sparkdata/data/logging.sparkd/log" ]; then
mv "/home/bts/sparkdata/data/logging.sparkd/log" "/home/bts/sparkdata/data/logging.sparkd/log_$(date '+%Y%m%d-%H%M')"
fi
if [ -e "/home/bts/sparkdata/conf/requirements.txt" ] && [ ! -d "/home/bts/sparkdata/conf/.venv" ]; then
echo 'Creating virtual environment and installing requirements'
python3 -m venv /home/bts/sparkdata/conf/.venv
source /home/bts/sparkdata/conf/.venv/bin/activate
pip install --upgrade pip
pip install -r /home/bts/sparkdata/conf/requirements.txt
fi
if [ -e "/home/bts/sparkdata/conf/.env.local" ]; then
echo 'Sourcing /home/bts/sparkdata/conf/.env.local'
set -a
@@ -45,7 +51,6 @@ services:
- IMAGE=${SPARKD__IMAGE}
- SOURCE_DIR=${SPARKD__SOURCE_DIR}
- CONFIG_DIR=${SPARKD__CONFIG_DIR}
- PYTHON_REQUIREMENTS=${SPARKD__PYTHON_REQUIREMENTS}
shm_size: "${SPARKD__SHM_SIZE:-512mb}"
command: >
sh -c "
@@ -55,6 +60,13 @@ services:
fi
ls -alh /home/bts/sparkdata/conf/
cat /home/bts/sparkdata/conf/.env.local
if [ -e "/home/bts/sparkdata/conf/requirements.txt" ] && [ ! -d "/home/bts/sparkdata/conf/.venv" ]; then
echo 'Creating virtual environment and installing requirements'
python3 -m venv /home/bts/sparkdata/conf/.venv
source /home/bts/sparkdata/conf/.venv/bin/activate
pip install --upgrade pip
pip install -r /home/bts/sparkdata/conf/requirements.txt
fi
if [ -e "/home/bts/sparkdata/conf/.env.local" ]; then
echo 'Sourcing /home/bts/sparkdata/conf/.env.local'
set -a
@@ -1,49 +0,0 @@
using System;
namespace BCG {
[DataStreamUserRespresentation("count")]
public class CountStream : ViewStream {
private DataPacket myLastFiredDataPacket;
private IBlueTableSchema mySchema = new BlueTableSchema("count",
new ColumnInfo("id", 0, typeof(int)),
new ColumnInfo("count", 1, typeof(int)));
private int myId = 0;
private ViewStream myVS;
public CountStream(ViewStream vs) {
myVS = vs;
}
public CountStream(string tableName)
: this(new TableStream(tableName)) { }
protected override void myOnFirstSubscriber() {
myVS.onChange += new ChangeDelegate(myHandleChanges);
}
protected override void myOnLastSubscriber() {
myVS.onChange -= new ChangeDelegate(myHandleChanges);
}
private void myHandleChanges(object newValue) {
myFireEvent(
myLastFiredDataPacket =
myLastFiredDataPacket == null ?
new DataPacket(myGetRows(newValue, ++myId), myGetRows(newValue, myId),
new BlueTable(mySchema))
:
new DataPacket(myGetRows(newValue, ++myId), myGetRows(newValue, myId),
myLastFiredDataPacket.newRows)
);
}
private IBlueRowCollection myGetRows(object dp, int id) {
BlueTable retVal = new BlueTable(mySchema);
retVal.addRow
(new BlueRow(mySchema, id, (dp as DataPacket).newRows.Count));
return retVal;
}
}
}
@@ -1,38 +0,0 @@
namespace bts.BlueOrb {
using bts;
using BCG;
using System;
[DataStreamUserRespresentation("DateTimeExpanderStream")]
public class DateTimeExpanstionStream : ViewStream {
public DateTimeExpanstionStream(ViewStream parent_stream, string inp_col) {
Func<IBlueTableSchema, IBlueTableSchema> get_schema = (s) => {
IBlueTableSchema rv = s.add_column(System.TypeCode.DateTime, $"{inp_col}_date");
rv = rv.add_column(System.TypeCode.String, $"{inp_col}_day_of_week");
rv = rv.add_column(System.TypeCode.Int32, $"{inp_col}_day_of_month");
rv = rv.add_column(System.TypeCode.Int32, $"{inp_col}_month");
rv = rv.add_column(System.TypeCode.Int32, $"{inp_col}_hour");
rv = rv.add_column(System.TypeCode.Int32, $"{inp_col}_minute");
rv = rv.add_column(System.TypeCode.Int32, $"{inp_col}_seconds_of_day");
return rv;
};
ViewStream rs = new MapStream(parent_stream,
s => get_schema(s),
r => {
DateTime dt = r.get_datetime(inp_col);
IBlueTableSchema s = get_schema(r.schema);
MutableBlueRow rv = new MutableBlueRow(s, r);
rv.setValue($"{inp_col}_date", dt.Date);
rv.setValue($"{inp_col}_day_of_week", dt.DayOfWeek.ToString());
rv.setValue($"{inp_col}_day_of_month", dt.Day);
rv.setValue($"{inp_col}_month", dt.Date.Month);
rv.setValue($"{inp_col}_hour", dt.Hour);
rv.setValue($"{inp_col}_minute", dt.Minute);
rv.setValue($"{inp_col}_seconds_of_day", (dt - dt.Date).TotalSeconds);
return rv;
});
mySetRealSource(rs);
}
}
}
@@ -1,158 +0,0 @@
namespace bts.BlueOrb {
using BCG;
using System;
using System.Text.RegularExpressions;
public class RegExpFilter : IStreamFilter {
private Regex[] myRegExps;
private string[] myCols;
public RegExpFilter(string col, string regExp)
: this(new string[] { col }, new string[] { regExp }) { }
public RegExpFilter(string[] cols, string[] regExps) {
myCols = cols;
myRegExps = new Regex[regExps.Length];
for (int i = 0; i < regExps.Length; i++)
myRegExps[i] = new Regex(regExps[i]);
}
#region IStreamFilter Members
public bool filter(IBlueRow dr) {
bool rv = true;
for (int i = 0; i < myCols.Length; i++) {
rv &= myRegExps[i].IsMatch(dr[myCols[i]] is DBNull ? "" : dr[myCols[i]].ToString());
if (!rv)
return rv;
}
return rv;
}
#endregion
}
//[DataStreamUserReprsentation("FilterStream")]
public class FilterStream : ViewStream {
private BlueTable myLastFiredRows;
private IBlueRowCollection myLastReceivedRows;
private IStreamFilter myFilter;
private ViewStream myParentStream;
private string myFilterString;
private bool myUseNewFilter = false;// Configuration.get<DataStreamConfig>().UseNewFilter;
#region constructors
public delegate bool OnFilter(IBlueRow r);
private class OnFilterWrapper : IStreamFilter {
public OnFilterWrapper(OnFilter f) { this.f = f; }
public bool filter(IBlueRow dr) { return f(dr); }
private OnFilter f;
}
public FilterStream(ViewStream vs, OnFilter f)
: this(vs, new OnFilterWrapper(f)) { }
public FilterStream(ViewStream vs, IStreamFilter filter) {
{
myParentStream = vs;
myFilter = filter;
myFilterString = filter.ToString();
if (filter is IDynamicStreamFilter)
(filter as IDynamicStreamFilter).onFilterChange += new FilterChangeDelegate(myHandleFilterChange);
}
}
public FilterStream(ViewStream vs, string filterExpression)
: this(vs, new BooleanExpressionFilter(filterExpression)) {
myFilterString = filterExpression;
}
public FilterStream(ViewStream vs, string col, string regExp)
: this(vs, new RegExpFilter(col, regExp)) {
myFilterString = col + " , " + regExp;
}
public FilterStream(ViewStream vs)
: this(vs, new BinaryFilter(true)) {
myFilterString = true.ToString();
}
#endregion
#region overridden methods
protected override void myOnFirstSubscriber() { myParentStream.onChange += new ChangeDelegate(myHandleCallbacks); }
protected override void myOnLastSubscriber() {
myLastFiredRows = null;
myParentStream.onChange -= new ChangeDelegate(myHandleCallbacks);
}
protected override void myFireEvent(object o) {
DataPacket dp = o as DataPacket;
if (myLastValue == null || dp.addedRows.Count + dp.deletedRows.Count > 0)
base.myFireEvent(o);
}
#endregion
private void myHandleCallbacks(object newValue) {
DataPacket retVal;
DataPacket arrivedPacket = newValue as DataPacket;
myLastReceivedRows = arrivedPacket.newRows;
if (myLastFiredRows != null) {
retVal = new DataPacket(myLastFiredRows, new BlueTable(myLastFiredRows.schema),
new BlueTable(myLastFiredRows.schema));
foreach (IBlueRow r in arrivedPacket.deletedRows)
if (myLastFiredRows.containsID((int)r["id"])) {
myLastFiredRows.deleteRow(r);
(retVal.deletedRows as BlueTable).addRow(r);
}
foreach (IBlueRow r in arrivedPacket.addedRows)
if (myFilter.filter(r)) {
myLastFiredRows.addRow(r);
(retVal.addedRows as BlueTable).addRow(r);
}
} else {
myLastFiredRows = new BlueTable(arrivedPacket.schema);
retVal = new DataPacket(myLastFiredRows, myLastFiredRows, new BlueTable(myLastFiredRows.schema));
foreach (IBlueRow r in arrivedPacket.newRows)
if (myFilter.filter(r))
myLastFiredRows.addRow(r);
}
myFireEvent(retVal);
}
public IStreamFilter filter {
set {
{
myFilter = value;
if (numberOfSubscribers > 0) {
if (myLastReceivedRows != null) {
BlueTable nRows = new BlueTable(myLastReceivedRows.schema);
foreach (IBlueRow r in myLastReceivedRows)
if (myFilter.filter(r))
nRows.addRow(r);
IBlueRowCollection dRows = new BlueTable(nRows.schema),
aRows = new BlueTable(nRows.schema);
BlueDataUtility.computeDifferences(nRows, myLastFiredRows, out dRows, out aRows);
myLastFiredRows = nRows;
myFireEvent(new DataPacket(nRows, aRows, dRows));
}
}
}
}
get {
return myFilter;
}
}
//public string filterString { set { filter = new BooleanExpressionFilter(value); } }
public override string ToString() {
return GetType().ToString() + " " + myFilterString;
}
private void myHandleFilterChange() {
filter = myFilter;
}
}
}
@@ -1,121 +0,0 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text.Json;
using bts;
//using bts.SilverPine;
using BCG;
namespace bts.BlueOrb {
[DataStreamUserRespresentation("GroupByStream")]
public class GroupByStream : ViewStream {
public interface INonIncGroupByEvaluator {
// grouped_rows
IBlueRowCollection evaluate(IDictionary<int, List<IBlueRow>> grouped_rows);
string[] get_group_by_cols();
}
private class DelNonIncGroupByEvaluator : INonIncGroupByEvaluator {
Func<string[]> _groupby_cols_func;
Func<List<IBlueRow>, List<IBlueRow>> _evaluate_func;
public DelNonIncGroupByEvaluator(Func<string[]> groupby_cols_func,
Func<List<IBlueRow>, List<IBlueRow>> evaluate_func) {
_groupby_cols_func = groupby_cols_func;
_evaluate_func = evaluate_func;
}
public IBlueRowCollection evaluate(IDictionary<int, List<IBlueRow>> grouped_rows) {
BlueTable rv = null;
foreach (var kv in grouped_rows) {
List<IBlueRow> g_rows = kv.Value;
List<IBlueRow> out_rows = _evaluate_func.Invoke(g_rows);
if (out_rows != null && out_rows.Count > 0) {
rv = rv ?? new BlueTable(out_rows.ElementAt(0).schema);
}
rv.addRows(out_rows);
}
return rv;
}
public string[] get_group_by_cols() {
return _groupby_cols_func();
}
}
private ViewStream _main_stream;
private INonIncGroupByEvaluator _groupby_func;
private BlueRowHasher _row_hasher;
private string[] _groupby_cols;
private Stopwatch _timer = new();
private int _n_total_fires = 0, _n_expensive_fires = 0;
public GroupByStream(ViewStream main_stream, INonIncGroupByEvaluator groupby_func) {
_main_stream = main_stream;
_groupby_func = groupby_func;
_groupby_cols = _groupby_func.get_group_by_cols();
_row_hasher = new BlueRowHasher(_groupby_cols == null ? 0 : _groupby_cols.Length);
}
public GroupByStream(ViewStream main_stream, Func<string[]> groupby_cols_func,
Func<List<IBlueRow>, List<IBlueRow>> evaluate_func)
: this(main_stream, new DelNonIncGroupByEvaluator(groupby_cols_func, evaluate_func)) { }
public GroupByStream(ViewStream parent_str, string groupby_col, string standard_group_by_text)
: this(parent_str, groupby_col, standard_group_by_text, String.Empty) { }
// this is the main constructor that does everything
public GroupByStream(ViewStream parent_str, string groupby_col, string standard_group_by_ops, string optional_sort_col) {
MultiRowOpHelper std_group_by = MultiRowOpHelper.parse_ops(standard_group_by_ops);
Func<string[]> groupby_cols_func = () => {
return String.IsNullOrEmpty(groupby_col) ? null : new string[] { groupby_col };
};
Func<List<IBlueRow>, List<IBlueRow>> evaluate_func = (group_rows) => {
if (!String.IsNullOrEmpty(optional_sort_col))
group_rows.Sort((r1, r2) => BlueDataUtility.compare_rows(r2, r1, optional_sort_col));
// this si where the group by aggregation happens
IBlueRow rv = MultiRowOpHelper.group_by_compute(group_rows, std_group_by);
return new IBlueRow[] { rv }.ToList();
};
ViewStream rs = new GroupByStream(parent_str, groupby_cols_func, evaluate_func);
mySetRealSource(rs);
}
public GroupByStream(ViewStream parent_str, string standard_group_by_text) :
this(parent_str, String.Empty, standard_group_by_text) {
}
protected override void myOnFirstSubscriber() { _main_stream.onChange += _main_stream_onchange; }
protected override void myOnLastSubscriber() { _main_stream.onChange -= _main_stream_onchange; }
private void _main_stream_onchange(object nv) {
DataPacket dp = nv as DataPacket;
///grouping together
_timer.Restart();
IDictionary<int, List<IBlueRow>> grouped_rows = (_groupby_cols == null || _groupby_cols.Length == 0) ?
new Dictionary<int, List<IBlueRow>>() { { 0, dp.newRows.to_list() } } :
BlueDataUtility.group_together(dp.newRows, _groupby_cols, ref _row_hasher);
_timer.Stop();
var groupElapsedMs = _timer.ElapsedMilliseconds;
if (groupElapsedMs >= 50) {
Log.DataFlow.info($"Time to group {dp.newRows.Count} rows into {grouped_rows.Count} groups: {_timer.ElapsedMilliseconds}ms");
}
// Log.log(bts.Log.Level.Info, bts.log.Subsystem.gen, String.Format("Time to group {0} rows into {1} groups = {2}", dp.newRows.Count, grouped_rows.Count, _timer.lastIntervalTime));
/// reducing
_timer.Restart();
IBlueRowCollection total_state = _groupby_func.evaluate(grouped_rows);
_timer.Stop();
var applyElapsedMs = _timer.ElapsedMilliseconds;
_n_total_fires++;
if (applyElapsedMs >= 50) {
_n_expensive_fires++;
Log.DataFlow.info(
$"Time to evaluate {grouped_rows.Count} groups {_timer.ElapsedMilliseconds}ms, ratio {_n_expensive_fires}/{_n_total_fires}");
}
// TODO,
if (total_state != null)
myFireEvent(new DataPacket(total_state, total_state, myLastValue == null ? new BlueTable(total_state.schema) : (myLastValue as DataPacket).newRows));
}
}
}
@@ -1,17 +0,0 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using BCG;
namespace bts.BlueOrb {
public interface INonIncJoinInlineMapper
{
IBlueTableSchema get_output_schema(IBlueTableSchema schema);
IBlueRowCollection map_rows(IBlueRowCollection rows);
event MapperChangeDelegate onMapperChange;
}
}
@@ -1,152 +0,0 @@
using bts;
using bts.collections;
using mbts;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Security.Cryptography;
using BCG;
namespace bts.BlueOrb {
internal class NonIncJoinBookkeeper : IDisposable {
public Action on_new_data;
private readonly NonIncJoinSpec _join_spec;
private readonly int _join_position;
private readonly int _start_index_in_output;
private readonly BlueRowHasher _row_hasher;
private readonly bool _is_left_most;
private readonly bool _is_tag_along;
private int _n_total_fires = 0, _n_expensive_fires = 0;
private IBlueTableSchema _schema = null;
private IDictionary<int, IBlueRow> _row_map = new Dictionary<int, IBlueRow>();
private List<Pair<int, IBlueRow>> _row_list = new List<Pair<int, IBlueRow>>();
private IBlueRow _solitary_row = null;
private int[] _join_col_indices = null,
_data_col_indices = null;
public NonIncJoinBookkeeper(NonIncJoinSpec the_join_spec, int join_position, int start_index_in_output, BlueRowHasher row_hasher) {
_join_spec = the_join_spec;
_join_position = join_position;
_start_index_in_output = start_index_in_output;
_row_hasher = row_hasher;
_is_left_most = _join_position == 0;
_is_tag_along = _join_spec.join_col_names.Length == 0;
}
private bool _filled_missing_data = false;
private void _fill_missing_data(DataPacket dp) {
if (_filled_missing_data)
return;
Func<string[], bool> null_or_empty = (str_arr) => {
if (str_arr == null || str_arr.Length == 0)
return true;
foreach (string s in str_arr) {
if (!String.IsNullOrEmpty(s)) {
return false;
}
}
return true;
};
Func<string[], string[]> remove_bad_cols = (col_names) => {
HashSet<string> schema_col_map = new HashSet<string>(dp.schema.columnNames);
List<string> cols_list = new List<string>(col_names);
cols_list.RemoveWhere(c => !schema_col_map.Contains(c));
return cols_list.ToArray();
};
if (null_or_empty(_join_spec.data_col_names)) {
_join_spec.data_col_names = dp.schema.columnNames;
}
_join_spec.data_col_names = remove_bad_cols(_join_spec.data_col_names);
if (null_or_empty(_join_spec.data_col_names)) {
_join_spec.data_col_names = dp.schema.columnNames;
}
if (null_or_empty(_join_spec.projected_col_names)) {
_join_spec.projected_col_names = _join_spec.data_col_names;
}
}
public void init() {
_join_spec.stream.onChange += _on_stream_cb;
}
public void _on_stream_cb(object dp_obj) {
//Log.Gen.info($"NonIncJoinBookkeeper._on_stream_cb dp_obj: {dp_obj}");
var timer = Stopwatch.StartNew();
DataPacket dp = (DataPacket)dp_obj;
_fill_missing_data(dp);
if (_schema == null) {
_schema = dp.schema;
_join_col_indices = BlueDataUtility.get_column_indices(_schema, _join_spec.join_col_names);
_data_col_indices = BlueDataUtility.get_column_indices(_schema, _join_spec.data_col_names);
}
if (_is_left_most) {
_row_list.Clear();
foreach (IBlueRow r in dp.newRows)
_row_list.Add(new Pair<int, IBlueRow>(_row_hasher.get_hash(r, _join_col_indices), r));
} else {
if (_is_tag_along)
_solitary_row = BlueDataUtility.extractRow(dp.newRows, 0);
else {
_row_map.Clear();
foreach (IBlueRow r in dp.newRows)
_row_map[_row_hasher.get_hash(r, _join_col_indices)] = r;
}
}
timer.Stop();
_n_total_fires++;
if (timer.ElapsedMilliseconds >= 50) {
_n_expensive_fires++;
Log.Gen.info($"NonIncJoinBookkeepingData._on_stream_cb {dp.newRows.Count} rows {_join_col_indices.Length} "
+ $"key cols {_join_position} join pos time {timer.Elapsed} ratio {_n_expensive_fires}/{_n_total_fires}");
}
if (on_new_data != null)
on_new_data();
}
public bool is_ready { get { return _schema != null; } }
public void constribute_cols_to_schema(List<ColumnInfo> cis) {
for (int i = 0; i < _join_spec.data_col_names.Length; i++)
cis.Add(new ColumnInfo(_join_spec.projected_col_names[i], cis.Count, _schema[_join_spec.data_col_names[i]].colType));
}
public List<Pair<int, IBlueRow>> get_list() { return _row_list; }
public void copy_master_row_to_output_row(int row_index, object[] output_row) {
IBlueRow this_row = _row_list[row_index].second;
BlueDataUtility.copy_to_row(this_row, _data_col_indices, output_row, _start_index_in_output);
}
public void copy_matching_row_to_output_row(int row_hash, object[] output_row) {
if (_is_tag_along) {
if (_solitary_row != null)
BlueDataUtility.copy_to_row(_solitary_row, _data_col_indices, output_row, _start_index_in_output);
else
copy_dbnull_to_output_row(output_row);
} else {
IBlueRow this_row = null;
if (_row_map.TryGetValue(row_hash, out this_row))
BlueDataUtility.copy_to_row(this_row, _data_col_indices, output_row, _start_index_in_output);
else
copy_dbnull_to_output_row(output_row);
}
}
private void copy_dbnull_to_output_row(object[] output_row) {
for (int i = 0; i < _data_col_indices.Length; i++)
output_row[i + _start_index_in_output] = DBNull.Value;
}
#region IDisposable
public void Dispose() { _join_spec.stream.onChange -= _on_stream_cb; }
#endregion
}
}
@@ -1,67 +0,0 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using BCG;
using mbts;
namespace bts.BlueOrb {
// this a collection of spec, theres no joining logic in here
public class NonIncJoinSpec {
public ViewStream stream;
// join col names are the names on which the streams are joined (for equality),
// data col names are the names in the original stream
// projected col names are their new names, ie if its x:l_x, then x is in the
// data_col_names and l_x in projected cols
public string[] join_col_names, data_col_names, projected_col_names;
public NonIncJoinSpec() { }
public NonIncJoinSpec(ViewStream the_stream, string[] the_join_col_names, string[] the_data_col_names)
: this(the_stream, the_join_col_names, the_data_col_names, the_data_col_names) { }
public NonIncJoinSpec(ViewStream the_stream, string[] the_join_col_names, string[] the_data_col_names, string[] the_projected_col_names) {
stream = the_stream;
join_col_names = the_join_col_names;
data_col_names = the_data_col_names;
projected_col_names = the_projected_col_names;
}
public static NonIncJoinSpec[] from_triplet(params Triplet<ViewStream, string, string>[] join_specs) {
List<NonIncJoinSpec> rv = new List<NonIncJoinSpec>();
foreach (Triplet<ViewStream, string, string> r in join_specs) {
string[] keys = String.IsNullOrEmpty(r.second) ? new string[] { } : r.second.Split(' ');
string[] visible_cols = r.third.Split(' ');
List<string> data_col_names = new List<string>(), mapped_col_names = new List<string>();
foreach (string s in visible_cols) {
string[] toks = s.Split(':');
if (toks.Length == 1) {
data_col_names.Add(toks[0]);
mapped_col_names.Add(toks[0]);
} else if (toks.Length == 2) {
data_col_names.Add(toks[0]);
mapped_col_names.Add(toks[1]);
}
}
rv.Add(new NonIncJoinSpec(r.first, keys, data_col_names.ToArray(), mapped_col_names.ToArray()));
}
return rv.ToArray();
}
public override string ToString() {
Func<string[], string> arr_txt = (a) => {
string rv = "";
foreach (string s in a) {
rv += "{s} ";
}
rv = rv.Trim();
return rv;
};
string rv = $"stream: {stream} join_col_names: {arr_txt(join_col_names)} data_col_names: {arr_txt(data_col_names)} projected_col_names: {arr_txt(projected_col_names)}";
return rv;
}
}
}
@@ -1,165 +0,0 @@
using bts;
using mbts;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using BCG;
namespace bts.BlueOrb {
[DataStreamUserRespresentation("JoinStream")]
public class NonIncJoinStream : ViewStream, BCG.Utilities.General.IWorker {
private readonly NonIncJoinSpec[] _join_specs;
private readonly INonIncJoinInlineMapper _output_mapper;
private readonly BlueRowHasher _row_hasher;
private readonly TimeSpan _throttle_period;
private readonly ViewStreamDatabase _vsd;
private List<NonIncJoinBookkeeper> _book_keeping_data = new List<NonIncJoinBookkeeper>();
private IBlueTableSchema _output_schema = null;
private DateTime _last_fire_time = DateTime.MinValue;
private bool _is_dirty = false;
public NonIncJoinStream(ViewStream left, string left_join_cols, string left_proj,
ViewStream right, string right_join_cols, string right_proj)
: this(new Triplet<ViewStream, string, string>[]
{
new Triplet<ViewStream, string, string> (left, left_join_cols, left_proj),
new Triplet<ViewStream, string, string> (right, right_join_cols, right_proj)
}) { }
public NonIncJoinStream(params Triplet<ViewStream, string, string>[] join_specs)
: this(NonIncJoinSpec.from_triplet(join_specs), null, 0) { }
public NonIncJoinStream(ViewStreamDatabase vsd, INonIncJoinInlineMapper output_mapper, int throttles_ms,
params Triplet<ViewStream, string, string>[] join_specs)
: this(vsd, NonIncJoinSpec.from_triplet(join_specs), output_mapper, throttles_ms) {
}
public NonIncJoinStream(NonIncJoinSpec[] join_specs, INonIncJoinInlineMapper output_mapper, int throttles_ms = 0)
: this(ViewStreamDatabase.instance, join_specs, output_mapper, throttles_ms) { }
public NonIncJoinStream(ViewStreamDatabase vsd, NonIncJoinSpec[] join_specs, INonIncJoinInlineMapper output_mapper, int throttles_ms = 0) {
_join_specs = join_specs;
_output_mapper = output_mapper;
_row_hasher = new BlueRowHasher(_join_specs[0].join_col_names.Length);
_throttle_period = throttles_ms == 0 ? TimeSpan.Zero : new TimeSpan(0, 0, 0, 0, throttles_ms);
_vsd = vsd;
}
protected override void myOnFirstSubscriber() {
int data_col_no = 1;
for (int jp = 0; jp < _join_specs.Length; jp++) {
NonIncJoinBookkeeper s = new NonIncJoinBookkeeper(_join_specs[jp], jp, data_col_no, _row_hasher);
//Log.Gen.info($"NonIncJoinStream.myOnFirstSubscriber jp: {jp} spec: {_join_specs[jp]}");
data_col_no += _join_specs[jp].data_col_names.Length;
s.on_new_data += _stream_callback;
_book_keeping_data.Add(s);
}
if (_output_mapper != null)
_output_mapper.onMapperChange += _on_mapper_change;
_vsd.addWorker(this);
foreach (var bkd in _book_keeping_data)
bkd.init();
}
protected override void myOnLastSubscriber() {
foreach (NonIncJoinBookkeeper d in _book_keeping_data)
d.Dispose();
if (_output_mapper != null)
_output_mapper.onMapperChange -= _on_mapper_change;
_vsd.removeWorker(this);
}
private void _stream_callback() {
//Log.Gen.info("NonIncJoinStream._stream_callback received callback");
_is_dirty = true;
doWork();
}
private void _on_mapper_change() {
_is_dirty = true;
doWork();
}
private bool _is_ready() {
bool rv = true;
foreach (NonIncJoinBookkeeper d in _book_keeping_data)
rv &= d.is_ready;
return rv;
}
private void _construct_schema() {
if (_output_schema == null) {
List<ColumnInfo> cis = new List<ColumnInfo>(new ColumnInfo[] { new ColumnInfo("id", 0, typeof(int)) });
foreach (NonIncJoinBookkeeper d in _book_keeping_data)
d.constribute_cols_to_schema(cis);
_output_schema = new BlueTableSchema("NonIncJoinStream", cis.ToArray());
}
}
private int _n_total_fires = 0, _n_expensive_fires = 0;
private void _form_and_fire() {
//Log.Gen.info($"NonIncJoinStream._form_and_fire _is_ready(): {_is_ready()}");
if (_is_ready()) {
_construct_schema();
var timer = Stopwatch.StartNew();
IList<Pair<int, IBlueRow>> left_row_list = _book_keeping_data[0].get_list();
List<object[]> output_rows = new List<object[]>();
for (int i = 0; i < left_row_list.Count; i++) {
object[] or = new object[_output_schema.columnInfo.Length];
or[0] = i;
output_rows.Add(or);
}
for (int ji = 0; ji < _book_keeping_data.Count; ji++) {
NonIncJoinBookkeeper bk_data = _book_keeping_data[ji];
for (int i = 0; i < left_row_list.Count; i++) {
if (ji == 0)
bk_data.copy_master_row_to_output_row(i, output_rows[i]);
else
bk_data.copy_matching_row_to_output_row(left_row_list[i].first, output_rows[i]);
}
}
IBlueRowCollection dp_output_rows = _apply_map(new SequentialIDBlueTable(_output_schema, output_rows));
timer.Stop();
_n_total_fires++;
//if (timer.ElapsedMilliseconds >= 250) {
_n_expensive_fires++;
//string log_msg = $"Join and map time for {_book_keeping_data.Count} "
// + $"tables {timer.Elapsed.TotalMilliseconds} msecs, ratio {_n_expensive_fires}/{_n_total_fires}";
//Log.Gen.info(log_msg);
//}
var dp_out = BlueDataUtility.build_dp_inc(dp_output_rows, myLastValue as DataPacket, true);
myFireEvent(dp_out);
//myFireEvent(new DataPacket(dp_output_rows, dp_output_rows, myLastValue == null ? new BlueTable(_output_schema) : (myLastValue as DataPacket).newRows));
}
}
private IBlueRowCollection _apply_map(IBlueRowCollection unmapped_rows) {
if (_output_mapper != null) {
IBlueRowCollection output_rows;
var timer = Stopwatch.StartNew();
output_rows = _output_mapper.map_rows(unmapped_rows);
timer.Stop();
if (timer.ElapsedMilliseconds >= 50) {
Log.Gen.info($"Time to map {unmapped_rows.Count} rows into {output_rows.Count} rows {timer.Elapsed}");
}
return output_rows;
} else {
return unmapped_rows;
}
}
public void doWork() {
//Log.Gen.info($"NonIncJoinStream.doWork _last_fire_time: {_last_fire_time} _is_dirty: {_is_dirty}");
if (DateTime.Now - _last_fire_time >= _throttle_period && _is_dirty) {
_is_dirty = false;
_last_fire_time = DateTime.Now;
_form_and_fire();
}
}
}
}
@@ -1,28 +0,0 @@
//namespace bts.Blueorb {
// using BCG;
// public class JsonSaverStream : ViewStream {
// private ViewStream _stream;
// private string _file_name;
// public JsonSaverStream(ViewStream vs, string file_name) {
// _stream = vs;
// _file_name = file_name;
// }
// protected override void myOnFirstSubscriber() {
// _stream.onChange += _on_data;
// }
// private void _on_data(object dp_obj) {
// if (dp_obj == null) {
// return;
// }
// DataPacket dp = dp_obj as DataPacket;
// dp.newRows.write_to_json_file(dp.newRows, _file_name);
// }
// protected override void myOnLastSubscriber() {
// _stream.onChange -= _on_data;
// }
// }
//}
@@ -1,91 +0,0 @@
using System;
using System.Collections;
using System.Collections.Generic;
using Amazon.Runtime.Internal.Util;
using BCG;
using BCG.Utilities;
using bts;
namespace bts.BlueOrb{
public class LiteralStream : ViewStream {
public static List<Tuple<ColumnInfo, object>> parse_columns(string key_value_pairs) {
List<Tuple<ColumnInfo, object>> rv = new List<Tuple<ColumnInfo, object>>();
string[] pairs = key_value_pairs.Split(' ');
foreach (string p in pairs) {
string[] kvp = p.Split(':');
string col_name = kvp[0];
string val = kvp[1];
var tup = BlueDataUtility.guess_type(val);
var ci = new ColumnInfo(col_name, rv.Count, tup.Item1);
rv.Add(Tuple.Create(ci, tup.Item2));
}
return rv;
}
//literal("table1", "x:123, y:1.001, z:abc;x:123, y:1.001, z:abc")
// col_name: col_value ... ; col_name: col_value
public LiteralStream(string table_name, string rows) {
Func<string, int, IBlueRow> get_row = (json_row, id) => {
Dictionary<string, object> map = new Dictionary<string, object>();
string[] toks = json_row.Split(',', StringSplitOptions.TrimEntries);
//Logger.assert(col_vals.Length == 2);
foreach (string t in toks) {
var col_vals = t.Split(':', StringSplitOptions.TrimEntries);
string col = col_vals[0];
string val = col_vals[1];
map.Add(col, val);
}
map["id"] = id;
return new BlueRow(map);
};
string[] row_arr = rows.Split(';', StringSplitOptions.TrimEntries);
BlueTable table = null;
foreach (string json_string in row_arr) {
IBlueRow r = //BlueDataUtility.convert_json_to_row(json_string, table == null ? 0 : table.Count);
get_row(json_string, table == null ? 0 : table.Count);
table = table ?? new BlueTable(r.schema);
table.addRow(r);
}
ViewStream str = new BlueTableStream(table);
str = new SchemaGuesserStream(str);
mySetRealSource(str);
}
public LiteralStream(string tableName, string schema, string value) {
string[] columns = schema.Split(' ');
ColumnInfo[] ci = new ColumnInfo[columns.Length + 1];
ci[0] = new ColumnInfo("id", 0, typeof(int));
int n = 1;
foreach (string c in columns) {
string[] columnAndType = c.Split('/');
if (columnAndType.Length == 1)
ci[n] = new ColumnInfo(c, n, typeof(double));
else
ci[n] = new ColumnInfo(columnAndType[0], n, Type.GetType("System." + columnAndType[1], true));
n++;
}
BlueTable t = new BlueTable(new BlueTableSchema(tableName, ci));
string[] rows = value.Split(';');
n = 0;
foreach (string r in rows)
try {
if (r == string.Empty)
continue;
t.addRow(
BlueDataUtility.createRowFromCSVFileRow(r, t.schema, ++n));
} catch (Exception e) {
throw (new Exception("Error occured at line: " + n.ToString() + " " + e.Message));
}
mySetRealSource(new BlueRowCollectionStaticStream(t));
}
}
}
@@ -1,89 +0,0 @@
using System;
using BCG.Utilities.General;
using BCG;
using bts;
namespace bts.BlueOrb {
[DataStreamUserRespresentation("MillisecondStream")]
public class MillisecondStream : ViewStream, IWorker {
private DateTime _last_fire_time = DateTime.MinValue;
private readonly TimeSpan _period;
private readonly int _size;
private readonly ViewStreamDatabase _vsd;
private IBlueTableSchema _schema;
private bts.CircularBufferTable _table;
public MillisecondStream(int interval_msecs, int size)
: this(ViewStreamDatabase.instance, interval_msecs, size) {
}
public MillisecondStream(ViewStreamDatabase vsd, int interval_msecs, int size) {
_vsd = vsd;
_period = TimeSpan.FromMilliseconds(interval_msecs);
_size = size;
_schema = BlueDataUtility.form_schema("Milliseconds",
Tuple.Create("id", TypeCode.Int32),
Tuple.Create("time", TypeCode.DateTime),
Tuple.Create("display_time", TypeCode.String));
_table = new CircularBufferTable(_schema, _size);
}
protected override void myOnFirstSubscriber() {
_vsd.addWorker(this);
}
protected override void myOnLastSubscriber() {
_vsd.removeWorker(this);
}
private int _id = 0;
public void doWork() {
if (DateTime.Now - _last_fire_time >= _period) {
_last_fire_time = DateTime.Now;
IBlueRow nr = new BlueRow(_schema, _id++, _last_fire_time,
_last_fire_time.ToString("yyyy-MM-dd HH:mm:ss.fff"));
_table.add_row(nr);
var dp = BlueDataUtility.build_dp_inc(_table, myLastValue as DataPacket);
myFireEvent(dp);
}
}
}
public class MSecondStream : TimeUnitStream {
public MSecondStream(int intervalInms, int size)
: base(DateTime.Now, new TimeSpan(0, 0, 0, 0, intervalInms), size, ViewStreamDatabase.instance) { }
protected override DateTime myRoundTime(DateTime dt) { return dt; }
}
public class DisplayableMillisecondStream : ViewStream, IStreamMapper {
private IBlueTableSchema mySchema = new BlueTableSchema("ms",
new ColumnInfo("id", 0, typeof(int)),
new ColumnInfo("time", 1, typeof(string)));
public DisplayableMillisecondStream(int intervalInMs, int sz) {
mySetRealSource(new MapStream(new MSecondStream(intervalInMs, sz), this));
}
#region IStreamMapper Members
public IBlueTableSchema getOutputSchema(IBlueTableSchema inputSchema) {
return mySchema;
}
#endregion
#region IDynamicStreamMapper Members
public IBlueRow map(IBlueRow row) {
DateTime time = row.get_datetime("time");
var time_txt = time.ToString("yyyy-MM-dd HH:mm:ss.fff");
return new BlueRow(mySchema, row["id"], time_txt);
}
#endregion
private string myToString(DateTime dt) {
string sep = ":";
return dt.Hour + sep + dt.Minute + sep + dt.Second + sep + dt.Millisecond;
}
}
}
@@ -1,127 +0,0 @@
namespace bts.BlueOrb {
using BCG;
using System;
using System.Collections.Generic;
using System.Linq;
//using MathNet.Numerics.Statistics;
public enum MultiRowOp {
Mean,
Median,
Mode,
Range,
StdDev,
Max,
Min,
First,
Last,
Sum,
EdgeDiff,
EdgeDiffPercentage,
//IsEdgeDiff,
}
public class MultiRowOpHelper {
public readonly List<Tuple<string, MultiRowOp>> operations;
public static MultiRowOpHelper parse_ops(string text) {
// text is inp_col: op, inp_col: op, inp_col:op
List<Tuple<string, MultiRowOp>> rv = new List<Tuple<string, MultiRowOp>>();
string[] calcs = text.Split(',');
foreach (string c in calcs) {
string[] toks = c.Split(':');
string inp_col = toks[0].Trim();
string op_txt = toks[1].Trim();
MultiRowOp op = Enum.Parse<MultiRowOp>(op_txt, true);
rv.Add(Tuple.Create(inp_col, op));
}
return new MultiRowOpHelper(rv);
}
public MultiRowOpHelper(List<Tuple<string, MultiRowOp>> operations_) {
operations = operations_;
}
// aggregates multiple rows into one - ie does group_by compute
public static IBlueRow group_by_compute(List<IBlueRow> rows, MultiRowOpHelper parameters) {
if (rows.Count() == 0) {
return null;
}
IBlueRow ir = rows.ElementAt(0);
IBlueTableSchema in_schema = ir.schema;
IBlueTableSchema out_schema = in_schema;
IDictionary<string, Tuple<string, MultiRowOp>> out_col_map =
new Dictionary<string, Tuple<string, MultiRowOp>>();
foreach (var t in parameters.operations) {
string out_col_name = $"{t.Item1}_{t.Item2}".ToLower();
out_schema = out_schema.add_column(TypeCode.Double, out_col_name);
out_col_map[out_col_name] = t;
}
MutableBlueRow or = new MutableBlueRow(out_schema, ir);
foreach (var kv in out_col_map) {
string out_col_name = kv.Key;
string in_col_name = kv.Value.Item1;
MultiRowOp op = kv.Value.Item2;
IEnumerable<double> col_vals = BlueDataUtility.get_colvalues<double>(rows, in_col_name);
double comp_res = compute_op(col_vals, op);
or.setValue(out_col_name, comp_res);
}
return or;
}
// rolling_rows are the rolling window of rows. the 0th row is the one thats gettin
// calculated. rows 1 - n-1 are its window rows. the schema is based on the operations.
public IBlueRow rolling_calc(List<IBlueRow> rolling_rows) {
if (rolling_rows.Count() == 0) {
return null;
}
IBlueRow ir = rolling_rows.ElementAt(0);
IBlueTableSchema in_schema = ir.schema;
IBlueTableSchema out_schema = in_schema;
IDictionary<string, Tuple<string, MultiRowOp>> out_col_map =
new Dictionary<string, Tuple<string, MultiRowOp>>();
foreach (var t in operations) {
string out_col_name = $"{t.Item1}_{t.Item2}".ToLower();
out_schema = out_schema.add_column(TypeCode.Double, out_col_name);
out_col_map[out_col_name] = t;
}
MutableBlueRow or = new MutableBlueRow(out_schema, ir);
foreach (var kv in out_col_map) {
string out_col_name = kv.Key;
string in_col_name = kv.Value.Item1;
MultiRowOp op = kv.Value.Item2;
IEnumerable<double> col_vals = BlueDataUtility.get_colvalues<double>(rolling_rows, in_col_name);
double comp_res = compute_op(col_vals, op);
or.setValue(out_col_name, comp_res);
}
return or;
}
public static double compute_op(IEnumerable<double> col_vals, MultiRowOp op) {
switch (op) {
case MultiRowOp.Max:
return col_vals.Max<double>();
case MultiRowOp.Min:
return col_vals.Min<double>();
case MultiRowOp.Mean:
return col_vals.Sum() / col_vals.Count();
case MultiRowOp.Sum:
return col_vals.Sum();
case MultiRowOp.First:
return col_vals.First();
case MultiRowOp.Last:
return col_vals.Last();
case MultiRowOp.StdDev:
return MathNet.Numerics.Statistics.Statistics.StandardDeviation(col_vals);
default:
// TODO
Log.assert(false);
return Double.NaN;
}
}
}
}
@@ -1,207 +0,0 @@
namespace bts.BlueOrb {
using BCG;
using ProtoBuf;
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
#region mappings
internal class ProjectData {
// schema from the base stream
public readonly IBlueTableSchema in_schema;
// this is the projected schema
public readonly IBlueTableSchema out_schema;
// projected cols to the input cols, for cols who's names are changed
public readonly IDictionary<string, string> mapped_to_orig_col_map = new Dictionary<string, string>();
public ProjectData(IBlueTableSchema in_schema_, IBlueTableSchema out_schema_,
IDictionary<string, string> mapped_to_orig_col_map_) {
in_schema = in_schema_;
out_schema = out_schema_;
mapped_to_orig_col_map = mapped_to_orig_col_map_;
}
}
#endregion
#region Row with mapipings build in
internal class NewProjectRow : IBlueRow {
#region IBlueRow members
private ProjectData _project_data;
private IBlueRow _row;
public NewProjectRow(ProjectData project_data, IBlueRow row) {
_project_data = project_data;
_row = row;
}
public IBlueTableSchema schema { get { return _project_data.out_schema; } }
#endregion
#region ICollection members
public int Count { get { return schema.columnNames.Length; } }
public bool IsSynchronized { get { return false; } }
public object SyncRoot { get { return this; } }
public void CopyTo(Array array, int index) {
int i = index;
foreach (object o in this as IEnumerable)
array.SetValue(o, i++);
}
#endregion
#region IDictionary members
public bool IsFixedSize { get { return true; } }
public bool IsReadOnly { get { return true; } }
public object this[object key] {
get {
if (key is string)
return this[(string)key];
return this[(int)key];
}
set { throw new NotSupportedException("The IBlueRow is readonly"); }
}
public object this[int index] {
get {
return this[schema.columnInfo[index].colName];
}
}
public object this[string col_name] {
get {
string mapped_to_col = null;
if (_project_data.mapped_to_orig_col_map.TryGetValue(col_name, out mapped_to_col)) {
return _row[mapped_to_col];
}
return _row[col_name];
}
}
public ICollection Keys { get { return schema.columnNames; } }
public ICollection Values {
get {
object[] retVal = new object[schema.columnNames.Length];
CopyTo(retVal, 0);
return retVal;
}
}
public void Add(object key, object value) { throw new NotSupportedException("The IBlueRow is readonly"); }
public void Clear() { throw new NotSupportedException("The IBlueRow is readonly"); }
public bool Contains(object key) { return schema[(string)key] != null; }
/// <summary>
/// Note, the default enumerator is the IDictionary's enumerator, so if user needs to
/// iterate through the values, typecast this into an IEnumerable
/// </summary>
/// <returns></returns>
public IDictionaryEnumerator GetEnumerator() { return new BlueRowDictionaryEnumerator(this); }
IEnumerator IEnumerable.GetEnumerator() { return new BlueRowEnumerator(this); }
public void Remove(object key) { throw new NotSupportedException("The IBlueRow is readonly"); }
#endregion
}
#endregion
[DataStreamUserRespresentation("ProjectStream")]
public class NewProjectStream : ViewStream, IStreamMapper {
private ViewStream _stream;
private string _project_expression;
private ProjectData _project_data = null;
//public ProjectData
public NewProjectStream(ViewStream vs, string project_expression) {
_stream = vs;
_project_expression = project_expression;
_stream = new MapStream(vs, this);
mySetRealSource(_stream);
}
public IBlueTableSchema getOutputSchema(IBlueTableSchema input_schema) {
_project_data = _create_project_data(input_schema, _project_expression);
return _project_data.out_schema;
}
public IBlueRow map(IBlueRow row) {
return new NewProjectRow(_project_data, row);
}
private static ProjectData _create_project_data(IBlueTableSchema input_schema,
string project_expression) {
project_expression = project_expression.Trim();
string[] project_toks = project_expression.Split(' ');
HashSet<string> removed_cols = new HashSet<string>();
Dictionary<string, string> mapped_to_orig_map = new Dictionary<string, string>();
Dictionary<string, string> orig_to_mapped = new Dictionary<string, string>();
// by default every col will show up, so if we have the projection as (a:a1)
// it will convert a to a1,
// examples: if a schema has cols a b c
// project(parent_str, "a:aa ") -> aa b c
// project(parent_str, "a:aa !c") -> aa b
// project(parent_str, "a:aa !") -> aa
// project(parent_str, "a:aa b")) -> aa b
bool hide_all = false;
foreach (string p_tok in project_toks) {
if (p_tok.StartsWith("!")) {
if (p_tok.Length == 1) {
// if it is a "blanket "!"=> hide all cols except the one that
// are stated explicitly
hide_all = true;
} else {
string removed_col = p_tok.Substring(1);
removed_cols.Add(removed_col);
}
} else if (p_tok.Contains(":")) {
string[] map_toks = p_tok.Split(":");
string orig_col = map_toks[0];
string mapped_col = map_toks[1];
mapped_to_orig_map.Add(mapped_col, orig_col);
orig_to_mapped.Add(orig_col, mapped_col);
} else {
if (input_schema.columnNames.Contains(p_tok)) {
string orig_col = p_tok;
string mapped_col = orig_col;
orig_to_mapped.Add(orig_col, mapped_col);
}
}
}
// make the new schema
List<ColumnInfo> new_cols = new List<ColumnInfo>();
foreach (ColumnInfo ci in input_schema.columnInfo) {
if (ci.colName == "id") {
ColumnInfo nc = new ColumnInfo(ci.colName, new_cols.Count, ci.colType);
new_cols.Add(nc);
continue;
}
if (removed_cols.Contains(ci.colName)) {
continue;
} else if (orig_to_mapped.ContainsKey(ci.colName)) {
string mapped_name = orig_to_mapped[ci.colName];
ColumnInfo nc = new ColumnInfo(mapped_name, new_cols.Count, ci.colType);
new_cols.Add(nc);
} else {
if (hide_all)
continue;
//if (ci.colName == "id" || removed_cols.Count > 0) {
// unless columns are explicitely hidden, all columns show up
ColumnInfo nc = new ColumnInfo(ci.colName, new_cols.Count, ci.colType);
new_cols.Add(nc);
//}
}
}
IBlueTableSchema new_schema = new BlueTableSchema(input_schema.tableName, new_cols.ToArray());
return new ProjectData(input_schema, new_schema, mapped_to_orig_map);
}
}
}
@@ -1,30 +0,0 @@
using BCG;
namespace bts.BlueOrb {
public class SchemaGuesserStream : ViewStream {
private IBlueRowCollection _rows = null;
private ViewStream _vs;
public SchemaGuesserStream(ViewStream vs) {
_vs = vs;
}
protected override void myOnFirstSubscriber() {
_vs.onChange += _on_data;
}
protected override void myOnLastSubscriber() {
_vs.onChange -= _on_data;
}
private void _on_data(object obj) {
DataPacket dp = obj as DataPacket;
if (_rows == null) {
_rows = BlueDataUtility.get_typed_table(dp.newRows);
}
var dp_out = BlueDataUtility.build_dp_inc(_rows, myLastValue as DataPacket);
myFireEvent(dp_out);
}
}
}
@@ -1,67 +0,0 @@
using Bts.ArithmeticExpressions;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using BCG;
namespace bts.BlueOrb {
[DataStreamUserRespresentation("SingleRowMathStream")]
public class MathStream : ViewStream, IStreamMapper {
private IArithmeticProgram myExp;
private BooleanExpressionFilter _bool;
List<string> myExtraColumns = new List<string>();
BlueTableSchema mySchema;
public MathStream(ViewStream vs, string mathExpression)
: this(vs, mathExpression, string.Empty) {
}
public MathStream(ViewStream vs, string mathExpression, string boolean_expression) {
myExp = ArithmeticProgramParser.Parse(mathExpression) ??
throw new Exception($"Failed to parse ArithmeticExpression '{mathExpression}'");
myExtraColumns = myExp.OutputIdentifiers.ToList();
if (!string.IsNullOrEmpty(boolean_expression))
_bool = new BooleanExpressionFilter(boolean_expression);
mySetRealSource(new MapStream(vs, this));
}
#region IStreamMapper Members
public IBlueTableSchema getOutputSchema(IBlueTableSchema inputSchema) {
if (mySchema == null) {
Dictionary<string, ColumnInfo> colIdx = new Dictionary<string, ColumnInfo>();
foreach (ColumnInfo ci in inputSchema.columnInfo)
colIdx[ci.colName] = ci;
foreach (string ncn in myExtraColumns) {
ColumnInfo newC;
if (colIdx.ContainsKey(ncn))
newC = new ColumnInfo(colIdx[ncn].colName, colIdx[ncn].colNo, typeof(double));
else
newC = new ColumnInfo(ncn, colIdx.Count, typeof(double));
colIdx[ncn] = newC;
}
mySchema = new BlueTableSchema(colIdx.Values.ToArray(), "math(" + inputSchema.tableName + ")");
}
return mySchema;
}
#endregion
#region IDynamicStreamMapper Members
public IBlueRow map(IBlueRow row) {
MutableBlueRow result = new MutableBlueRow(mySchema, row);
if (_bool == null || _bool.filter(row)) {
myExp.Evaluate(result.AsDictionary());
}
return result;
}
#endregion
}
}
@@ -1,11 +0,0 @@
namespace bts.BlueOrb {
using BCG;
[DataStreamUserRespresentation("TopStream")]
public class TopStream : ViewStream {
public TopStream(ViewStream vs, int n_rows, string sort_col) {
ViewStream rs = new BCG.TopStream (vs, n_rows, sort_col);
mySetRealSource(rs);
}
}
}
@@ -1,261 +0,0 @@
using System;
using System.Collections.Generic;
using mbts;
using BCG;
namespace bts.BlueOrb {
internal class MutableAppendTable : IBlueRowCollection {
private class MyRow : IBlueRow {
private IBlueRow myRow;
private int myID;
public MyRow(IBlueRow r, int id) {
myRow = r;
myID = id;
}
#region IBlueRow Members
public IBlueTableSchema schema { get { return myRow.schema; } }
public object this[int index] {
get {
if (myRow.schema[index].colName == "id")
return myID;
else
return myRow[index];
}
}
public object this[string colName] { get { return colName == "id" ? (object)myID : myRow[colName]; } }
#endregion
#region IDictionary Members
public void Add(object key, object value) { throw new Exception("Row is read only."); }
public void Clear() { throw new Exception("Row is read only."); }
public bool Contains(object key) { return myRow.Contains(key); }
public System.Collections.IDictionaryEnumerator GetEnumerator() { return new BlueRowDictionaryEnumerator(this); }
public bool IsFixedSize { get { return true; } }
public bool IsReadOnly { get { return true; } }
public System.Collections.ICollection Keys { get { return myRow.Keys; } }
public void Remove(object key) { throw new Exception("The method or operation is not implemented."); }
public System.Collections.ICollection Values {
get {
object[] vals = new object[myRow.Count];
CopyTo(vals, 0);
return vals;
}
}
public object this[object key] {
get { return key is string ? this[key as string] : this[(int)key]; }
set { throw new Exception("The method or operation is not implemented."); }
}
#endregion
#region ICollection Members
public void CopyTo(Array array, int index) {
myRow.CopyTo(array, index);
array.SetValue(myID, index + myRow.schema["id"].colNo);
}
public int Count { get { return myRow.Count; } }
public bool IsSynchronized { get { return myRow.IsSynchronized; } }
public object SyncRoot { get { return myRow.SyncRoot; } }
#endregion
#region IEnumerable Members
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() { return new BlueRowEnumerator(this); }
#endregion
}
private BlueTable myRows;
private int myID = 0;
/// <summary>
/// maps the table no to a dictionary containing the
/// local IDs to a row in the final big table
/// </summary>
private Dictionary<int, Dictionary<int, IBlueRow>> myIDMap = new Dictionary<int, Dictionary<int, IBlueRow>>();
public MutableAppendTable(IBlueTableSchema schema) { myRows = new BlueTable(schema); }
public IBlueRow addRow(IBlueRow r, int tableNo) {
if (!myIDMap.ContainsKey(tableNo))
myIDMap.Add(tableNo, new Dictionary<int, IBlueRow>());
IBlueRow nr = new MyRow(r, myID++); // BlueDataUtility.getNewReplacedRow(r, myID++, "id");
myRows.addRow(nr);
myIDMap[tableNo].Add((int)r["id"], nr);
return nr;
}
public IBlueRow removeRow(IBlueRow r, int tableNo) {
IBlueRow dr = myIDMap[tableNo][(int)r["id"]];
myRows.deleteRow(dr);
myIDMap[tableNo].Remove((int)r["id"]);
return dr;
}
public IBlueRowCollection removeTable(int tableNo) {
if (!myIDMap.ContainsKey(tableNo))
return null;
else {
BlueTable rv = new BlueTable(myRows.schema);
foreach (IBlueRow r in myIDMap[tableNo].Values)
rv.addRow(r);
myRows.deleteRows(rv);
myIDMap.Remove(tableNo);
return rv;
}
}
public void clear() {
myRows.clear();
myIDMap.Clear();
}
#region IBlueRowCollection Members
public IBlueRow this[int id] { get { return myRows[id]; } }
public IBlueTableSchema schema { get { return myRows.schema; } }
public bool containsID(int id) { return myRows.containsID(id); }
#endregion
#region ICollection Members
public void CopyTo(Array array, int index) { myRows.CopyTo(array, index); }
public int Count { get { return myRows.Count; } }
public bool IsSynchronized { get { return myRows.IsSynchronized; } }
public object SyncRoot { get { return myRows.SyncRoot; } }
#endregion
#region IEnumerable Members
public System.Collections.IEnumerator GetEnumerator() { return myRows.GetEnumerator(); }
#endregion
}
/// <summary>
/// MutableStream can append multiple streams
/// as well as give the user the flexibility
/// to append (or remove) more streams at
/// runtime
/// </summary>
}
namespace bts.BlueOrb {
[DataStreamUserRespresentation("UnionStream")]
public class MutableAppendStream : ViewStream {
private bool mySubscriptionActive = false;
private MutableAppendTable myTable;
private int myMovingIndex;
private Dictionary<int, Pair<ViewStream, ViewStream>> myStreamIndex = new Dictionary<int, Pair<ViewStream, ViewStream>>();
private bool _allign_col_orders = true;
public MutableAppendStream(params ViewStream[] streams)
: this(new List<ViewStream>(streams)) { }
public MutableAppendStream(bool allign_col_order, params ViewStream[] streams)
: this(allign_col_order, new List<ViewStream>(streams)) { }
public MutableAppendStream(ICollection<ViewStream> streams)
: this(true, streams) { }
public MutableAppendStream(bool allign_col_orders, ICollection<ViewStream> streams) {
_allign_col_orders = allign_col_orders;
foreach (ViewStream vs in streams) {
myMovingIndex++;
ViewStream pj = new ChangeTableNameStream(vs, myMovingIndex.ToString());
if (_allign_col_orders)
pj = new ProjectStream(pj, "!");
myStreamIndex.Add(myMovingIndex, new Pair<ViewStream, ViewStream>(vs, pj));
}
}
public void addStream(ViewStream vs) {
myMovingIndex++;
///the change name is done to identify the callbacks, and the project to
///make all the schemas col orders the same
ViewStream pj = new ChangeTableNameStream(vs, myMovingIndex.ToString());
if (_allign_col_orders)
pj = new ProjectStream(pj, "!");
myStreamIndex.Add(myMovingIndex, new Pair<ViewStream, ViewStream>(vs, pj));
if (mySubscriptionActive)
pj.onChange += new ChangeDelegate(myHandleStreamChanges);
}
public void removeStream(ViewStream vs) {
int index = -1;
foreach (KeyValuePair<int, Pair<ViewStream, ViewStream>> p in myStreamIndex) {
if (p.Value.first == vs) {
index = p.Key;
if (mySubscriptionActive) {
p.Value.second.onChange -= new ChangeDelegate(myHandleStreamChanges);
if (myTable != null) {
IBlueRowCollection delRows = myTable.removeTable(p.Key);
if (delRows != null)
myFireEvent(new DataPacket(myTable, new BlueTable(myTable.schema), delRows));
}
}
break;
}
if (index > 0)
myStreamIndex.Remove(index);
}
}
protected override void myOnFirstSubscriber() {
mySubscriptionActive = true;
foreach (KeyValuePair<int, Pair<ViewStream, ViewStream>> p in myStreamIndex)
p.Value.second.onChange += new ChangeDelegate(myHandleStreamChanges);
}
protected override void myOnLastSubscriber() {
mySubscriptionActive = false;
foreach (KeyValuePair<int, Pair<ViewStream, ViewStream>> p in myStreamIndex)
p.Value.second.onChange -= new ChangeDelegate(myHandleStreamChanges);
if (myTable != null)
myTable.clear();
}
private void myHandleStreamChanges(object newValue) {
DataPacket dp = newValue as DataPacket;
myTable = myTable ?? new MutableAppendTable(new BlueTableSchema("MutableAppend", dp.schema.columnInfo));
BlueTable adds = new BlueTable(myTable.schema),
dels = new BlueTable(myTable.schema);
int index = Int32.Parse(dp.schema.tableName);
foreach (IBlueRow r in dp.deletedRows) {
IBlueRow dr = myTable.removeRow(r, index);
dels.addRow(dr);
}
foreach (IBlueRow r in dp.addedRows) {
IBlueRow nr = myTable.addRow(r, index);
adds.addRow(nr);
}
myFireEvent(new DataPacket(myTable, adds, dels));
}
}
}
@@ -18,7 +18,6 @@
<ItemGroup>
<PackageReference Include="HtmlAgilityPack" Version="1.11.57" />
<PackageReference Include="IronPython" Version="3.4.2" />
<PackageReference Include="Microsoft.Data.Analysis" Version="0.21.1" />
<PackageReference Include="MongoDB.Bson" Version="2.23.1" />
<PackageReference Include="MongoDB.Driver" Version="2.23.1" />
@@ -1,47 +1,52 @@
using System;
namespace bts.BlueOrb {
using BCG;
[DataStreamUserRespresentation("CountStream")]
public class CountStream : ViewStream {
namespace BCG
{
[DataStreamUserRespresentation ("count")]
public class CountStream : ViewStream
{
private DataPacket myLastFiredDataPacket;
private IBlueTableSchema mySchema = new BlueTableSchema("count",
new ColumnInfo("id", 0, typeof(int)),
new ColumnInfo("count", 1, typeof(int)));
private IBlueTableSchema mySchema = new BlueTableSchema ("count",
new ColumnInfo ("id", 0, typeof(int)),
new ColumnInfo ("count", 1, typeof(int)));
private int myId = 0;
private ViewStream myVS;
public CountStream(ViewStream vs) {
public CountStream (ViewStream vs)
{
myVS = vs;
}
public CountStream(string tableName)
: this(new TableStream(tableName)) { }
public CountStream (string tableName)
: this (new TableStream (tableName))
{}
protected override void myOnFirstSubscriber() {
protected override void myOnFirstSubscriber () {
myVS.onChange += new ChangeDelegate(myHandleChanges);
}
protected override void myOnLastSubscriber() {
protected override void myOnLastSubscriber () {
myVS.onChange -= new ChangeDelegate(myHandleChanges);
}
private void myHandleChanges(object newValue) {
myFireEvent(
private void myHandleChanges(object newValue)
{
myFireEvent (
myLastFiredDataPacket =
myLastFiredDataPacket == null ?
new DataPacket(myGetRows(newValue, ++myId), myGetRows(newValue, myId),
new BlueTable(mySchema))
new DataPacket (myGetRows (newValue, ++myId), myGetRows (newValue, myId),
new BlueTable (mySchema))
:
new DataPacket(myGetRows(newValue, ++myId), myGetRows(newValue, myId),
new DataPacket (myGetRows (newValue, ++myId), myGetRows (newValue, myId),
myLastFiredDataPacket.newRows)
);
}
private IBlueRowCollection myGetRows(object dp, int id) {
BlueTable retVal = new BlueTable(mySchema);
private IBlueRowCollection myGetRows (object dp, int id)
{
BlueTable retVal = new BlueTable (mySchema);
retVal.addRow
(new BlueRow(mySchema, id, (dp as DataPacket).newRows.Count));
(new BlueRow (mySchema, id, (dp as DataPacket).newRows.Count));
return retVal;
}
@@ -5,20 +5,25 @@ using mbts.config;
using System;
using System.Text.RegularExpressions;
namespace BCG.StreamWorld.Addition {
[DataStreamUserRespresentation(false)]
public class FilterStream : ViewStream {
namespace BCG.StreamWorld.Addition
{
[DataStreamUserRespresentation (false)]
public class FilterStream : ViewStream
{
private ViewStream myBaseStream;
private BlueTable myBaseTable, myStateTable;
private IStreamFilter myFilter;
public FilterStream(ViewStream vs)
: this(vs, "1=1") { }
: this(vs, "1=1")
{ }
public FilterStream(ViewStream vs, string filterExp)
: this(vs, new BooleanExpressionFilter(filterExp)) { }
: this(vs, new BooleanExpressionFilter(filterExp))
{ }
public FilterStream(ViewStream vs, IStreamFilter filter) {
public FilterStream(ViewStream vs, IStreamFilter filter)
{
myBaseStream = vs;
myFilter = filter;
if (filter is IDynamicStreamFilter)
@@ -26,35 +31,43 @@ namespace BCG.StreamWorld.Addition {
}
protected override void myOnFirstSubscriber() {
protected override void myOnFirstSubscriber()
{
myBaseStream.onChange += new ChangeDelegate(myHandleChanges);
}
protected override void myOnLastSubscriber() {
protected override void myOnLastSubscriber()
{
myBaseStream.onChange -= new ChangeDelegate(myHandleChanges);
myBaseTable.clear();
myStateTable.clear();
}
private void myHandleChanges(object newValue) {
private void myHandleChanges(object newValue)
{
DataPacket dp = newValue as DataPacket;
if (myStateTable == null) {
if (myStateTable == null)
{
bts.Log.assert(myBaseTable == null);
myStateTable = new BlueTable(dp.schema);
myBaseTable = new BlueTable(dp.schema);
}
BlueTable dels = new BlueTable(dp.schema),
adds = new BlueTable(dp.schema);
foreach (IBlueRow r in dp.deletedRows) {
foreach (IBlueRow r in dp.deletedRows)
{
myBaseTable.deleteRow(r);
if (myStateTable.containsID((int)r["id"])) {
if (myStateTable.containsID((int)r["id"]))
{
dels.addRow(r);
myStateTable.deleteRow(r);
}
}
foreach (IBlueRow r in dp.addedRows) {
foreach (IBlueRow r in dp.addedRows)
{
myBaseTable.addRow(r);
if (myFilter.filter(r)) {
if (myFilter.filter(r))
{
adds.addRow(r);
myStateTable.addRow(r);
}
@@ -63,10 +76,13 @@ namespace BCG.StreamWorld.Addition {
myFireEvent(new DataPacket(myStateTable, adds, dels));
}
public IStreamFilter filter {
set {
public IStreamFilter filter
{
set
{
myFilter = value;
if (myBaseTable != null) {
if (myBaseTable != null)
{
IBlueRowCollection oldState = new BlueTable(myStateTable);
myStateTable.clear();
foreach (IBlueRow r in myBaseTable)
@@ -5,7 +5,7 @@ using System.Collections.Generic;
using System.Linq;
namespace BCG {
internal class GroupByStream : MethodDrivenStream {
public class GroupByStream : MethodDrivenStream {
private class MyFuncGroupByEvaluator : IGroupByEvaluator {
private string[] _groupby_cols;
private Func<DataPacket, IBlueRow> _groupby_func;
@@ -40,4 +40,4 @@ namespace BCG {
myFireSwitch = myLastFiredPacket.addedRows.Count + myLastFiredPacket.deletedRows.Count == 0 ? false : true;
}
}
}
}
@@ -280,28 +280,6 @@ namespace BCG {
#region json
public static void write_to_json_file(this IBlueRowCollection table, string file_path) {
write_dict_to_json_file(file_path, table.to_namevalue_pairs());
}
public static void write_dict_to_json_file(string file_path, List<Dictionary<string, string>> data) {
var options = new JsonSerializerOptions { WriteIndented = true };
string json = System.Text.Json.JsonSerializer.Serialize(data, options);
File.WriteAllText(file_path, json);
}
public static List<Dictionary<string, string>> ReadFromFile(string filePath) {
string json = File.ReadAllText(filePath);
return System.Text.Json.JsonSerializer.Deserialize<List<Dictionary<string, string>>>(json);
}
public static void write_table_to_json(IBlueRowCollection table) {
List<Dictionary<string, string>> rows = new List<Dictionary<string, string>>();
foreach (IBlueRow row in table) {
row.to_json();
}
}
public static Dictionary<string, string> convert_json_to_dictionary(string json_string) {
Dictionary<string, string> dictionary = JsonConvert.DeserializeObject<Dictionary<string, string>>(json_string);
return dictionary;
@@ -370,14 +348,6 @@ namespace BCG {
}
return rv;
}
public static List<Dictionary<string,string>> to_namevalue_pairs(this IBlueRowCollection table) {
List<Dictionary<string, string>> rv = new List<Dictionary<string, string>>();
foreach (IBlueRow r in table) {
rv.Add(r.to_dict());
}
return rv;
}
#region Schema types
// these are the only types allowed in schemas
private static IReadOnlyDictionary<TypeCode, Type> _typecode_to_type = new Dictionary<TypeCode, Type>
@@ -1255,20 +1225,6 @@ namespace BCG {
return rv;
}
public static Dictionary<string, string> to_dict(this IBlueRow r) {
Dictionary<string, string> map = new Dictionary<string, string>();
foreach (ColumnInfo c in r.schema.columnInfo) {
map.Add(c.colName, r[c.colName].ToString());
}
return map;
}
public static string to_json(this IBlueRow r) {
Dictionary<string, string> map = r.to_dict();
string json = System.Text.Json.JsonSerializer.Serialize(map);
return json;
}
public static void compute_differences_fast(IBlueRowCollection new_rows, IBlueRowCollection old_rows,
out IBlueRowCollection out_d_rows, out IBlueRowCollection out_a_rows) {
if (old_rows == null || old_rows.Count == 0) {
@@ -47,10 +47,10 @@ namespace BCG {
/// </summary>
/// <param name="assemblyName"></param>
public void loadAssembly(string assemblyName) {
Func<Type, bool> is_blue_orb = (t) => {
//if (!assemblyName.Contains("SharedStreams.dll"))
// return true;
return t.Namespace == "bts.BlueOrb";// || t.Namespace == "BCG";
Func<Type, bool> is_silverpine_only = (t) => {
if (!assemblyName.Contains("SharedStreams.dll"))
return true;
return t.Namespace == "bts.SilverPine";
};
bts.Log.Gen.info($"PrimitiveManager.loadAssembly assemblyName: {assemblyName}");
if (myAssemblyNames.Contains(assemblyName))
@@ -61,7 +61,7 @@ namespace BCG {
Assembly.LoadFrom(assemblyName);
foreach (Type type in loadedAssembly.GetTypes())
if (!type.IsInterface && is_blue_orb(type)
if (!type.IsInterface && is_silverpine_only(type)
&& !type.IsAbstract
&& type.IsSubclassOf(typeof(ViewStream))
&& type.IsPublic) {
@@ -77,7 +77,7 @@ namespace BCG {
break;
}
if (isVisible) {
// Log.assert_d(myTypeLookupTable[userRep] == null, type.FullName);
Log.assert_d(myTypeLookupTable[userRep] == null, type.FullName);
myTypeLookupTable[userRep] = type;
}
}
@@ -1,5 +1,4 @@
namespace bts.BlueOrb {
using bts.SilverPine;
namespace bts.SilverPine {
using BCG;
using System;
@@ -1,118 +1,117 @@
// TODO: Remove
//// namespace bts.SilverPine
//// using bts;
//// using BCG;
//// using MongoDB.Driver.Linq;
//// using System.ComponentModel.DataAnnotations;
//// using ZstdSharp.Unsafe;
// namespace bts.SilverPine {
// using bts;
// using BCG;
// using MongoDB.Driver.Linq;
// using System.ComponentModel.DataAnnotations;
// using ZstdSharp.Unsafe;
//// [DataStreamUserRespresentation("VWAP")]
//// public class VWAPStream : ViewStream {
//// // returns the VWAP stream for each contract
//// private static ViewStream build_vwap_stream(string symbol, DateTime begin_dt, DateTime end_dt) {
//// var ohlcv_params = OHLCVStream.get_params(begin_dt, end_dt);
//// ViewStream ohlc_str = new OHLCVStream(symbol, ohlcv_params);
//// // stream is grouped per contract and then vwap is calculated per contract
//// GlobalMapStream rs = new GlobalMapStream(ohlc_str, "ts_event", "contract",
//// (rows, idx) => {
//// Log.Gen.info($"VWAPStream.GlobalMapStream delegate n_rows: {rows.Count} idx: {idx}");
//// IBlueRow ir = rows.ElementAt((int)idx);
//// var sel_rows = rows.Skip((int)idx).Where(r => r.get_datetime("ts_event") >= end_dt && r.get_datetime("ts_event") <= begin_dt);
//// var volumeprices = sel_rows.Select(r => Tuple.Create(r.get<int>("volume"), r.get<double>("close"))).ToList();
//// double vwap = IndicatorCalc.calculate_VWAP(volumeprices);
//// IBlueTableSchema schema = ir.schema.add_columns(Tuple.Create(typeof(double), "vwap"));
//// MutableBlueRow nr = new MutableBlueRow(schema, ir);
//// //vwap = -313.987;
//// nr.setValue("vwap", vwap);
//// return nr;
//// });
//// return rs;
//// }
//// // the args want dt1 > dt2, if its nto we, swap them
//// static void swap_if_necessary(ref DateTime dt1, ref DateTime dt2) {
//// if (dt1 < dt2) {
//// DateTime aux = dt1;
//// dt1 = dt2;
//// dt2 = aux;
//// }
//// }
// [DataStreamUserRespresentation("VWAP")]
// public class VWAPStream : ViewStream {
// // returns the VWAP stream for each contract
// private static ViewStream build_vwap_stream(string symbol, DateTime begin_dt, DateTime end_dt) {
// var ohlcv_params = OHLCVStream.get_params(begin_dt, end_dt);
// ViewStream ohlc_str = new OHLCVStream(symbol, ohlcv_params);
// // stream is grouped per contract and then vwap is calculated per contract
// GlobalMapStream rs = new GlobalMapStream(ohlc_str, "ts_event", "contract",
// (rows, idx) => {
// Log.Gen.info($"VWAPStream.GlobalMapStream delegate n_rows: {rows.Count} idx: {idx}");
// IBlueRow ir = rows.ElementAt((int)idx);
// var sel_rows = rows.Skip((int)idx).Where(r => r.get_datetime("ts_event") >= end_dt && r.get_datetime("ts_event") <= begin_dt);
// var volumeprices = sel_rows.Select(r => Tuple.Create(r.get<int>("volume"), r.get<double>("close"))).ToList();
// double vwap = IndicatorCalc.calculate_VWAP(volumeprices);
// IBlueTableSchema schema = ir.schema.add_columns(Tuple.Create(typeof(double), "vwap"));
// MutableBlueRow nr = new MutableBlueRow(schema, ir);
// //vwap = -313.987;
// nr.setValue("vwap", vwap);
// return nr;
// });
// return rs;
// }
// // the args want dt1 > dt2, if its nto we, swap them
// static void swap_if_necessary(ref DateTime dt1, ref DateTime dt2) {
// if (dt1 < dt2) {
// DateTime aux = dt1;
// dt1 = dt2;
// dt2 = aux;
// }
// }
//// // this assumes the ohlc_str has diff contracts
//// public VWAPStream(string symbol_or_contract, DateTime begin_dt, DateTime end_dt) {
//// swap_if_necessary(ref begin_dt, ref end_dt);
//// Tuple<ContractUtils.ContractCategory, string> ccat_sym_contr = ContractUtils.get_symbol_category(symbol_or_contract);
//// string symbol = String.Empty;
//// symbol = ccat_sym_contr.Item2;
//// var ohlcv_params = OHLCVStream.get_params(begin_dt, end_dt);
//// ViewStream vwap_str = build_vwap_stream(symbol, begin_dt, end_dt);
//// // vwap_str now has vwap for all contracts. if the arg was a symbol, we need to filter it for the active contract,
//// // whihc we do with FutureActivesContractStream. If the arg was a contract we'll simply filter it
//// ViewStream rs = vwap_str;
//// switch (ccat_sym_contr.Item1) {
//// case ContractUtils.ContractCategory.FutureString:
//// case ContractUtils.ContractCategory.SpreadString:
//// rs = new FilterStream(vwap_str, r => r.get<string>("contract").CompareTo(ccat_sym_contr.Item2) == 0);
//// break;
//// case ContractUtils.ContractCategory.Symbol:
//// rs = new FuturesActiveContractStream(vwap_str);
//// break;
//// }
//// rs = new FilterStream(rs, r => r.get_datetime("ts_event").CompareTo(begin_dt) == 0);
//// rs = new NewProjectStream(rs, "! vwap");
//// mySetRealSource(rs);
//// }
// // this assumes the ohlc_str has diff contracts
// public VWAPStream(string symbol_or_contract, DateTime begin_dt, DateTime end_dt) {
// swap_if_necessary(ref begin_dt, ref end_dt);
// Tuple<ContractUtils.ContractCategory, string> ccat_sym_contr = ContractUtils.get_symbol_category(symbol_or_contract);
// string symbol = String.Empty;
// symbol = ccat_sym_contr.Item2;
// var ohlcv_params = OHLCVStream.get_params(begin_dt, end_dt);
// ViewStream vwap_str = build_vwap_stream(symbol, begin_dt, end_dt);
// // vwap_str now has vwap for all contracts. if the arg was a symbol, we need to filter it for the active contract,
// // whihc we do with FutureActivesContractStream. If the arg was a contract we'll simply filter it
// ViewStream rs = vwap_str;
// switch (ccat_sym_contr.Item1) {
// case ContractUtils.ContractCategory.FutureString:
// case ContractUtils.ContractCategory.SpreadString:
// rs = new FilterStream(vwap_str, r => r.get<string>("contract").CompareTo(ccat_sym_contr.Item2) == 0);
// break;
// case ContractUtils.ContractCategory.Symbol:
// rs = new FuturesActiveContractStream(vwap_str);
// break;
// }
// rs = new FilterStream(rs, r => r.get_datetime("ts_event").CompareTo(begin_dt) == 0);
// rs = new NewProjectStream(rs, "! vwap");
// mySetRealSource(rs);
// }
//// public VWAPStream(string symbol_or_contract, string begin_dt, string end_dt)
//// :
//// this(symbol_or_contract, DateTime.Parse(begin_dt), DateTime.Parse(end_dt)) { }
//// }
//// }
// public VWAPStream(string symbol_or_contract, string begin_dt, string end_dt)
// :
// this(symbol_or_contract, DateTime.Parse(begin_dt), DateTime.Parse(end_dt)) { }
// }
// }
//using System;
using System;
//using BCG;
using BCG;
//using bts.api;
using bts.api;
//namespace bts.SilverPine {
// /*
// ----------------------------------------------------------------------
// *Expression*
// ----------------------------------------------------------------------
// //nl("Find Settlement price of the SP500 future in the first week of 2023.")
namespace bts.SilverPine {
/*
----------------------------------------------------------------------
*Expression*
----------------------------------------------------------------------
//nl("Find Settlement price of the SP500 future in the first week of 2023.")
// r=FutureHistoricalBars("ES", "One_Day", "1/1/2024", "2/1/2024")
// r=math(r, "vprice = volume * close")
// r=aggregate(r, "vprice:Sum, volume:Sum")
// r=math(r, "vwap = vprice_sum / volume_sum")
// r=newProject(r, "vwap !")
r=FutureHistoricalBars("ES", "One_Day", "1/1/2024", "2/1/2024")
r=math(r, "vprice = volume * close")
r=aggregate(r, "vprice:Sum, volume:Sum")
r=math(r, "vwap = vprice_sum / volume_sum")
r=newProject(r, "vwap !")
// ----------------------------------------------------------------------
// *Schema*
// ----------------------------------------------------------------------
// id System.Int32
// vwap System.Double
// */
----------------------------------------------------------------------
*Schema*
----------------------------------------------------------------------
id System.Int32
vwap System.Double
*/
// [DataStreamUserRespresentation("VWAP")]
// public class VWAPStream : ViewStream {
// public VWAPStream(string symbol_or_contract, DateTime beg_dt, DateTime end_dt)
// :
// this(symbol_or_contract, beg_dt, end_dt, ViewStreamDatabase.instance) { }
[DataStreamUserRespresentation("VWAP")]
public class VWAPStream : ViewStream {
public VWAPStream(string symbol_or_contract, DateTime beg_dt, DateTime end_dt)
:
this(symbol_or_contract, beg_dt, end_dt, ViewStreamDatabase.instance) { }
// public VWAPStream(string symbol_or_contract, DateTime beg_dt, DateTime end_dt, ViewStreamDatabase db) {
// //var fc = new ContractUtils.FutureContract(symbol_or_contract);
// ViewStream r = new FutureHistoricalBarsStream(symbol_or_contract, beg_dt.ToString(), end_dt.ToString());
// // since this is already filtered, we can groupBy iit without a groupBy column
// r = new MathStream(r,
// "vprice = volume * close");
// r = new NonIncGroupByStream(r,
// "vprice:Sum, volume:Sum");
// r = new MathStream(r,
// "vwap = vprice_sum / volume_sum");
// r = new NewProjectStream(r,
// "vwap !");
// mySetRealSource(r);
// }
// }
//}
public VWAPStream(string symbol_or_contract, DateTime beg_dt, DateTime end_dt, ViewStreamDatabase db) {
//var fc = new ContractUtils.FutureContract(symbol_or_contract);
ViewStream r = new FutureHistoricalBarsStream(symbol_or_contract, beg_dt.ToString(), end_dt.ToString());
// since this is already filtered, we can groupBy iit without a groupBy column
r = new MathStream(r,
"vprice = volume * close");
r = new NonIncGroupByStream(r,
"vprice:Sum, volume:Sum");
r = new MathStream(r,
"vwap = vprice_sum / volume_sum");
r = new NewProjectStream(r,
"vwap !");
mySetRealSource(r);
}
}
}
@@ -1,5 +1,4 @@
namespace bts.BlueOrb {
using bts.SilverPine;
namespace bts.SilverPine {
using BCG;
using System.Runtime.CompilerServices;
@@ -46,7 +45,7 @@ namespace bts.BlueOrb {
}
[DataStreamUserRespresentation("FuturesHistorical1DayBars")]
internal class FuturesHistorical1DayBarsStream : ViewStream {
public class FuturesHistorical1DayBarsStream : ViewStream {
// this gets the raw bars for a symbol (perhaps make it into a raw bars stream?)
public FuturesHistorical1DayBarsStream(string symbol) :
@@ -104,7 +103,7 @@ namespace bts.BlueOrb {
}
[DataStreamUserRespresentation("FuturesHistorical1MinBars")]
internal class FuturesHistorical1MinuteBarsStream : ViewStream {
public class FuturesHistorical1MinuteBarsStream : ViewStream {
// same as the above but for one min
// this gets the raw bars for a symbol
public FuturesHistorical1MinuteBarsStream(string symbol) :
@@ -188,6 +187,46 @@ namespace bts.BlueOrb {
}
}
[DataStreamUserRespresentation("FuturePriceDiff")]
public class FutureHistoricalBarsPriceDiffStream : ViewStream {
public FutureHistoricalBarsPriceDiffStream(string future_contract, string dt1, string dt2)
:
this(new ContractUtils.FutureContract(future_contract), DateTime.Parse(dt1), DateTime.Parse(dt2), "close", ViewStreamDatabase.instance) {
}
public FutureHistoricalBarsPriceDiffStream(string future_contract, DateTime dt1, DateTime dt2)
:
this(new ContractUtils.FutureContract(future_contract), dt1, dt2, "close", ViewStreamDatabase.instance) {
}
public FutureHistoricalBarsPriceDiffStream(string future_contract, DateTime dt1, DateTime dt2, string col)
:
this(new ContractUtils.FutureContract(future_contract), dt1, dt2, col, ViewStreamDatabase.instance) {
}
public FutureHistoricalBarsPriceDiffStream(ContractUtils.FutureContract fc, DateTime dt1, DateTime dt2, string col, ViewStreamDatabase vsd) {
BarSize bsize = dt2 - dt1 >= TimeSpan.FromDays(1.0) ? BarSize.ONE_DAY : BarSize.ONE_MIN;
ViewStream rs = new FutureHistoricalBarsStream(fc, BarSize.ONE_DAY, new DateTimeRange(dt1, dt2));
rs = new ColumnDifferenceStream(rs, r => {
DateTime r_dt = r.get_datetime("ts_event");
var rv = dt1.Equals(r_dt);
return rv;
},
//r => dt2.Equals(r.get_datetime("ts_event")), col, col);
r => {
DateTime r_dt = r.get_datetime("ts_event");
var rv = dt2.Equals(r_dt);
return rv;
}, col, col);
rs = new NewProjectStream(rs, "diff !");
mySetRealSource(rs);
}
}
// takes the contract name and expands it into multuiple columns corresponding to the contracts.
// the contract columns are a union, of all properties with some of them empty
public class ContractExpanderStream : ViewStream {
@@ -208,29 +247,3 @@ namespace bts.BlueOrb {
}
}
namespace bts.BlueOrb {
using BCG;
using bts.api;
using Newtonsoft.Json.Serialization;
[DataStreamUserRespresentation("FuturesContinousContract")]
public class FuturesContinousContractStream : ViewStream {
// the argument is symbol, like "ES" etc and not the contract itself
public FuturesContinousContractStream(string symbol) : this(symbol, ViewStreamDatabase.instance) { }
public FuturesContinousContractStream(string symbol, ViewStreamDatabase db) {
symbol = symbol.EndsWith("*") ? symbol : symbol + "*";
ViewStream r = new FuturesHistorical1DayBarsStream(symbol);
r = new ContractExpanderStream(r, "symbol");
r = new FilterStream(r, "contract_type='Future'");
// finding the max volume across symbols, if we need to do this for open_interest, then join here
// with the open interest stream on symbol/contract
r = new NonIncGroupByStream(r, "ts_event", "volume:Max");
r = new NewProjectStream(r, "ts_event symbol !");
r = new SortStream(r, "-ts_event");
mySetRealSource(r);
}
}
}
@@ -1,53 +1,53 @@
//using BCG;
using BCG;
//namespace bts.SilverPine {
// /*
// ----------------------------------------------------------------------
// *Expression*
// ----------------------------------------------------------------------
// //r=FutureHistoricalBars("ES*", "02/01/2024 09:30 AM", "02/01/2024 09:30 PM")
// r=FutureHistoricalBars("ES*","One_Day", 02/01/2024, 02/01/2025)
// r=contractExpander(r,"symbol")
// r=filter(r,"contract_type='FutureSpread'")
// r=aggregate(r,"symbol","volume:Sum","ts_event")
// r=aggregate(r,"volume_sum:Max")
// r=newProject(r,"volume_sum_max:volume symbol !")
namespace bts.SilverPine {
/*
----------------------------------------------------------------------
*Expression*
----------------------------------------------------------------------
//r=FutureHistoricalBars("ES*", "02/01/2024 09:30 AM", "02/01/2024 09:30 PM")
r=FutureHistoricalBars("ES*","One_Day", 02/01/2024, 02/01/2025)
r=contractExpander(r,"symbol")
r=filter(r,"contract_type='FutureSpread'")
r=aggregate(r,"symbol","volume:Sum","ts_event")
r=aggregate(r,"volume_sum:Max")
r=newProject(r,"volume_sum_max:volume symbol !")
// ----------------------------------------------------------------------
// *Schema*
// ----------------------------------------------------------------------
// symbol System.String
// id System.Int32
// volume System.Double
// */
----------------------------------------------------------------------
*Schema*
----------------------------------------------------------------------
symbol System.String
id System.Int32
volume System.Double
*/
// [DataStreamUserRespresentation("VolumeAgg")]
// public class VolumeAggregationStream : ViewStream {
// public VolumeAggregationStream(string future_contract, string contract_type, DateTime range_beg, DateTime range_end)
// : this(future_contract, contract_type, range_beg, range_end, ViewStreamDatabase.instance) { }
[DataStreamUserRespresentation("VolumeAgg")]
public class VolumeAggregationStream : ViewStream {
public VolumeAggregationStream(string future_contract, string contract_type, DateTime range_beg, DateTime range_end)
: this(future_contract, contract_type, range_beg, range_end, ViewStreamDatabase.instance) { }
// public VolumeAggregationStream(string future_contract, string contract_type, string range_beg, string range_end)
// :
// this(future_contract, contract_type, DateTime.Parse(range_beg), DateTime.Parse(range_end), ViewStreamDatabase.instance) { }
public VolumeAggregationStream(string future_contract, string contract_type, string range_beg, string range_end)
:
this(future_contract, contract_type, DateTime.Parse(range_beg), DateTime.Parse(range_end), ViewStreamDatabase.instance) { }
// public VolumeAggregationStream(string future_contract, string contract_type, DateTime range_beg, DateTime range_end, ViewStreamDatabase db) {
// future_contract = future_contract.EndsWith("*") ? future_contract : future_contract + "*";
// ViewStream r;
// r = new FutureHistoricalBarsStream(future_contract,
// range_beg.ToString(), range_end.ToString());
// r = new ContractExpanderStream(r, "symbol");
// r = new FilterStream(r, $"contract_type='{contract_type}'");
// r = new NonIncGroupByStream(r, "symbol", "volume:Sum", "ts_event");
// r = new NonIncGroupByStream(r, "volume_sum:Max");
// r = new NewProjectStream(r, "volume_sum_max:volume symbol !");
// mySetRealSource(r);
// }
// }
public VolumeAggregationStream(string future_contract, string contract_type, DateTime range_beg, DateTime range_end, ViewStreamDatabase db) {
future_contract = future_contract.EndsWith("*") ? future_contract : future_contract + "*";
ViewStream r;
r = new FutureHistoricalBarsStream(future_contract,
range_beg.ToString(), range_end.ToString());
r = new ContractExpanderStream(r, "symbol");
r = new FilterStream(r, $"contract_type='{contract_type}'");
r = new NonIncGroupByStream(r, "symbol", "volume:Sum", "ts_event");
r = new NonIncGroupByStream(r, "volume_sum:Max");
r = new NewProjectStream(r, "volume_sum_max:volume symbol !");
mySetRealSource(r);
}
}
// [DataStreamUserRespresentation("PeriodicVolumeAgg")]
// public class PeriodicVolumeAggregationStream : ViewStream {
// public PeriodicVolumeAggregationStream(string future_contract, string contract_type, DateTime range_beg, DateTime range_end,
// TimeSpan period, ViewStreamDatabase db) {
// }
// }
//}
[DataStreamUserRespresentation("PeriodicVolumeAgg")]
public class PeriodicVolumeAggregationStream : ViewStream {
public PeriodicVolumeAggregationStream(string future_contract, string contract_type, DateTime range_beg, DateTime range_end,
TimeSpan period, ViewStreamDatabase db) {
}
}
}
@@ -76,3 +76,30 @@ namespace bts.SilverPine {
}
}
}
namespace bts.SilverPine {
using BCG;
using bts.api;
using Newtonsoft.Json.Serialization;
[DataStreamUserRespresentation("FuturesContinousContract")]
public class FuturesContinousContractStream : ViewStream {
// the argument is symbol, like "ES" etc and not the contract itself
public FuturesContinousContractStream(string symbol) : this(symbol, ViewStreamDatabase.instance) { }
public FuturesContinousContractStream(string symbol, ViewStreamDatabase db) {
symbol = symbol.EndsWith("*") ? symbol : symbol + "*";
ViewStream r = new FuturesHistorical1DayBarsStream(symbol);
r = new ContractExpanderStream(r, "symbol");
r = new FilterStream(r, "contract_type='Future'");
// finding the max volume across symbols, if we need to do this for open_interest, then join here
// with the open interest stream on symbol/contract
r = new NonIncGroupByStream(r, "ts_event", "volume:Max");
r = new NewProjectStream(r, "ts_event symbol !");
r = new SortStream(r, "-ts_event");
mySetRealSource(r);
}
}
}
@@ -1,18 +1,16 @@
namespace bts.SilverPine {
using bts;
using BCG;
//TODO: Remove
//namespace bts.SilverPine {
// using bts;
// using BCG;
// // it uses the volume (better to use open interest perhap) to detect the rollover, ie when a new
// // future exp becomes the highest volume one
// [DataStreamUserRespresentation("FuturesRoll")]
// public class FuturesRollStream : ViewStream {
// public FuturesRollStream(string future_symbol) {
// ViewStream rs = new FuturesContinousContractStream(future_symbol);
// rs = new ColumnEvolutionStream(rs, "ts_event", "symbol");
// rs = new NewProjectStream(rs, "symbol ts_event:roll_date !");
// mySetRealSource(rs);
// }
// }
//}
// it uses the volume (better to use open interest perhap) to detect the rollover, ie when a new
// future exp becomes the highest volume one
[DataStreamUserRespresentation("FuturesRoll")]
public class FuturesRollStream : ViewStream {
public FuturesRollStream(string future_symbol) {
ViewStream rs = new FuturesContinousContractStream(future_symbol);
rs = new ColumnEvolutionStream(rs, "ts_event", "symbol");
rs = new NewProjectStream(rs, "symbol ts_event:roll_date !");
mySetRealSource(rs);
}
}
}
@@ -1,27 +1,26 @@
// TODO: Remove
//namespace bts.SilverPine {
// using BCG;
namespace bts.SilverPine {
using BCG;
// using bts;
// // used directly in blueorb
// [DataStreamUserRespresentation("Settlement")]
// public class SettlementPriceStream : ViewStream {
using bts;
// used directly in blueorb
[DataStreamUserRespresentation("Settlement")]
public class SettlementPriceStream : ViewStream {
// // public SettlementPriceStream(string future_name, string date_txt)
// // : this(future_name, date_txt, date_txt) { }
// public SettlementPriceStream(string future_name, string date_txt)
// : this(future_name, date_txt, date_txt) { }
// // public SettlementPriceStream(string future_name, string begin_dt, string end_dt)
// // : this(future_name, DateTime.Parse(begin_dt).Date, DateTime.Parse(end_dt).Date) { }
// public SettlementPriceStream(string future_name, string begin_dt, string end_dt)
// : this(future_name, DateTime.Parse(begin_dt).Date, DateTime.Parse(end_dt).Date) { }
// public SettlementPriceStream(string future_contract, DateTime begin_dt)
// : this(future_contract, begin_dt, begin_dt) { }
public SettlementPriceStream(string future_contract, DateTime begin_dt)
: this(future_contract, begin_dt, begin_dt) { }
// public SettlementPriceStream(string future_contract, DateTime begin_dt, DateTime end_dt) {
// ContractUtils.FutureContract fc = new ContractUtils.FutureContract(future_contract);
// ViewStream rs = new FutureHistoricalBarsStream(fc, BarSize.ONE_DAY, new DateTimeRange(begin_dt, end_dt));
// rs = new NewProjectStream(rs, "symbol ts_event:date close:settlement !");
// mySetRealSource(rs);
// }
public SettlementPriceStream(string future_contract, DateTime begin_dt, DateTime end_dt) {
ContractUtils.FutureContract fc = new ContractUtils.FutureContract(future_contract);
ViewStream rs = new FutureHistoricalBarsStream(fc, BarSize.ONE_DAY, new DateTimeRange(begin_dt, end_dt));
rs = new NewProjectStream(rs, "symbol ts_event:date close:settlement !");
mySetRealSource(rs);
}
// }
//}
}
}
@@ -1,571 +1,276 @@
# BlueOrbDSL Master Learning Prompt v1.1 (Optimized)
# **DSL Master Learning Prompt**
**This version is the *final, frozen* canonical Master Learning Prompt for BlueOrbDSL model training and generation.**
**No further edits are required. This is the single source of truth for Learning Mode.**
**Reviewed and validated as optimal on \[insert date here, e.g. June 9, 2025\] after full Meta Mode review.**
This prompt is designed to teach the foundational concepts of the DSL for financial data streaming and transformation. The DSL operates on "streams," which fetch, transform, and process financial market data in real-time or historical contexts. Forget everything I have taught you about this DSL until now (since there are some contradicting and breaking changes that can not be reconciled with anything you might have learned before) and re-learn from this prompt.
**Change Log — v1 → v1.1**
(June 7, 2025\)
# **Introduction to Streams**
✅ Applied full Meta Mode review recommendations:
The DSL operates on "streams," its core functional units. Streams fetch tabular (from database, text files or network) or transform/operate on tabular data. They either act as **Datasources** (e.g., FutureHistoricalBars), that bootstrap the data or **Operators** (e.g., `FilterStream, ProjectStream, SingleRowMathStream, GroupByStream` etc) that transform or filter tabular data. Though the DSL can work on any tabular data, it is primarily written for streaming or static financial data
- Standardized **Arguments** section (bullet lists, first arg \= `parent_stream` for operators)
- Added **Adds Columns?** and **Modifies Schema?** rows for *all* streams
- Clarified **Special Notes** for all streams
- Clearly stated UnionStream schema requirements
- Explicitly documented JoinStream as inner join, with left/right prefix behavior
- Clarified behavior of **TimeBasedRollingFilterStream**
- Changed DateTimeExpanderStream new columns to a bullet list for clarity
- Standardized all **Example Usage** to fenced `dsl` code blocks
- Shortened **PythonDataframeStream** example; big one should go to appendix if needed
- Minor language and consistency cleanups across stream definitions
# **Streams**
**This version is now ready to serve as the authoritative MLP for BlueOrbDSL training and generation.**
When instructed to "Switch to Learning Mode," this will become the model’s single source of truth.
## **FutureHistoricalBars (DataSource Stream)**
The `FutureHistoricalBars` stream fetches OHLCV (Open, High, Low, Close, Volume) bars for financial futures contracts. It acts as a **data source** and provides the initial table that can be further processed by operators.
#### **Stream arguments (parameters):**
1. **`Future_contract` (string):** Specifies the contract to fetch data for. This can be one of the following 3 types
1. **Continuous contracts:** Use the base symbol (e.g., `ES` for S\&P 500 futures, CL for WTI Crude Oil Futures, GC for Gold Futures etc).
2. **Specific contracts:** Use a detailed symbol (e.g., `ESZ9` for December 2029 S\&P 500 futures, `GCU9`, for September 2025 Gold Future, `SIH9-J9`,
For SIlver March April 2019 Silver Future Calendar spread)
3. **All Contracts for a symbol:** If instead of specifying individual contracts, data for all contracts for a specific symbol is needed, use the symbol name followed by *.* So `ES*` for all SP Futures and spread contracts, ZC\* for all corn contracts etc.
2. **`Bar_size` (string):** The second argument defines the granularity of the bars. Accepted values:
* `ONE_DAY` – Daily bars.
* `ONE_MIN` – Minute bars.
3. **Optional:**
* **`range_begin` (datetime):** Start of the desired time range.
* **`range_end` (datetime):** End of the desired time range.
#### **Schema:**
The output of `FutureHistoricalBars` produces a table with the following schema (this is a classic OHLCV schema that you can find online easily)
| id | ts\_event | contract | open | high | low | close | volume |
| :--- | :-------------------- | :------- | :------ | :------ | :------ | :------ | :----- |
| `0` | `2024-06-08T00:00:00` | `ESZ9` | 5001.50 | 5052.25 | 4990.75 | 5025.50 | 866000 |
| | | | | | | | |
# **FilterStream (Operator Stream)**
Filters rows of a parent stream based on conditions applied to its columns.
#### **Stream arguments (parameters):**
1. `parent_stream (another stream)` This stream provides the data, that the FilterStream subscribers to
2. `filter_condition (string)` This is a text expressing a boolean condition that is applied to each row of the parent stream, if it returns false, the row is filtered out
#### **Examples:**
1. **NL prompt:** Show me all the days when the open price for SP 500 Futures was greater than or equal to 5000:
**DSL:** `r = FilterStream(FutureHistoricalBars("ES", "ONE_DAY"), "open >= 5000")`
2. **NL prompt:** Show me all the days when the close price for SP 500 Futures was less than 3000 and volume was greater than half a million
**DSL:** `r = FilterStream(FutureHistoricalBars("ES", "ONE_MIN"), "close < 3000 and volume > 500000)`
3. **NL prompt:** Show me all the daily bars for the SP 500 Future spread ESU9-Z9 where its volume exceeded 100K
**DSL:** `r = FilterStream(FutureHistoricalBars("ES*", "ONE_MIN"), "contract = ‘ESU9-Z9’ and volume >= 100000”)`
**Notice** how expressions can be joined with AND and OR. These expressions can be nested with parenthesis.Also string needs to be in single quotes.
# **ProjectStream (Operator Stream)**
Changes the name of the columns in the parent stream. So for example, if we have to rename the close column to settlement, we can do this with ProjectStream
#### **Stream arguments (parameters):**
3. `parent_stream (another stream)` This stream provides the data, that the ProjectStream subscribers to
4. `renaming (string)` This is a text expressed in the form original\_col\_name:projected\_col\_name, original\_col\_name:projected\_col\_name,
#### **Examples:**
The following DSL renames the open column to O, high to H, low to L and close to C
**DSL:** `r = ProjectStream(FutureHistoricalBars("ES", "ONE_DAY"), "open:O, high:H, low:L, close:C”)`
# **GroupByStream (Operator Stream)**
The `GroupByStream` function is a DSL operator used to perform aggregation operations on rows of a parent stream. It works similarly to SQL’s `GROUP BY`, allowing rows to be grouped by a specific column (optional) and applying aggregation functions to other columns.
---
# Table of Contents
#### **Stream arguments (parameters):**
1. Preamble
2. Streams Overview
3. Data Sources
3.1 FutureHistoricalBarsStream
3.2 LiveBBOStream
3.3 LiteralStream
4. Operators
4.1 FilterStream
4.2 ProjectStream
4.3 GroupByStream
4.4 SingleRowMathStream
4.5 MultiRowRollingMathStream
4.6 UnionStream
4.7 JoinStream
4.8 DateTimeExpanderStream
4.9 ContractExpanderStream
4.10 CountStream
4.11 TimeSeriesStream
4.12 TimeBasedRollingFilterStream
4.13 PythonDataframeStream
5. Reserved Words & Enumerations
6. Common Pitfalls
1. **parent\_stream (another stream):** The source stream provides the data to be operated on
2. **Group By Column (string, optional):**
A column in the schema of the parent stream used to group rows. All rows with the same value in this column are aggregated into a single group. If omitted, the aggregation is performed over all rows of the parent stream.
3. **Operation Text (string):** A string defining the column(s) to be aggregated and the operation(s) to apply, in the format:
`<column_name: operation>, <column_name: operation>, …` Each `<column_name: operation>` pair produces a new column in the output stream. The new column name is formatted as `<column_name>_<operation>` in lowercase.
---
## 1 Preamble
#### **Valid Aggregation Operations:**
BlueOrbDSL turns natural-language finance questions into **stream pipelines** that operate on tabular market data (historical or real-time).
Each *stream* is either a **Data Source** (fetches data) or an **Operator** (transforms data).
Streams compose: the output table of one becomes the input of the next.
**Convention:** For all operators, the **first argument** is always the `parent_stream`. Subsequent arguments are constructor parameters.
* `Mean`: Arithmetic mean of the values.
* `Median`: Median of the values.
* `Mode`: Most frequently occurring value.
* `Range`: Difference between the max and min.
* `StdDev`: Standard deviation of the values.
* `Max`: Maximum value.
* `Min`: Minimum value.
* `First`: First value in the group.
* `Last`: Last value in the group.
* `Sum`: Sum of the values.
* `EdgeDiff`: Difference between the edge values, meaning the difference between a certain col belonging to the first and last rows respectively
* `EdgeDiffPercentage`: Same as `EdgeDiff` but expressed as percentage instead
---
## 2 Streams Overview
#### **Examples:**
All stream definitions follow this table layout:
| Field | Meaning |
| :------------------- | :----------------------------------------------------- |
| **Type** | DataSource / Operator |
| **Purpose** | One-sentence description |
| **Arguments** | Bullet list; first arg \= parent\_stream (if Operator) |
| **Output Schema** | Columns created or removed |
| **Adds Columns?** | Yes / No |
| **Modifies Schema?** | Yes / No |
| **Example Usage** | Fenced `dsl` block |
| **Special Notes** | Constraints / caveats |
1. **NL:**
Find the total SP 500 volume executed in the first week of 2021\.
**DSL:**
`bars = FutureHistoricalBars("ES", “ONE_DAY”, “2021-01-03T00:00:00”,“2021-01-11T00:00:00”)`
`result = GroupByStream(bars,”volume_Sum”)`
2. **NL:**
**Find the highest price and lowest price of the WTI Crude Oil futures over the first half of 2024**
**DSL:**
`bars = FutureHistoricalBars("CL", “ONE_DAY”, “2021-01-03T00:00:00”,“2021-01-06T00:00:00”)`
`result = GroupByStream(bars,”high:Max low:Min”)`
`result = ProjectStream(result,”high_max:highest_price min_low: lowest_price”)`
3. **NL:**
**Find the Average daily volume per contract in Natural Gas in 2023**
**DSL:**
`bars = FutureHistoricalBars("NG*", “ONE_DAY”, “2023-01-03T00:00:00”,“2023-31-12T23:59:59”)`
`result = GroupByStream(bars, “contract”, ”volume:Mean”)`
`result = ProjectStream(result,”mean_volume:average_daily_volume”)`
---
## 3 Data Sources
#### **Schema Impact:**
### 3.1 FutureHistoricalBarsStream
The `AggregateStream` modifies the schema of the parent stream by adding new columns for each aggregation operation. Existing columns are unaffected.
| Field | Description |
| :------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Type** | DataSource |
| **Purpose** | Fetch historical OHLCV bars for futures contracts |
| **Arguments** | • `contract` — continuous (`ES`), specific (`ESZ9`), or wildcard (`ES*`) • \``bar_size`\` — ONE\_DAY, ONE\_MIN • `` `range_begin` `` — ISO datetime, optional • `` `range_end` `` — ISO datetime, optional |
| **Output Schema** | `id, ts_event, date, contract, open, high, low, close, volume` |
| **Adds Columns?** | Yes |
| **Modifies Schema?** | No |
| **Example Usage** | `bars = FutureHistoricalBarsStream("ES", "ONE_DAY", "2023-01-01", "2023-12-31")` |
| **Special Notes** | Whereas the ts\_event column is the timestamp of the column, date is the date of the day. |
# **SingleRowMathStream (Operator Stream)**
### 3.2 LiveBBOStream
| Field | Description |
| :------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Type** | DataSource |
| **Purpose** | Stream live best-bid/offer quotes |
| **Arguments** | • `contract` — e.g. `"NGU5"` or `"CL*"` |
| **Output Schema** | `ts_event, contract, bid_price, bid_size, ask_price, ask_size` |
| **Adds Columns?** | Yes |
| **Modifies Schema?** | No |
| **Example Usage** | `bbo = LiveBBOStream("NGU5")` |
| **Special Notes** | LiveBBOStream produces the latest value of the bid/ask price and size. It does not maintain history. Each new row overwrites the old one. For history use TimeSeriesStream |
### 3.3 LiteralStream
| Field | Description |
| :------------------- | :---------------------------------------------------------------------------------------------------------------- |
| **Type** | DataSource |
| **Purpose** | Inject small static tables via JSON records |
| **Arguments** | • `json_text` — string |
| **Output Schema** | Inferred from JSON keys |
| **Adds Columns?** | Yes |
| **Modifies Schema?** | No |
| **Example Usage** | `meta = LiteralStream('[{"event":"OPEC","date":"2024-06-01"}]')` |
| **Special Notes** | If you download the data from the internet, and want to use it conjunction with the BlueOrbDSL this can be useful |
The `SingleRowMathStream` is a DSL operator used to perform math operations, one row at a time, on the parent stream.
---
## 4 Operators
#### **Stream arguments (parameters):**
### 4.1 FilterStream
1. **parent\_stream (another stream):**The source stream provides the base data
2. **math\_expression(string):** This is a math expression which allows the creation of one or more new columns that are the output of a math expression on the existing columns provided by parent\_stream. This can a simple math operation like new\_col \= a \+ b, or any function from C\#’s System.Math namespace, like new\_column \= Math.Log(Math.Pow(volume, 1.1))
| Field | Description |
| :------------------- | :--------------------------------------------------------------------------------------------------------------------------- |
| **Type** | Operator |
| **Purpose** | Keep rows where a boolean condition is true |
| **Arguments** | • `parent_stream` •`` `condition` `` — string |
| **Output Schema** | Same as input |
| **Adds Columns?** | No |
| **Modifies Schema?** | No |
| **Special Notes** | Condition syntax: use ASCII single quotes (`contract = 'ESZ9'`). Supports `and`, `or`, parentheses, equality and inequality. |
#### **Examples:**
### 4.2 ProjectStream
4. **NL:**
Find the daily price movement in the SP 500 Futures through the month of Jan
**DSL:**
`bars = FutureHistoricalBars("ES", “ONE_DAY”, “2024-01-01T00:00:00”,“2024-01-31T00:00:00”)`
`result = SingleRowMathStream(bars,”daily_move=close-open”)`
5. **NL:**
Find the daily typical price of the WTI Crude Oil futures over the first half of 2024
**DSL:**
`bars = FutureHistoricalBars("CL", “ONE_DAY”, “2021-01-03T00:00:00”,“2021-01-06T00:00:00”)`
`result = SingleRowMathStream(bars, “typical_price = (high+low+close)/3.0”)`
| Field | Description |
| :------------------- | :---------------------------------------------------------------------------------- |
| **Type** | Operator |
| **Purpose** | Rename or drop columns |
| **Arguments** | • `parent_stream` • `` `column_map ``\` — string |
| **Output Schema** | Renamed subset |
| **Adds Columns?** | No |
| **Modifies Schema?** | Yes |
| **Example Usage** | `clean = ProjectStream(bars, "close:settlement, volume")` |
| **Special Notes** | **Cannot add new columns or constants.** Use LiteralStream \+ JoinStream if needed. |
#### **Input Space (Scope):** The scope of this operation is a **single row.** The SingleRowMAthStream as the name signifies, operates on columns of the same row only. **No multi row operations are allowed.** For multi row operations using what follows.
**Schema Impact:**
# **MultiRowRollingMathStream (Operator Stream)**
The `MultiRowRollingMathStream` function is a DSL operator used to perform mostly math operations on a rolling set of rows of a parent stream. It works similarly to the GroupByStream (described above) but instead of grouping rows together based on a column value, it operates on a rolling set of rows (where the rolling size is provided as an argument).
---
### 4.3 GroupByStream
#### **Stream arguments (parameters):**
| Field | Description |
| :------------------- | :----------------------------------------------------------------------------------------------------- |
| **Type** | Operator |
| **Purpose** | Aggregates rows by key (or globally) |
| **Arguments** | • `parent_stream` • `` `group_by` `` — string (blank for global) • `` `aggregation_string` `` — string |
| **Output Schema** | Adds `<column>_<aggregation>` columns |
| **Adds Columns?** | Yes |
| **Modifies Schema?** | Yes |
| **Example Usage** | `daily = GroupByStream(bars, "date", "volume:Sum")` |
| **Special Notes** | None |
### 4.4 SingleRowMathStream
| Field | Description |
| :------------------- | :--------------------------------------------------------------------------- |
| **Type** | Operator |
| **Purpose** | Add new columns from row-wise math expressions |
| **Arguments** | • `parent_stream` • `` `math_expression` `` — string |
| **Output Schema** | Adds new column(s) |
| **Adds Columns?** | Yes |
| **Modifies Schema?** | No |
| **Example Usage** | `r = SingleRowMathStream(bars, "daily_return = close - open")` |
| **Special Notes** | Operates on one row at a time only. Cannot reference values from other rows. |
### 4.5 MultiRowRollingMathStream
| Field | Description |
| :------------------- | :------------------------------------------------------------------------------------------------------------------------------------------ |
| **Type** | Operator |
| **Purpose** | Aggregates over rolling window of rows |
| **Arguments** | • `parent_stream` • `` `aggregation_string` `` — string • `` `sort_column` `` — string with \+/- prefix • `` `rolling_window_size` `` — int |
| **Output Schema** | Adds `<column>_<aggregation>` columns |
| **Adds Columns?** | Yes |
| **Modifies Schema?** | Yes |
| **Example Usage** | `ma20 = MultiRowRollingMathStream(bars, "close:Mean", "-ts_event", 20)` |
| **Special Notes** | sort\_column should be prefixed with \+ (ascending) or \- (descending). Example: "-ts\_event". |
### 4.6 UnionStream
| Field | Description |
| :------------------- | :------------------------------------------------------- |
| **Type** | Operator |
| **Purpose** | Append rows from two streams |
| **Arguments** | • `parent_stream1` • `parent_stream2` |
| **Output Schema** | Same as inputs |
| **Adds Columns?** | No |
| **Modifies Schema?** | No |
| **Example Usage** | `unioned = UnionStream(bars1, bars2)` |
| **Special Notes** | **Schemas must match exactly (column names AND types).** |
### 4.7 JoinStream
| Field | Description |
| :------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Type** | Operator |
| **Purpose** | Join two streams on equality of selected columns |
| **Arguments** | • `` `left_parent_stream` `` • `` `left_join_cols` `` — comma-separated string • `` `right_parent_stream` `` • `` `right_join_cols` `` — comma-separated string |
| **Output Schema** | Combined columns from both streams |
| **Adds Columns?** | Yes |
| **Modifies Schema?** | Yes |
| **Example Usage** | `joined = JoinStream(a, "ts_event", b, "ts_event")` |
| **Special Notes** | Always performs an **inner join** on equality. If columns overlap, prefix applied (`left_`, `right_`). If join columns args left empty, does cartesian product |
### 4.8 DateTimeExpanderStream
| Field | Description |
| :------------------- | :--------------------------------------------------------------------------------------------------------------------------------- |
| **Type** | Operator |
| **Purpose** | Adds columns with date/time components from a datetime column |
| **Arguments** | • `parent_stream` • `datetime_column` — string |
| **Output Schema** | Adds: • \`\_date\` • \`\_day\_of\_week\` • \`\_day\_of\_month\` • \`\_month\` • \`\_hour\` • \`\_minute\` • \`\_seconds\_of\_day\` |
| **Adds Columns?** | Yes |
| **Modifies Schema?** | Yes |
| **Example Usage** | `expanded = DateTimeExpanderStream(bars, "ts_event")` |
| **Special Notes** | None |
### 4.9 ContractExpanderStream
| Field | Description |
| :------------------- | :---------------------------------------------------- |
| **Type** | Operator |
| **Purpose** | Adds metadata columns based on `contract` |
| **Arguments** | • `parent_stream` • contract\_column — string |
| **Output Schema** | Adds: `• contract_type` |
| **Adds Columns?** | Yes |
| **Modifies Schema?** | Yes |
| **Example Usage** | `expanded = ContractExpanderStream(bars, "contract")` |
| **Special Notes** | None |
### 4.10 CountStream
| Field | Description |
| :------------------- | :-------------------------------- |
| **Type** | Operator |
| **Purpose** | Count number of rows |
| **Arguments** | • `parent_stream` |
| **Output Schema** | `id, count` |
| **Adds Columns?** | Yes |
| **Modifies Schema?** | Yes |
| **Example Usage** | `total_count = CountStream(bars)` |
| **Special Notes** | None |
### 4.11 TimeSeriesStream
| Field | Description |
| :------------------- | :------------------------------------------------------------------- |
| **Type** | Operator |
| **Purpose** | Append each update from parent stream into time series |
| **Arguments** | • `parent_stream` • \``buffer_size`\` — int |
| **Output Schema** | Parent schema \+ `timestamp` (if not already present) |
| **Adds Columns?** | Yes |
| **Modifies Schema?** | No |
| **Example Usage** | `ts_bbo = TimeSeriesStream(bbo, 10000)` |
| **Special Notes** | Append-only stream, does not deduplicate or collapse repeated values |
### 4.12 TimeBasedRollingFilterStream
| Field | Description |
| :------------------- | :--------------------------------------------------------------------------- |
| **Type** | Operator |
| **Purpose** | Keep rows newer than rolling window of N seconds |
| **Arguments** | • `parent_stream` • \``no_of_seconds`\` — int |
| **Output Schema** | Same as input |
| **Adds Columns?** | No |
| **Modifies Schema?** | No |
| **Example Usage** | `recent = TimeBasedRollingFilterStream(trades, 60)` |
| **Special Notes** | Drops rows older than `no_of_seconds` compared to current max timestamp seen |
### 4.13 PythonDataframeStream
| Field | Description |
| :------------------- | :---------------------------------------------------------------------------------------------------------------------- |
| **Type** | Operator |
| **Purpose** | Apply arbitrary Python transformation to DataFrame |
| **Arguments** | • `parent_stream` • `` `python_function` `` — string |
| **Output Schema** | Defined by function |
| **Adds Columns?** | Yes/No — depends on function |
| **Modifies Schema?** | Yes/No — depends on function |
| **Example Usage** | `r = PythonDataframeStream(bars, "def transform(df): return df[df.volume > 1e6]")` |
| **Special Notes** | **Use only as last resort.** Function should be small and well-documented. Avoid large multi-step logic where possible. |
1. **Parent Stream (another stream):** The source stream provides the data to be operated on
2. **Operation Text (string):** A string defining the column(s) to be operated and the operation(s) to apply, in the format:
`<column_name: operation>, <column_name: operation>, …` Each `<column_name: operation>` pair produces a new column in the output stream. The new column name is formatted as `<column_name>_<operation>` in lowercase. This is like the GroupByStream’s operations string.
3. **Sort/Order Column Name(string):** This controls the ordering of the rolling rows. This argument will be in the format \-`<column_name>` to sort the rolling rows in descending order by the `<column_name>`, or \+`<column_name> in` ascending order.
4. **Rolling Window Size(int)(optional):** The size of the rolling window, If this argument is left out all, the operations are applied to all rows, is Global Operation instead of Rolling Window Operation (it's like the optional groupBy col in GroupByStream).
---
## 5 Reserved Words & Enumerations
#### **Valid Aggregation Operations:**
- **Bar\_size:** `ONE_DAY`, `ONE_MIN`
- **Aggregation Operations:** `Mean`, `Median`, `Mode`, `Sum`, `Max`, `Min`, `StdDev`, `Range`, `First`, `Last`, `EdgeDiff`, `EdgeDiffPercentage`
* `Mean`: Arithmetic mean of the values.
* `Median`: Median of the values.
* `Mode`: Most frequently occurring value.
* `Range`: Difference between the max and min.
* `StdDev`: Standard deviation of the values.
* `Max`: Maximum value.
* `Min`: Minimum value.
* `First`: First value in the group.
* `Last`: Last value in the group.
* `Sum`: Sum of the values.
* `EdgeDiff`: Difference between the edge values, meaning the difference between a certain col belonging to the first and last rows respectively
* `EdgeDiffPercentage`: Same as `EdgeDiff` but expressed as percentage instead
---
## 6 Common Pitfalls
#### **Examples:**
| Pitfall | Correct Approach |
| :---------------------------------------------- | :--------------------------------------------------- |
| Adding literal columns in ProjectStream | Use LiteralStream → JoinStream |
| Smart quotes in conditions | Use `'...'` ASCII quotes |
| Large functions in PythonDataframeStream | Keep minimal or move to appendix |
| Ambiguous TimeBasedRollingFilterStream behavior | Now clearly stated: drops rows older than N seconds. |
1. **NL:**
**Find the 20 d MA of SP 500 Futures close prices**
`bars = FutureHistoricalBars("ES", “ONE_DAY”, “2021-01-03T00:00:00”,“2021-01-11T00:00:00”)`
`result = MultirowRollingMathStream(bars, “close:Mean”, “-ts_event”, 20)`
2. **NL:**
**Find the 90d Standard Deviation of WTI Crude Oil during 2023**
## 7 Reinforced Learning DSL Examples
**DSL:**
`bars = FutureHistoricalBars("CL", “ONE_DAY”, “2023-01-01T00:00:00”,“2023-12-31T00:00:00”)`
`// calculating typical_price`
This section contains **complete worked examples** of BlueOrbDSL pipelines answering complex natural language queries.
`result = SingleRowMathStream(bars, “typical_price = (high+low+close)/3.0”)`
`result = MultirowRollingMathStream(bars,“typical_price:StdDev”, “-ts_event”, 90)`
Each example shows:
`result=ProjectStream(result, “stddev_typical_price:90dStdDev”)`
- The **NL Query** (natural language question)
- The **DSL Solution** in full
- Optional **Notes**
3. **NL:**
**Find the Average daily volume of Natural Gas in 2023**
**DSL:**
`bars = FutureHistoricalBars("NG", “ONE_DAY”, “2023-01-03T00:00:00”,“2023-31-12T23:59:59”)`
`result = MultirowRollingMathStream(bars, ”volume:Mean”, “-ts_event”)`
`result = ProjectStream(result,”mean_volume:average_daily_volume”)`
---
### Example 1 — On which dates did SP 500 Futures open 2% higher than the previous day's close?
#### **Schema Impact:**
#### DSL Solution
The `MultirowRollingMathStream` modifies the schema of the parent stream by adding new columns for each operation. Existing columns are unaffected.
```
bars = FutureHistoricalBarsStream("ES", "ONE_DAY", "1900-01-01T00:00:00", "2100-12-31T23:59:59")
rolling_bars = MultiRowRollingMathStream(bars, "close:First, open:Last", "+ts_event", 2)
percent_change = SingleRowMathStream(rolling_bars, "percent_change = 100 * (open_last - close_first) / close_first")
filtered_bars = FilterStream(percent_change, "percent_change >= 2")
result = ProjectStream(filtered_bars, "ts_event, percent_change")
```
# **UnionStream (Operator Stream)**
---
Like the SQL Union appends all the rows from 2 streams together.
### Example 2 — What is the average price movement for e-minis on Wednesday between 9:30 am and 9:40 AM CST?
#### **Stream arguments (parameters):**
#### DSL Solution
5. `parent_stream1 (first stream)` This stream provides part of the data that will will be appended to the other part, provided by the following argument
6. `parent_stream2 (second stream)` This stream provides the other part of the data that will will be appended to the first part, provided by the first argument
```
bars = FutureHistoricalBarsStream("ES", "ONE_MIN", "1900-01-01T00:00:00", "2100-12-31T23:59:59")
expanded_bars = DateTimeExpanderStream(bars, "ts_event")
filtered_bars = FilterStream(expanded_bars, "ts_event_day_of_week = 'Wednesday' and ts_event_hour = 9 and ts_event_minute >= 30 and ts_event_minute < 40")
price_movement = SingleRowMathStream(filtered_bars, "movement = close - open")
average_movement = GroupByStream(price_movement, "", "movement:Mean")
result = ProjectStream(average_movement, "movement_mean:average_price_movement")
```
#### **Examples:**
---
4. **NL prompt:** Show me all the bars across SP 500 Futures and Nasdaq Futures
**DSL:** `r = UnionStream(FutureHistoricalBars("ES", "ONE_DAY"), FutureHistoricalBars("NQ", "ONE_DAY"))`
### Example 3 — What did the volume and price of ZN in the 5 minutes following the last 10 FOMC rates?
#### **Schema Impact:**
#### DSL Solution
None
```
bars_1 = FutureHistoricalBarsStream("ZN", "ONE_MIN", "2023-11-01T14:00:00", "2023-11-01T14:05:00")
bars_2 = FutureHistoricalBarsStream("ZN", "ONE_MIN", "2023-09-20T14:00:00", "2023-09-20T14:05:00")
bars_3 = FutureHistoricalBarsStream("ZN", "ONE_MIN", "2023-07-26T14:00:00", "2023-07-26T14:05:00")
bars_4 = FutureHistoricalBarsStream("ZN", "ONE_MIN", "2023-06-14T14:00:00", "2023-06-14T14:05:00")
bars_5 = FutureHistoricalBarsStream("ZN", "ONE_MIN", "2023-05-03T14:00:00", "2023-05-03T14:05:00")
bars_6 = FutureHistoricalBarsStream("ZN", "ONE_MIN", "2023-03-22T14:00:00", "2023-03-22T14:05:00")
bars_7 = FutureHistoricalBarsStream("ZN", "ONE_MIN", "2023-02-01T14:00:00", "2023-02-01T14:05:00")
bars_8 = FutureHistoricalBarsStream("ZN", "ONE_MIN", "2022-12-14T14:00:00", "2022-12-14T14:05:00")
bars_9 = FutureHistoricalBarsStream("ZN", "ONE_MIN", "2022-11-02T14:00:00", "2022-11-02T14:05:00")
bars_10 = FutureHistoricalBarsStream("ZN", "ONE_MIN", "2022-09-21T14:00:00", "2022-09-21T14:05:00")
# **DateTimeExpanderStream (Operator Stream)**
all_bars = UnionStream(bars_1, bars_2)
all_bars = UnionStream(all_bars, bars_3)
all_bars = UnionStream(all_bars, bars_4)
all_bars = UnionStream(all_bars, bars_5)
all_bars = UnionStream(all_bars, bars_6)
all_bars = UnionStream(all_bars, bars_7)
all_bars = UnionStream(all_bars, bars_8)
all_bars = UnionStream(all_bars, bars_9)
all_bars = UnionStream(all_bars, bars_10)
DateTimExpanderStream expands one datetime column into various datetime components, like seconds in day, day of week, date etc, so new columns are added for those. The existing columns remain as is.
grouped_data = GroupByStream(all_bars, "date", "open:First, close:Last, high:Max, low:Min, volume:Sum")
result = ProjectStream(grouped_data, "ts_event_date, open_first:5_min_open, close_last:5_min_close, high_max:5_min_high, low_min:5_min_low, volume_sum:5_min_volume")
```
#### **Stream arguments (parameters):**
---
1. `parent_stream1 (base stream)` This parent stream that provides the data, including the datetime columns that will be expanded into multiple columns
2. `Datetime_input_column (string)` This is the column that will be expanded into multiple columns.
### Example 4 — How much volume of RBOB typically trades during the settlement window?
**New Columns:**
`<input_dt_col>_date` (Date of the value, without the time part)
#### DSL Solution
* `<input_dt_col>_day,` (Day of the week)
* `<input_dt_col>_day_of_month` (day of the month, This is an int between 0 and 31\)
* `<input_dt_col>_month,` (month of the year, this is a numerical value between 1 and 12
* `input_dt_col>_hour,` (Hour of the day, value bw 0 and 23\)
* `<input_dt_col>_min",` (month part of this DateTime. Value is integer between 1 and 12).
* `<input_dt_col>_seconds_of_day",` (Total seconds of the day)
```
rbob_bars = FutureHistoricalBarsStream("RB", "ONE_MIN", "2023-01-01T00:00:00", "2023-12-31T23:59:59")
expanded_bars = DateTimeExpanderStream(rbob_bars, "ts_event")
settlement_window = FilterStream(expanded_bars, "(ts_event_hour = 14 and ts_event_minute >= 28) or (ts_event_hour = 14 and ts_event_minute < 30)")
typical_settlement_volume = GroupByStream(settlement_window, "ts_event_date", "volume:Sum")
result = ProjectStream(typical_settlement_volume, "ts_event_date, volume_sum:daily_settlement_volume")
```
#### **Examples:**
---
**DSL:** `r = DateTimeExpanderStream(FutureHistoricalBars("ES", "ONE_DAY"), “ts_event”)`
**Input Schema:**
### Example 5 — How much SP Futures during the settlement window, average per day, during the 2 week windows before future expiration, in 2023?
| id | ts\_event | contract | open | high | low | close | volume |
| :--- | :-------- | :------- | :--- | :--- | :--- | :---- | :----- |
#### DSL Solution
**Output Schema:**
```
sp_bars = FutureHistoricalBarsStream("ES", "ONE_MIN", "2023-01-01T00:00:00", "2023-12-31T23:59:59")
expanded_bars = DateTimeExpanderStream(sp_bars, "ts_event")
settlement_window = FilterStream(expanded_bars, "(ts_event_hour = 15 and ts_event_minute >= 58) or (ts_event_hour = 16 and ts_event_minute < 1)")
filtered_bars = FilterStream(settlement_window, "(ts_event_date >= '2023-03-03' and ts_event_date < '2023-03-17') or (ts_event_date >= '2023-06-02' and ts_event_date < '2023-06-16') or (ts_event_date >= '2023-09-01' and ts_event_date < '2023-09-15') or (ts_event_date >= '2023-12-01' and ts_event_date < '2023-12-15')")
daily_volume = GroupByStream(filtered_bars, "ts_event_date", "volume:Sum")
average_daily_volume = GroupByStream(daily_volume, "", "volume_sum:Mean")
result = ProjectStream(average_daily_volume, "volume_sum_mean:average_daily_volume")
```
| id | ts\_event | contract | open | high | low | close | volume | | | | | | | |
| :--- | :-------- | :------- | :--- | :--- | :--- | :---- | :----- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
---
### Example 6 — Find the difference between the number of days in 2023 that SP 500 went up 2% and when it went down 2%
#### DSL Solution
```
bars = FutureHistoricalBarsStream("ES", "ONE_DAY", "2023-01-01T00:00:00", "2023-12-31T23:59:59")
percentage_change = SingleRowMathStream(bars, "percent_change = 100 * (close - open) / open")
days_up_2_percent = FilterStream(percentage_change, "percent_change >= 2")
days_up_count = CountStream(days_up_2_percent)
days_down_2_percent = FilterStream(percentage_change, "percent_change <= -2")
days_down_count = CountStream(days_down_2_percent)
joined_counts = JoinStream(days_up_count, "", days_down_count, "")
difference_result = SingleRowMathStream(joined_counts, "difference = days_up_count - days_down_count")
result = ProjectStream(difference_result, "difference")
```
---
### Example 7 — Monitor which CL contracts (CL) are trading with bid size \>= 1000, and record the last 1000 events\*
#### DSL Solution
```
bbo = LiveBBOStream("CL*")
large_bid = FilterStream(bbo, "bid_size >= 1000")
large_bid_ts = TimeSeriesStream(large_bid, 1000)
result = ProjectStream(large_bid_ts, "timestamp, ts_event, contract, bid_size, bid_price, ask_price")
```
---
### Example 8 — Identify FutureSpreads in CrudeOil and monitor their spread width
#### DSL Solution
```
bbo = LiveBBOStream("CL*")
expanded_bbo = ContractExpanderStream(bbo, "contract")
future_spreads = FilterStream(expanded_bbo, "contract_type = 'FutureSpread'")
spread_stream = SingleRowMathStream(future_spreads, "spread = ask_price - bid_price")
result = ProjectStream(spread_stream, "ts_event, contract, spread")
```
---
### Example 9 — Track when bid prices in NaturalGas Futures cross above 3.00 and keep only the last 10 seconds of such events
#### DSL Solution
```
bbo = LiveBBOStream("NG")
high_bid = FilterStream(bbo, "bid_price > 3.00")
high_bid_ts = TimeSeriesStream(high_bid, 10000)
recent_high_bid = TimeBasedRollingFilterStream(high_bid_ts, 10)
result = ProjectStream(recent_high_bid, "ts_event, contract, bid_price, bid_size, ask_price")
```
---
### Example 11 — Detect sequences in NaturalGas future where bid\_price increased 3 times consecutively within a rolling 30-second window
#### DSL Solution
```
bbo = LiveBBOStream("NG")
bbo_ts = TimeSeriesStream(bbo, 10000)
recent_bbo = TimeBasedRollingFilterStream(bbo, 30) // 30-second window of updates
result = PythonDataframeStream(recent_bbo, "
def transform(df):
import pandas as pd
df = df.sort_values('ts_event')
df['bid_increase'] = df['bid_price'].diff() > 0
df['consec_increase'] = df['bid_increase'].rolling(window=3).sum()
# Find rows where we just reached 3 consecutive increases
df['trigger'] = (df['consec_increase'] == 3) & (df['consec_increase'].shift(1) < 3)
return df[df['trigger']][['ts_event', 'contract', 'bid_price', 'bid_size']]
")
```
---
### Example 12 — When we get a 5+ level bid sweep in SP 500, show me how the bid behaves in the 10 minutes after — mean, stddev, min/max bid\_price
#### DSL Solution
```
bbo = LiveBBOStream("ES")
bbo_ts = TimeSeriesStream(bbo, 10000)
recent_bbo = TimeSeriesStream(bbo, 5000) // keep large enough window for event tracking
result = PythonDataframeStream(recent_bbo, "
def transform(df):
import pandas as pd
df = df.sort_values('ts_event')
rows = []
in_measurement = False
measurement_start_time = None
measurement_end_time = None
bid_prices_in_window = []
min_bid_size_threshold = 100 # defines what we consider 'stable large bid'
min_levels_stable = 5
prior_bid_sizes = []
for i, row in df.iterrows():
bid_price = row['bid_price']
bid_size = row['bid_size']
ts = pd.to_datetime(row['ts_event'])
# Maintain rolling window of prior bid_sizes
prior_bid_sizes.append(bid_size)
if len(prior_bid_sizes) > min_levels_stable:
prior_bid_sizes.pop(0)
if not in_measurement:
# Detect sweep trigger
if all(x > min_bid_size_threshold for x in prior_bid_sizes[:-1]) and bid_size < min_bid_size_threshold:
# Sweep detected → start measurement window
measurement_start_time = ts
measurement_end_time = measurement_start_time + pd.Timedelta(minutes=10)
bid_prices_in_window = []
in_measurement = True
else:
# Inside measurement window
if ts <= measurement_end_time:
bid_prices_in_window.append(bid_price)
else:
# Window complete → emit stats
if bid_prices_in_window:
mean_bid = pd.Series(bid_prices_in_window).mean()
stddev_bid = pd.Series(bid_prices_in_window).std()
min_bid = pd.Series(bid_prices_in_window).min()
max_bid = pd.Series(bid_prices_in_window).max()
rows.append({
'sweep_time': measurement_start_time,
'measurement_end': measurement_end_time,
'mean_bid_price': mean_bid,
'stddev_bid_price': stddev_bid,
'min_bid_price': min_bid,
'max_bid_price': max_bid
})
# Reset state
in_measurement = False
measurement_start_time = None
measurement_end_time = None
bid_prices_in_window = []
prior_bid_sizes = []
return pd.DataFrame(rows)
")
```
*End of Section 7 — Reinforced Learning DSL Examples*
@@ -1,27 +0,0 @@
**Enter Meta Mode**
You are not learning the DSL yet.
You are reviewing the following **BlueOrbDSL Master Learning Prompt** (MLP) as a *training document for yourself*.
Your goal is to optimize it to make it easier for you — or any future LLM — to learn from it and generate DSL code with high accuracy.
Please analyze the MLP and provide:
1. **Clarity review** — is each stream definition clear, unambiguous, easy to parse?
2. **Consistency review** — do all stream definitions follow the same structure and style?
3. **Terminology check** — are key terms (parent stream, arguments, schema impact, example usage, etc.) used consistently?
4. **Structural feedback** — is the overall document structured in a way that supports learning (headings, bold, tables, code blocks)?
5. **Formatting feedback** — are important warnings / "cannot do X" constraints highlighted clearly?
6. **Missing info** — is anything missing from any stream definition that would help you understand how to use it?
7. **Generation risk spots** — any parts where you foresee yourself making common mistakes due to ambiguity?
**Do not attempt to "learn" this DSL yet.** Stay in Meta Mode and provide feedback as an *editor and optimizer*.
When I later say:
→ **Switch to Learning Mode — this is now the final BlueOrbDSL MLP** → you will forget your Meta comments and treat the MLP as the single authoritative source for DSL generation.
+2 -12
View File
@@ -1,7 +1,6 @@
ARG IMAGE="sparkd"
ARG CONFIG_DIR="src/app_sparkd/configurations/dev"
ARG SOURCE_DIR="install/sparkd-release-gcc"
ARG PYTHON_REQUIREMENTS="requirements.txt"
# sparkd_base
FROM bluetradingsystems/linux_dotnet7 AS sparkd_base
@@ -11,11 +10,7 @@ ARG SOURCE_DIR
USER "root"
RUN apt-get update && apt-get install -y liblttng-ust-dev
RUN apt-get update && apt-get install -y libpcap0.8-dev
RUN apt-get update && apt-get install -y python3-dev
RUN apt-get update && apt-get install -y python3-pip
RUN python3 -m pip install --upgrade pip
COPY "${PYTHON_REQUIREMENTS}" "/home/bts/bin/requirements.txt"
RUN pip3 install -r "/home/bts/bin/requirements.txt"
RUN apt-get update && apt-get install -y python3.10-dev
USER "bts"
COPY --chown=bts:bts "${SOURCE_DIR}" "."
@@ -33,16 +28,11 @@ COPY --chown=bts:bts "${MD_RECORDING_DIR}" "/home/bts/sparkdata/data/md_recordin
# sparkd_symbols - like sparkd, but doesn't copy a config dir into the image
FROM bluetradingsystems/linux_dotnet7 AS sparkd_symbols
ARG SOURCE_DIR
ARG PYTHON_REQUIREMENTS="requirements.txt"
USER "root"
RUN apt-get update && apt-get install -y liblttng-ust-dev
RUN apt-get update && apt-get install -y libpcap0.8-dev
RUN apt-get update && apt-get install -y python3-dev
RUN apt-get update && apt-get install -y python3-pip
RUN python3 -m pip install --upgrade pip
COPY "${PYTHON_REQUIREMENTS}" "/home/bts/bin/requirements.txt"
RUN pip3 install -r "/home/bts/bin/requirements.txt"
RUN apt-get update && apt-get install -y python3.10-dev
USER "bts"
COPY --chown=bts:bts "${SOURCE_DIR}" "."
@@ -1,6 +0,0 @@
SPARKD__CONFIG_DIR=src/app_sparkd/configurations/alphaworx_zen_hk_live-preview
SPARKD__PYTHON_REQUIREMENTS=src/app_sparkd/configurations/alphaworx_zen_hk_live-preview/requirements.txt
SPARKD__SHM_SIZE=8gb
SPARKD__MEMORY=8gb
SPARKD__IMAGE=sparkd_symbols
SPARKD__RUNTIME_CONF_DIR=/home/bts/8000/sparkdata/conf
@@ -1,6 +0,0 @@
* Connects to Binance, Okx, and Bybit PRODUCTION sites with small accounts designated for test trades
* Api keys NOT included
* Configured for Alphaworx and Bts testing in a production environment
To retrieve api keys, run "python init_env.py". It should dump them to a file
called ".env.local".
@@ -1,9 +0,0 @@
{
"api_key": "${BINANCE_API_KEY}",
"secret_key": "${BINANCE_SECRET_KEY}",
"account": "PROD",
"rest_endpoint": "https://api.binance.com",
"ws_endpoint": "wss://ws-api.binance.com:443/ws-api/v3",
"ws_stream_endpoint": "wss://stream.binance.com:9443",
"use_cross_margin_api": true
}
@@ -1,9 +0,0 @@
{
"api_key": "${BINANCE_API_KEY}",
"secret_key": "${BINANCE_SECRET_KEY}",
"account": "PROD",
"rest_endpoint": "https://fapi.binance.com",
"ws_endpoint": "wss://ws-fapi.binance.com/ws-fapi/v1",
"ws_stream_endpoint": "wss://fstream.binance.com",
"use_cross_margin_api": false
}
@@ -1,99 +0,0 @@
import botocore
import json
import os
from boto3.session import Session
from botocore.exceptions import ClientError
from fileinput import FileInput
def update_env_file(path, env_vars: dict):
"""Update or add vars in a .env file in place."""
if not os.path.exists(path):
with open(path, "w") as file:
for k, v in env_vars.items():
file.write(f"{k}={v}\n")
return
with FileInput(path, inplace=True) as file:
for line in file:
if not line.startswith("#"):
stripped: str = line.lstrip()
key_len = stripped.find("=")
if key_len >= 0:
k = stripped[0:key_len]
if k in env_vars:
print(f"{k}={env_vars[k]}")
del env_vars[k]
continue
print(line, end="")
if len(env_vars) > 0:
with open(path, "a") as file:
for k, v in env_vars.items():
file.write(f"{k}={v}\n")
def get_secret():
secret_name = "alphaworx_api_keys"
region_name = "us-east-1"
profile_name = "bts-ap-northeast-1"
session = Session(profile_name=profile_name)
client = session.client(
service_name="secretsmanager",
region_name=region_name,
)
def on_token_error(e: Exception):
print(f"""
Failed to get secret value. {e}
Verify you are logged in to aws, then try again.
aws sso login --profile {profile_name}
If this error persists, verify you have access to these api keys.
For more help or information, talk to John.
""")
try:
get_secret_value_response = client.get_secret_value(
SecretId=secret_name)
except ClientError as e:
print(f"Failed to get secret value.\n{e}")
raise e
except botocore.exceptions.SSOTokenLoadError as tl_err:
on_token_error(tl_err)
return None
except botocore.exceptions.TokenRetrievalError as tr_err:
on_token_error(tr_err)
return None
return get_secret_value_response["SecretString"]
def _run():
this_dir = os.path.dirname(__file__)
secret_str = get_secret()
if secret_str is None:
exit(1)
secret = json.loads(secret_str)
env_vars = {}
def set_env_var(key_name: str, *json_path):
try:
val = secret[json_path[0]]
for p in json_path[1:]:
val = val[p]
env_vars[key_name] = val
print(f"Set {key_name}={val} ({'.'.join(json_path)})")
except KeyError:
print(f"Failed to set {key_name}= ({'.'.join(json_path)})")
set_env_var("BINANCE_API_KEY", "binance", "prod", "api_key")
set_env_var("BINANCE_SECRET_KEY", "binance", "prod", "secret_key")
set_env_var("BYBIT_API_KEY", "bybit", "prod_test", "api_key")
set_env_var("BYBIT_SECRET_KEY", "bybit", "prod_test", "secret_key")
update_env_file(f"{this_dir}/.env.local", env_vars)
if __name__ == "__main__":
_run()
@@ -1,15 +0,0 @@
{
"gateway_type": "bybit_execution_gateway.BybitExecutionGateway",
"account": "BYBIT_DEV",
"module_path": [
"python_interop_plugins"
],
"venue": "VENUE_BYBIT",
"api_key": "${BYBIT_API_KEY}",
"secret_key": "${BYBIT_SECRET_KEY}",
"use_testnet": true,
"logging": {
"console_level": "DEBUG",
"file_level": "DEBUG"
}
}
@@ -1,17 +0,0 @@
{
"currencies": [
"USDT"
],
"futures": [
"ANIME",
"BTC",
"ETH",
"MASK",
"RESOLV"
],
"use_testnet": true,
"logging": {
"console_level": "WARNING",
"file_level": "INFO"
}
}
@@ -1,11 +0,0 @@
certifi==2025.4.26
charset-normalizer==3.4.2
dotenv==0.9.9
idna==3.10
numpy==2.2.6
pybit==5.11.0
pycryptodome==3.23.0
python-dotenv==1.1.0
requests==2.32.4
urllib3==2.4.0
websocket-client==1.8.0
@@ -1,138 +0,0 @@
QuickbusCfg {
value {
node {
reconnect_period_secs: 1
hb_period_msecs: 15000
max_missed_hbs: 5
parent_host: ""
parent_port: 0
my_port: 0
}
client {
reconnect_period_secs: 1
hb_period_msecs: 15000
max_missed_hbs: 5
parent_host: @SystemConfig.config_ip
parent_port: @Ports.qbus_port
my_port: 0
}
}
}
JetFIX {
value {
# connection {
# name: "binance_1"
# venue: VENUE_BINANCE
# max_order_msgs_per_window: 300
# window_duration_msecs: 1000
# }
# connection {
# name: "binance_usds_futures_1"
# venue: VENUE_BINANCE_USDS_FUTURES
# max_order_msgs_per_window: 300
# window_duration_msecs: 1000
# }
connection {
name: "bybit_1"
venue: VENUE_BYBIT
max_order_msgs_per_window: 300
window_duration_msecs: 1000
}
# connection {
# name: "okx_1"
# venue: VENUE_OKX
# max_order_msgs_per_window: 300
# window_duration_msecs: 1000
# }
}
}
MarketDataConfig {
value {
# binance {
# num_depth_levels: 5
# url_rest_base: "https://api.binance.com"
# url_ws_base: "wss://stream.binance.com:9443"
# security_criteria {
# clauses {
# security_types: SECURITY_TYPE_SPOT
# symbols: @SymbolLists.list[Binance].symbol
# name_regexps: "USDC|USDT$"
# }
# }
# }
# binance_usds_futures {
# num_depth_levels: 5
# url_rest_base: "https://fapi.binance.com"
# url_ws_base: "wss://fstream.binance.com"
# security_criteria {
# clauses {
# security_types: SECURITY_TYPE_FUTURE
# symbols: @SymbolLists.list[BinanceUsdsFutures].symbol
# }
# }
# }
# okx {
# num_depth_levels: 5
# url_rest_base: "https://www.okx.com"
# url_ws_base: "wss://ws.okx.com:8443"
# security_criteria {
# clauses {
# security_types: SECURITY_TYPE_SPOT
# symbols: @SymbolLists.list[OkxSpots].symbol
# }
# clauses {
# security_types: SECURITY_TYPE_FUTURE
# symbols: @SymbolLists.list[OkxFutures].symbol
# }
# }
# is_application_ping_pong_enabled: true
# application_ping_interval_ms: 20000
# application_pong_timeout_ms: 10000
# is_protocol_ping_pong_enabled: true
# protocol_ping_interval_ms: 60000
# protocol_pong_timeout_ms: 30000
# }
python_md_providers {
venue: VENUE_BYBIT
module_path: "python_interop_plugins"
provider_type: "bybit_md_provider.BybitMdProvider"
}
}
}
NotSimExecution {
value {
# entries {
# venue: VENUE_BINANCE
# }
# entries {
# venue: VENUE_BINANCE_USDS_FUTURES
# }
entries {
venue: VENUE_BYBIT
}
# entries {
# venue: VENUE_OKX
# }
}
}
# OkxExeCfg {
# value {
# account: "PROD"
# ops_id: "op_id_bts"
# secret_key: "240038D7A28D7F9461247CFCB8E43710"
# api_key: "dc188532-c74c-4d40-affe-3c6bf4d0ca9a"
# passphrase: ":,7i%tA%ivfj%SCbX?Ut"
# ws_url: "wss://ws.okx.com:8443"
# rest_url: "https://www.okx.com"
# ws_ping_timeout_seconds: 10
# ws_ping_interval_seconds: 10
# ws_reconnect_interval_seconds: 5
# ws_reconnect_max_count: 5
# simulated_trading: false
# tdmode: "cross"
# }
# }
@@ -1,66 +0,0 @@
SymbolLists {
value {
list {
key: "Binance"
symbol: "ANIME"
symbol: "BTC"
symbol: "ETH"
symbol: "MASK"
symbol: "RESOLV"
}
list {
key: "BinanceUsdsFutures"
symbol: "ANIME"
symbol: "BTC"
symbol: "ETH"
symbol: "MASK"
symbol: "RESOLV"
}
list {
key: "OkxSpots"
symbol: "ANIME"
symbol: "BTC"
symbol: "ETH"
symbol: "MASK"
symbol: "RESOLV"
}
list {
key: "Bybit"
symbol: "ANIME"
symbol: "BTC"
symbol: "ETH"
symbol: "MASK"
symbol: "RESOLV"
}
list {
key: "OkxFutures"
symbol: "ANIME"
symbol: "BTC"
symbol: "ETH"
symbol: "MASK"
symbol: "RESOLV"
}
}
}
Symbols {
value {
# venues {
# name: "VENUE_BINANCE"
# spot: @SymbolLists.list[Binance].symbol
# }
# venues {
# name: "VENUE_BINANCE_USDS_FUTURES"
# future: @SymbolLists.list[BinanceUsdsFutures].symbol
# }
venues {
name: "VENUE_BYBIT"
future: @SymbolLists.list[Bybit].symbol
}
# venues {
# name: "VENUE_OKX"
# spot: @SymbolLists.list[OkxSpots].symbol
# future: @SymbolLists.list[OkxFutures].symbol
# }
}
}
@@ -35,8 +35,8 @@ def update_env_file(path, env_vars: dict):
def get_secret():
secret_name = "alphaworx_api_keys"
region_name = "us-east-1"
secret_name = "alphaworx-binance-api-keys"
region_name = "ap-northeast-1"
profile_name = "bts-ap-northeast-1"
session = Session(profile_name=profile_name)
client = session.client(
@@ -77,19 +77,10 @@ def _run():
exit(1)
secret = json.loads(secret_str)
env_vars = {}
def set_env_var(key_name: str, *json_path):
try:
val = secret[json_path[0]]
for p in json_path[1:]:
val = val[p]
env_vars[key_name] = val
print(f"Set {key_name}={val} ({'.'.join(json_path)})")
except KeyError:
print(f"Failed to set {key_name}= ({'.'.join(json_path)})")
set_env_var("BINANCE_API_KEY", "binance", "prod", "api_key")
set_env_var("BINANCE_SECRET_KEY", "binance", "prod", "secret_key")
if "api-key" in secret:
env_vars["BINANCE_API_KEY"] = secret["api-key"]
if "secret-key" in secret:
env_vars["BINANCE_SECRET_KEY"] = secret["secret-key"]
update_env_file(f"{this_dir}/.env.local", env_vars)
+1 -1
View File
@@ -1 +1 @@
SPARKD__CONFIG_DIR=src/app_sparkd/configurations/bybit_dev
SPARKD__CONFIG_DIR=src/app_sparkd/configurations/py_dummy_md
@@ -1,18 +1,12 @@
{
"currencies": [
"USDT"
],
"futures": [
"ADA",
"BNB",
"BTC",
"DUCK",
"ETH",
"MILK",
"PRIME",
"XRP"
"BTCUSDT",
"ETHUSDT",
"BNBUSDT",
"XRPUSDT",
"ADAUSDT",
"PRIMEUSDT"
],
"use_testnet": true,
"logging": {
"console_level": "WARNING",
"file_level": "INFO"
@@ -2,14 +2,12 @@ SymbolLists {
value {
list {
key: "Bybit"
symbol: "ADA"
symbol: "BNB"
symbol: "BTC"
symbol: "DUCK"
symbol: "ETH"
symbol: "MILK"
symbol: "PRIME"
symbol: "BNB"
symbol: "XRP"
symbol: "ADA"
symbol: "PRIME"
}
}
}
@@ -1,45 +0,0 @@
{
"api_key": "e4a0c6b198549042b13faa7645640d60",
"secret_key": "527c37877efc247bb0b6158cf3d5d18b84db720da894307f1e2ab88df3aaf0ee",
"ws_url": "wss://fx-ws-testnet.gateio.ws/v4/ws/usdt",
"currencies": [
"USDT"
],
"spots": [
"AMP",
"AR",
"ARB",
"ATOM",
"BTC",
"CAKE",
"CELO",
"CRV",
"ETH",
"FIL",
"GMT",
"GMX",
"HIGH",
"ICP",
"IMX",
"IOTX",
"LRC",
"MANA",
"MINA",
"NEAR",
"NOT",
"OMG",
"OP",
"OXY",
"PEOPLE",
"TURBO",
"VET",
"WAVES",
"YFI",
"ZIL",
"ZRX"
],
"logging": {
"console_level": "WARNING",
"file_level": "INFO"
}
}
@@ -2,37 +2,37 @@
value {
list {
key: "Gateio"
symbol: "AMP"
symbol: "AR"
symbol: "ARB"
symbol: "ATOM"
symbol: "BTC"
symbol: "CAKE"
symbol: "CAKE"
symbol: "ETH"
symbol: "MANA"
symbol: "OMG"
symbol: "GMX"
symbol: "ICP"
symbol: "LRC"
symbol: "MINA"
symbol: "ZIL"
symbol: "TURBO"
symbol: "OP"
symbol: "ZRX"
symbol: "VET"
symbol: "AR"
symbol: "IMX"
symbol: "HIGH"
symbol: "ARB"
symbol: "YFI"
symbol: "NEAR"
symbol: "OXY"
symbol: "WAVES"
symbol: "AMP"
symbol: "CELO"
symbol: "CRV"
symbol: "ETH"
symbol: "FIL"
symbol: "GMT"
symbol: "GMX"
symbol: "HIGH"
symbol: "ICP"
symbol: "IMX"
symbol: "IOTX"
symbol: "LRC"
symbol: "MANA"
symbol: "MINA"
symbol: "NEAR"
symbol: "NOT"
symbol: "OMG"
symbol: "OP"
symbol: "OXY"
symbol: "PEOPLE"
symbol: "TURBO"
symbol: "VET"
symbol: "WAVES"
symbol: "YFI"
symbol: "ZIL"
symbol: "ZRX"
symbol: "IOTX"
symbol: "ATOM"
symbol: "NOT"
symbol: "GMT"
}
}
}
@@ -254,7 +254,7 @@ struct PyExecutionGatewayAdapter::PImpl {
auto orig_signal_handler = std::signal(SIGINT, SIG_DFL);
std::future<shared_ptr<PythonObjectT<ExecutionGateway>>> gw_fut =
plugin_man->start_execution_gateway(
name, config.gateway_type, bridge_ptr);
name, config.gateway_type, config.module_path, bridge_ptr);
gw_fut.wait();
std::signal(SIGINT, orig_signal_handler);
bridge.run_py(&py_keep_going, gw_fut.get(), [&](auto callback) {
@@ -362,7 +362,6 @@ struct PyExecutionGatewayAdapter::PImpl {
bts_assert(tx_id.size() < er.exchange_fill_id.value().max_size());
bts_assert(!py_er.transact_id().empty());
er.exchange_fill_id->copy_from(py_er.transact_id());
er.msg = py_er.error_msg();
}
void to_py(
@@ -443,8 +442,6 @@ struct PyExecutionGatewayAdapter::PImpl {
py_req.set_key(msg.key.value());
py_req.set_exchange_security_id(msg.security_name());
py_req.set_venue(bts::api::venue_name_pretty(msg.venue()));
py_req.set_security_type(
bts::api::security_type_name_pretty(msg.security_type()));
py_req.set_is_buy(msg.side == bts::api::SIDE_BUY);
py_req.set_price(msg.price.value());
py_req.set_quantity(msg.quantity.full_value_float());
@@ -463,8 +460,6 @@ struct PyExecutionGatewayAdapter::PImpl {
std::to_string(msg.exchange_order_id.value()));
py_req.set_exchange_security_id(cf.security_name());
py_req.set_venue(bts::api::venue_name_pretty(cf.venue()));
py_req.set_security_type(
bts::api::security_type_name_pretty(cf.security_type()));
py_req.set_is_buy(cf.side == bts::api::SIDE_BUY);
py_req.set_price(cf.price.value());
py_req.set_quantity(cf.quantity.full_value_float());
@@ -476,8 +471,6 @@ struct PyExecutionGatewayAdapter::PImpl {
std::to_string(msg.exchange_order_id.value()));
py_req.set_exchange_security_id(msg.security_name());
py_req.set_venue(bts::api::venue_name_pretty(msg.venue()));
py_req.set_security_type(
bts::api::security_type_name_pretty(msg.security_type()));
py_req.set_is_buy(msg.side == bts::api::SIDE_BUY);
}
@@ -203,7 +203,9 @@ struct PyMarketDataSubscription::PImpl {
std::future<std::shared_ptr<
python_interop::PythonObjectT<bts::md::py_interop::MdProvider>>>
md_pro_fut = py_plugin_man->start_md_provider(
name, config.provider_type(), bridge_ptr);
name, config.provider_type(),
{config.module_path().begin(), config.module_path().end()},
bridge_ptr);
md_pro_fut.wait();
INFO_TRACE_PARAMS(
"run_py",
@@ -336,10 +338,6 @@ struct PyMarketDataSubscription::PImpl {
sec_comm.min_quantity =
std::max(qty_info.min_value(), sec_comm.min_quantity_increment);
}
if (spec.notional_quantity_infos().size() > 0) {
const auto &notional_qty_info = spec.notional_quantity_infos()[0];
sec_comm.min_notional_value = notional_qty_info.min_value();
}
sec_comm.desc = py_sec.name();
if (py_sec.is_spot()) {
sec.security = SpotSecurity{};
@@ -7,8 +7,6 @@
#include "spark/md/GeneralMarketDataListener.h"
#include "spark/md/GeneralSecurityIndex.h"
#include "spark/md/MDProducer.h"
#include "spark/md/SymbolLoader.h"
#include "spark/md/UserSymbol.h"
#include "spark/persistence/IDGenerator.h"
#include "sparkd/md/binance/BinanceSubscription.h"
#include "sparkd/md/binance_us/BinanceUsSubscription.h"
@@ -84,23 +82,9 @@ CCApiMarketDataProvider::get_sources() {
"CCApiMarketDataProvider::get_sources failed. Invalid config: {}",
fmt::to_string(fmt::join(cfg_errs, "\n")));
}
std::set<UserSymbol> syms;
auto is_venue_enabled = [&syms](api::domain::Venue venue) {
syms.clear();
SymbolLoader::load(syms, domain::SecurityCriteria::create({venue}));
if (syms.empty()) {
LOG(md, Warn,
R"([ccapi_md_pro_disabled.json]{{ "venue": "{}", "rsn": "No symbols config" }})",
venue);
return false;
}
return true;
};
// Binance
for (const auto &binance : cfg.binance()) {
if (!is_venue_enabled(api::domain::VENUE_BINANCE)) {
continue;
}
me_->sources.emplace_back(
api::domain::VENUE_BINANCE,
std::make_shared<binance::BinanceSubscription>(
@@ -108,9 +92,6 @@ CCApiMarketDataProvider::get_sources() {
}
// Binance US
for (const auto &binance_us : cfg.binance_us()) {
if (!is_venue_enabled(api::domain::VENUE_BINANCE_US)) {
continue;
}
me_->sources.emplace_back(
api::domain::VENUE_BINANCE_US,
std::make_shared<binance_us::BinanceUsSubscription>(
@@ -118,9 +99,6 @@ CCApiMarketDataProvider::get_sources() {
}
// Binance USDS Futures
for (const auto &binance_ufs : cfg.binance_usds_futures()) {
if (!is_venue_enabled(api::domain::VENUE_BINANCE_USDS_FUTURES)) {
continue;
}
me_->sources.emplace_back(
api::domain::VENUE_BINANCE_USDS_FUTURES,
std::make_shared<
@@ -129,9 +107,6 @@ CCApiMarketDataProvider::get_sources() {
}
// Coinbase
for (const auto &coinbase : cfg.coinbase()) {
if (!is_venue_enabled(api::domain::VENUE_COINBASE)) {
continue;
}
me_->sources.emplace_back(
api::domain::VENUE_COINBASE,
std::make_shared<coinbase::CoinbaseSubscription>(
@@ -139,9 +114,6 @@ CCApiMarketDataProvider::get_sources() {
}
// Deribit
for (const auto &deribit : cfg.deribit()) {
if (!is_venue_enabled(api::domain::VENUE_DERIBIT)) {
continue;
}
me_->sources.emplace_back(
api::domain::VENUE_DERIBIT,
std::make_shared<deribit::DeribitSubscription>(
@@ -149,9 +121,6 @@ CCApiMarketDataProvider::get_sources() {
}
// FTX
for (const auto &ftx : cfg.ftx()) {
if (!is_venue_enabled(api::domain::VENUE_FTX)) {
continue;
}
me_->sources.emplace_back(
api::domain::VENUE_FTX,
std::make_shared<ftx::FTXSubscription>(
@@ -159,9 +128,6 @@ CCApiMarketDataProvider::get_sources() {
}
// Okx
for (const auto &okx : cfg.okx()) {
if (!is_venue_enabled(api::domain::VENUE_OKX)) {
continue;
}
me_->sources.emplace_back(
api::domain::VENUE_OKX,
std::make_shared<okx::OkxSubscription>(
+2 -2
View File
@@ -110,8 +110,8 @@ namespace bts {
ConcurrentDictionary<SecurityKey, SecurityMD> exch_secs,
Dictionary<SecurityKey, ExecutionStrategyImmutable> synth_strategies
) {
last_synth_strategies.Where(kv => kv.Value.Deleted).difference_hard(
synth_strategies.Where(kv => kv.Value.Deleted),
synth_strategies.Where(kv => kv.Value.Deleted).difference_hard(
last_synth_strategies.Where(kv => kv.Value.Deleted),
out var deleted_synths,
out var undeleted_synths,
synth_id_comparer);
@@ -38,9 +38,6 @@ namespace mbts.spark {
Vega,
Theta,
Rho,
Currency,
CashPerPoint,
MinOrderQuantity,
MinOrderNotional
Currency
}
}
@@ -128,12 +128,6 @@ namespace bts.shell.market_view {
public DoubleMinMaxState theta { get; set; }
[ProtoMember(36)]
public DoubleMinMaxState rho { get; set; }
[ProtoMember(39)]
public DoubleMinMaxState cash_per_point { get; set; }
[ProtoMember(40)]
public DoubleMinMaxState min_order_quantity { get; set; }
[ProtoMember(41)]
public DoubleMinMaxState min_order_notional { get; set; }
public Func<MarketViewVM.Row, bool> build_predicate() {
var pb = new ColumnPredicateBuilder<MarketViewVM.Row>();
@@ -191,9 +185,6 @@ namespace bts.shell.market_view {
pb.add_numeric(vega, r => r.vega);
pb.add_numeric(theta, r => r.theta);
pb.add_numeric(rho, r => r.rho);
pb.add_numeric(cash_per_point, r => r.cash_per_point);
pb.add_numeric(min_order_quantity, r => r.min_order_quantity);
pb.add_numeric(min_order_notional, r => r.min_order_notional);
return pb.to_predicate();
}
}
@@ -209,13 +209,6 @@ namespace mbts.spark {
public bts.api.Venue venue => security_md.security.Exchange;
public uint security_key => security_md.security.Id;
public string currency => security_md.security.Currency;
public double cash_per_point => security_md.security.ContractSpec.CashPerPoint;
public double min_order_quantity => _spec.QuantityInfos.TryGetFirst(out var qi)
? qi.TickUnit.safe_divide(qi.TickBase)
: security_md.security.tick_factors().tick_unit_frac();
public double min_order_notional => _spec.NotionalQuantityInfos.TryGetFirst(out var qi)
? qi.MinValue
: security_md.security.tick_factors().tick_unit_frac();
public bool inside_bids => _inside_bids.value;
public bool inside_asks => _inside_asks.value;
@@ -351,9 +344,6 @@ namespace mbts.spark {
TickDisplaySettingsImmutable _qty_display =>
security_md.quantity_display;
ContractSpecificationImmutable _spec =>
security_md.security.spec_or_default();
public string format_price(double price) =>
security.ContractSpec.format(price);
@@ -457,9 +447,9 @@ namespace mbts.spark {
_bus.post(SparkTags.ShowOrderTicketTag, get_params(last_price, last_qty));
break;
}
OrderTicketParams get_params(double price, double quantity = 0) =>
new(new(new() { SecurityKey = security_key }), price, quantity);
new(new(new() {SecurityKey = security_key }), price, quantity);
}
void _refresh_orders(ref bool dirty) {
@@ -146,9 +146,6 @@ namespace bts.shell.market_view {
public static GridColumnSettings vega => _mk(Columns.Vega, "Vega", ColWidth.Greeks, false);
public static GridColumnSettings theta => _mk(Columns.Theta, "Theta", ColWidth.Greeks, false);
public static GridColumnSettings rho => _mk(Columns.Rho, "Rho", ColWidth.Greeks, false);
public static GridColumnSettings cash_per_point => _mk(Columns.CashPerPoint, "CPP", ColWidth.Greeks, false);
public static GridColumnSettings min_order_quantity => _mk(Columns.MinOrderQuantity, "Min Order Qty", 60, false);
public static GridColumnSettings min_notional_value => _mk(Columns.MinOrderNotional, "Min Order Notional", 60, false);
public static IEnumerable<SortingCriteria> default_sorting_criteria {
get {
@@ -198,9 +198,6 @@ namespace mbts.spark {
add_col(ColKey.Vega, r => fmt_greek(r.vega), nameof(Row.vega), vega_filter, FilterType.DoubleMinMax, ColWidth.Greeks);
add_col(ColKey.Theta, r => fmt_greek(r.theta), nameof(Row.theta), theta_filter, FilterType.DoubleMinMax, ColWidth.Greeks);
add_col(ColKey.Rho, r => fmt_greek(r.rho), nameof(Row.rho), rho_filter, FilterType.DoubleMinMax, ColWidth.Greeks);
add_col(ColKey.CashPerPoint, r => r.cash_per_point.ToString("0.00#####"), nameof(Row.cash_per_point), cash_per_point_filter, FilterType.DoubleMinMax, ColWidth.Greeks);
add_col(ColKey.MinOrderQuantity, r => r.min_order_quantity.ToString("0.00#####"), nameof(Row.min_order_quantity), min_order_quantity_filter, FilterType.DoubleMinMax, ColWidth.Qty);
add_col(ColKey.MinOrderNotional, r => r.min_order_notional.ToString("0.00#####"), nameof(Row.min_order_notional), min_order_notional_filter, FilterType.DoubleMinMax, ColWidth.Qty);
// row vm filter
// Althoguh securities are filtered at the row 'model' level (SecurityMD), the column base filters have to be applied at the row vm level,
@@ -439,9 +436,6 @@ namespace mbts.spark {
public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> gamma_filter { get; } = new DoubleMinMax<Row>(r => r.gamma, nameof(Row.gamma));
public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> theta_filter { get; } = new DoubleMinMax<Row>(r => r.theta, nameof(Row.theta));
public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> rho_filter { get; } = new DoubleMinMax<Row>(r => r.rho, nameof(Row.rho));
public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> cash_per_point_filter { get; } = new DoubleMinMax<Row>(r => r.cash_per_point, nameof(Row.cash_per_point));
public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> min_order_quantity_filter { get; } = new DoubleMinMax<Row>(r => r.min_order_quantity, nameof(Row.min_order_quantity));
public IDataGridColumnPersistentFilter<Row, DoubleMinMaxState> min_order_notional_filter { get; } = new DoubleMinMax<Row>(r => r.min_order_notional, nameof(Row.min_order_notional));
IEnumerable<IDataGridColumnFilter<Row>> _all_filters =>
_filter_getters.Select(g => g(this));
@@ -483,9 +477,6 @@ namespace mbts.spark {
ColKey.Vega => wrap(r => r.vega),
ColKey.Theta => wrap(r => r.theta),
ColKey.Rho => wrap(r => r.rho),
ColKey.CashPerPoint => wrap(r => r.cash_per_point),
ColKey.MinOrderQuantity => wrap(r => r.min_order_quantity),
ColKey.MinOrderNotional => wrap(r => r.min_order_notional),
_ => null,
};
@@ -592,9 +583,6 @@ namespace mbts.spark {
vega = vega_filter.get_state(),
theta = theta_filter.get_state(),
rho = rho_filter.get_state(),
cash_per_point = cash_per_point_filter.get_state(),
min_order_quantity = min_order_quantity_filter.get_state(),
min_order_notional = min_order_notional_filter.get_state(),
};
public void load_filters(MarketViewFilterStates filters) {
@@ -637,7 +625,6 @@ namespace mbts.spark {
vega_filter.load_state(filters.vega);
theta_filter.load_state(filters.theta);
rho_filter.load_state(filters.rho);
cash_per_point_filter.load_state(filters.cash_per_point);
}
static (string desc, Color fg) _fills_title(ColKey key) =>
@@ -647,7 +647,7 @@ namespace mbts.spark {
volume_column = VolumeColumnSettings.Default,
calculator_buttons = new double[] { 1, 5, 10, 25, 50, 100 },
scroll_bar_visible = false,
default_time_in_force = TimeInForce.Gtc,
default_time_in_force = TimeInForce.Day,
default_algo = new AlgoID(CustomOrderDefinition.CustomOneofCase.Limit),
animate_recenter = true,
badge_style = BadgeStyle.Normal,
+2 -2
View File
@@ -129,7 +129,7 @@ namespace mbts.spark.utilities {
};
public TimeInForce default_time_in_force() =>
TimeInForce.Gtc;
TimeInForce.Day;
public TimeInForce default_time_in_force(SecurityImmutable security) =>
default_time_in_force();
@@ -167,7 +167,7 @@ namespace mbts.spark.utilities {
var rv = new OrderDefinition {
SecurityKey = security.Id,
Side = Side.Buy,
TimeInForce = default_time_in_force(security),
TimeInForce = TimeInForce.Day,
CustomDef = OrderDataHelper.Defaults.CustomOrderDefinition(),
CtiCode = CTICode.CtiCode1,
CustomerOrFirm = CustomerOrFirm.Firm,
+1 -1
View File
@@ -30,7 +30,7 @@
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="Google.Protobuf" Version="3.24.3" />
<PackageReference Include="IronPython" Version="3.4.2" />
<PackageReference Include="IronPython" Version="3.4.1" />
<PackageReference Include="MathNet.Numerics" Version="5.0.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.2.0" />
<PackageReference Include="NAudio" Version="2.0.1" />
@@ -15,10 +15,6 @@ namespace bts.md {
implied_size = implied_sz;
price = prx;
}
public PriceSize(Size sz, double prx)
: this(sz, sz, 0, prx) { }
public override string ToString() {
return $"{price} {size} {implied_size} ";
}
@@ -1,184 +0,0 @@
#nullable enable
namespace Spark.Api.MarketData;
using bts;
using bts.api;
using bts.md;
using bts.utils;
using System;
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.Linq;
public static class SyntheticBookCalculator {
public static Book GetSyntheticBook(uint securityId, IEnumerable<(ContractLegImmutable, Book)> legDepths) {
return new(
securityId,
GetSyntheticDepth(legDepths.Select(GetBidLegParams).ToList()).Select(ToPriceSize).ToList(),
GetSyntheticDepth(legDepths.Select(GetAskLegParams).ToList()).Select(ToPriceSize).ToList());
PriceSize ToPriceSize(DepthLevel level) =>
new((float)level.Quantity, level.Price);
(double Quantity, double TickSize, Book Book) GetBidLegParams((ContractLegImmutable, Book) ld) =>
(ld.Item1.signed_quantity(), ld.Item1.Security.ContractSpec.LowTick, ld.Item2);
(double Quantity, double TickSize, Book Book) GetAskLegParams((ContractLegImmutable, Book) ld) =>
(-ld.Item1.signed_quantity(), ld.Item1.Security.ContractSpec.LowTick, ld.Item2);
}
public static List<DepthLevel> GetSyntheticDepth(IEnumerable<(ContractLegImmutable, Book)> legDepths) {
return GetSyntheticDepth(
legDepths.Select(getLegDepthConsumer).ToList());
LegDepthConsumer getLegDepthConsumer((ContractLegImmutable, Book) legDepth) {
var (leg, book) = legDepth;
Log.assert(leg.Quantity.is_real(), "Leg quantity must be real");
Log.assert(leg.Security.HasContractSpec && leg.Security.ContractSpec.LowTick.is_real_positive(),
"Leg security LowTick must be real and positive");
return LegDepthConsumer.Create(leg.Quantity, leg.Security.ContractSpec.LowTick);
}
}
public static List<DepthLevel> GetSyntheticDepth(IEnumerable<(double Quantity, double TickSize, Book Book)> legDepths) {
return GetSyntheticDepth(
legDepths.Select(getLegDepthConsumer).ToList());
LegDepthConsumer getLegDepthConsumer((double Quantity, double TickSize, Book Book) legDepth) {
var (qty, tickSz, bk) = legDepth;
Log.assert(qty.is_real(), "Leg quantity must be real");
Log.assert(tickSz.is_real_positive(),
"Leg tickSz must be real and positive");
return LegDepthConsumer.Create(qty, tickSz);
}
}
public static List<DepthLevel> GetSyntheticDepth(List<LegDepthConsumer> legs) {
if (legs.Count == 0) {
return new(0);
}
var synthLeg = legs.Skip(1).Aggregate(
legs[0],
(l1, l2) => new LegDepthConsumer(new(getSyntheticDepth(l1, l2).ToList())) {
LegQuantity = 1,
QuantityTick = MathUtils.lcm(l1.QuantityTick, l2.QuantityTick),
});
return synthLeg.Depth.Levels;
}
static IEnumerable<DepthLevel> getSyntheticDepth(LegDepthConsumer leg1, LegDepthConsumer leg2) {
while (!leg1.Depth.IsDone && !leg2.Depth.IsDone) {
if (!leg1.LegQuantity.is_real() || !leg2.LegQuantity.is_real()) {
throw new ArgumentException("Invalid leg quantities");
}
double leg2PerLeg1 = (leg2.LegQuantity / leg1.LegQuantity).abs();
double leg1PerLeg2 = (leg1.LegQuantity / leg2.LegQuantity).abs();
double avail1 = leg1.Depth.AvailableQuantity(1);
double avail2 = leg2.Depth.AvailableQuantity(1);
(double take1, double take2) = (avail1 / leg1PerLeg2) < (avail2 / leg2PerLeg1)
? (avail1, avail1 * leg2PerLeg1)
: (avail2 * leg1PerLeg2, avail2);
var (qty1, cost1) = leg1.Depth.ConsumeQuantity(take1);
var (qty2, cost2) = leg2.Depth.ConsumeQuantity(take2);
Log.assert(qty2 / qty1 == leg2PerLeg1);
var price1 = cost1 / qty1;
var price2 = cost2 / qty2;
price1 *= leg1.LegQuantity;
price2 *= leg2.LegQuantity;
yield return new(Price: price1 + price2, Quantity: qty1);
}
}
public record DepthLevel(double Price, double Quantity) {
public override string ToString() {
return $"{Price} ({Quantity})";
}
}
public class DepthLevelComparer : IEqualityComparer<DepthLevel> {
readonly IEqualityComparer<double> priceComparer;
readonly IEqualityComparer<double> quantityComparer;
public DepthLevelComparer(IEqualityComparer<double>? priceComparer = null, IEqualityComparer<double>? quantityComparer = null) {
this.priceComparer = priceComparer ?? EqualityComparer<double>.Default;
this.quantityComparer = quantityComparer ?? EqualityComparer<double>.Default;
}
public bool Equals(DepthLevel? x, DepthLevel? y) {
if (x is null) {
return y is null;
}
return y is not null
&& priceComparer.Equals(x.Price, y.Price)
&& quantityComparer.Equals(x.Quantity, y.Quantity);
}
public int GetHashCode([DisallowNull] DepthLevel x) =>
HashCode.Combine(
priceComparer.GetHashCode(x.Price),
quantityComparer.GetHashCode(x.Quantity));
}
public class DepthConsumer {
public DepthConsumer() { }
public DepthConsumer(params DepthLevel[] levels) : this(levels.AsEnumerable()) { }
public DepthConsumer(IEnumerable<DepthLevel> levels) =>
Levels.AddRange(levels);
public int Pos { get; private set; } = 0;
public double CurrLevelConsumed { get; private set; } = 0;
public List<DepthLevel> Levels { get; } = new();
public bool IsDone =>
Pos == Levels.Count || (Pos == Levels.Count - 1 && Levels.Last().Quantity == CurrLevelConsumed);
public double AvailableQuantity(int levelLimit) {
int end = Math.Min(Pos + levelLimit, Levels.Count);
double rv = 0;
for (int i = Pos; i < end; i++) {
double levelQty = Levels[i].Quantity;
if (i == Pos) {
levelQty -= CurrLevelConsumed;
}
rv += levelQty;
}
return rv;
}
public (double Qty, double Price) ConsumeQuantity(double quantity) {
double consumed = 0;
double price = 0;
for (; Pos < Levels.Count; Pos++) {
double levelQty = Levels[Pos].Quantity - CurrLevelConsumed;
var rem = quantity - consumed;
if (levelQty > rem) {
CurrLevelConsumed += rem;
consumed = quantity;
price += rem * Levels[Pos].Price;
break;
}
consumed += levelQty;
price += levelQty * Levels[Pos].Price;
CurrLevelConsumed = 0;
}
return (consumed, price);
}
}
public class LegDepthConsumer {
public LegDepthConsumer() : this(new()) { }
public LegDepthConsumer(DepthConsumer depth) =>
Depth = depth;
public double LegQuantity { get; set; } = 1;
public double QuantityTick { get; set; } = 1;
public DepthConsumer Depth { get; } = new();
public static LegDepthConsumer Create(double legQuantity, double quantityTick, params DepthLevel[] levels) =>
new(new(levels)) {
LegQuantity = legQuantity,
QuantityTick = quantityTick
};
}
}
@@ -124,9 +124,8 @@ namespace bts.data.ticks {
do {
// Note that the modulus operator is not used here due to rounding errors.
// Such a rounding errror was the cause of the bug https://bluetradesys.atlassian.net/browse/SP-283.
if (MathUtils.is_int(factored_base / raw_base, MathUtils.API_DECIMAL_PRECISION)) {
return new TickFactors(factored_base, Math.Round(factored_base / raw_base));
}
if (MathUtils.is_int(factored_base / raw_base))
return new TickFactors(factored_base, factored_base / raw_base);
factored_base *= 10;
} while (++count < 10);
return null;
+3 -2
View File
@@ -122,7 +122,8 @@ namespace bts.utils {
return 0;
}
public static T lcm<T>(IEnumerable<T> nums) where T : INumber<T> {
public static T lcm<T>(IEnumerable<T> nums)
where T : INumber<T>, IMultiplyOperators<T, int, T> {
using var en = nums.Where(n => n != T.Zero).GetEnumerator();
if (!en.MoveNext()) {
return T.Zero;
@@ -134,7 +135,7 @@ namespace bts.utils {
return res;
}
public static T lcm<T>(T a, T b) where T : INumber<T> {
public static T lcm<T>(T a, T b) where T : INumber<T>, IMultiplyOperators<T, int, T> {
T num1, num2;
if (a > b) {
num1 = a;
@@ -1,96 +0,0 @@
#nullable enable
namespace bts.tests.security;
using bts.test;
using bts.utils;
using mbts;
using System;
using System.Collections.Generic;
using System.Linq;
using static Spark.Api.MarketData.SyntheticBookCalculator;
[TestGroup]
class SyntheticBookTests {
[Test]
public void TestFractionalLegs() {
var legs = new List<LegDepthConsumer>() {
LegDepthConsumer.Create(1, 0.001, new(40, 0.205), new(38, 0.195)),
LegDepthConsumer.Create(-0.5, 0.001, new(50, 0.365), new(55, 0.100)),
};
var synthBook = getSyntheticBook(legs);
var expectedBook = new List<DepthLevel>() {
new(15, 0.205),
new(13, 0.195),
};
Assert.That(synthBook.SequenceEqual(expectedBook));
legs = new List<LegDepthConsumer>() {
LegDepthConsumer.Create(1, 0.001, new(50, 0.365), new(55, 0.100)),
LegDepthConsumer.Create(-0.5, 0.001, new(40, 0.205), new(38, 0.195)),
};
synthBook = getSyntheticBook(legs);
expectedBook = new() {
new(30, 0.365),
new(35, 0.045),
new(36, 0.055),
};
var tf = new mbts.user.settings.TickFactors(1000, 1);
var priceComp = tf.tick_equality_comparer();
var qtyComp = new mbts.user.settings.TickFactors(1000000, 1).tick_equality_comparer();
var levelComp = new DepthLevelComparer(priceComp, qtyComp);
Assert.That(synthBook.SequenceEqual(expectedBook, levelComp));
legs = new List<LegDepthConsumer>() {
LegDepthConsumer.Create(1, 0.001, new(50, 0.365), new(45, 0.100)),
LegDepthConsumer.Create(1, 0.001, new(40, 0.205), new(38, 0.195)),
LegDepthConsumer.Create(-2, 0.001, new(40, 0.118), new(42.5, 0.068), new(45, 1.1)),
};
synthBook = getSyntheticBook(legs);
expectedBook = new() {
new(10, 0.059),
new(5, 0.034),
new(0, 0.112),
new(-2, 0.16),
new(-7, 0.035),
};
Assert.That(synthBook.SequenceEqual(expectedBook, levelComp));
}
static IEnumerable<DepthLevel> getSyntheticBook(LegDepthConsumer leg1, LegDepthConsumer leg2) {
while (!leg1.Depth.IsDone && !leg2.Depth.IsDone) {
if (!leg1.LegQuantity.is_real() || !leg2.LegQuantity.is_real()) {
throw new ArgumentException("Invalid leg quantities");
}
double leg2PerLeg1 = (leg2.LegQuantity / leg1.LegQuantity).abs();
double leg1PerLeg2 = (leg1.LegQuantity / leg2.LegQuantity).abs();
double avail1 = leg1.Depth.AvailableQuantity(1);
double avail2 = leg2.Depth.AvailableQuantity(1);
(double take1, double take2) = (avail1 / leg1PerLeg2) < (avail2 / leg2PerLeg1)
? (avail1, avail1 * leg2PerLeg1)
: (avail2 * leg1PerLeg2, avail2);
var (qty1, cost1) = leg1.Depth.ConsumeQuantity(take1);
var (qty2, cost2) = leg2.Depth.ConsumeQuantity(take2);
Log.assert(qty2 / qty1 == leg2PerLeg1);
var price1 = cost1 / qty1;
var price2 = cost2 / qty2;
price1 *= leg1.LegQuantity;
price2 *= leg2.LegQuantity;
yield return new(Price: price1 + price2, Quantity: qty1);
}
}
List<DepthLevel> getSyntheticBook(List<LegDepthConsumer> legs) {
if (legs.Count == 0) {
return new(0);
}
var synthLeg = legs.Skip(1).Aggregate(
legs[0],
(l1, l2) => new LegDepthConsumer(new(getSyntheticBook(l1, l2).ToList())) {
LegQuantity = 1,
QuantityTick = MathUtils.lcm(l1.QuantityTick, l2.QuantityTick),
});
return synthLeg.Depth.Levels;
}
}
@@ -424,7 +424,7 @@ namespace bts.websocket_core {
return (true, new OrderDefinition {
TimeInForce = TimeInForce.Gtc,
TimeInForce = TimeInForce.Day,
SecurityKey = sc_md.id,
Side = (Side)side,
UserId = user_id,
@@ -374,7 +374,7 @@ CrossMarginGateway::CrossMarginGateway(
std::move(jetfix_connection))) {
bts_assert(
me_->config.account.size() <=
bts::orderman::ExecutionReport{}.account_name.value().max_size());
bts::orderman::ExecutionReport{}.account_name.value().size());
}
CrossMarginGateway::~CrossMarginGateway() = default;
@@ -31,13 +31,6 @@ void PyCancelOrderRequest::set_venue(string value) {
venue_ = std::move(value);
}
const string &PyCancelOrderRequest::security_type() const {
return security_type_;
}
void PyCancelOrderRequest::set_security_type(string value) {
security_type_ = std::move(value);
}
bool PyCancelOrderRequest::is_buy() const { return is_buy_; }
void PyCancelOrderRequest::set_is_buy(bool value) {
is_buy_ = std::move(value);
@@ -48,7 +41,6 @@ void to_json(nlohmann::json &j, const PyCancelOrderRequest &req) {
j["eoid"] = req.exchange_order_id();
j["esid"] = req.exchange_security_id();
j["venue"] = req.venue();
j["styp"] = req.security_type();
j["side"] = req.is_buy() ? 'B' : 'S';
}
@@ -57,7 +49,6 @@ void from_json(const nlohmann::json &j, PyCancelOrderRequest &req) {
j.at("eoid").get_to(req.exchange_order_id_);
j.at("esid").get_to(req.exchange_security_id_);
j.at("venue").get_to(req.venue_);
j.at("styp").get_to(req.security_type_);
const auto str = j.at("side").get<string>();
if (!str.empty()) {
switch (str[0]) {
@@ -26,9 +26,6 @@ class PyCancelOrderRequest {
const std::string &venue() const;
void set_venue(std::string value);
const std::string &security_type() const;
void set_security_type(std::string value);
bool is_buy() const;
void set_is_buy(bool value);
@@ -37,7 +34,6 @@ class PyCancelOrderRequest {
std::string exchange_order_id,
std::string exchange_security_id,
std::string venue,
std::string security_type,
bool is_buy);
std::string to_string() const;
@@ -47,7 +43,6 @@ class PyCancelOrderRequest {
std::string exchange_order_id_;
std::string exchange_security_id_;
std::string venue_;
std::string security_type_;
bool is_buy_;
friend void from_json(const nlohmann::json &j, PyCancelOrderRequest &req);
@@ -78,13 +73,11 @@ inline void PyCancelOrderRequest::set_data(
std::string exchange_order_id,
std::string exchange_security_id,
std::string venue,
std::string security_type,
bool is_buy) {
key_ = key;
exchange_order_id_ = std::move(exchange_order_id);
exchange_security_id_ = std::move(exchange_security_id);
venue_ = std::move(venue);
security_type_ = std::move(security_type);
is_buy_ = is_buy;
}
@@ -31,13 +31,6 @@ void PyCancelReplaceOrderRequest::set_venue(string value) {
venue_ = std::move(value);
}
const string &PyCancelReplaceOrderRequest::security_type() const {
return security_type_;
}
void PyCancelReplaceOrderRequest::set_security_type(string value) {
security_type_ = std::move(value);
}
bool PyCancelReplaceOrderRequest::is_buy() const { return is_buy_; }
void PyCancelReplaceOrderRequest::set_is_buy(bool value) {
is_buy_ = std::move(value);
@@ -56,7 +49,6 @@ void to_json(nlohmann::json &j, const PyCancelReplaceOrderRequest &req) {
j["eoid"] = req.exchange_order_id();
j["esid"] = req.exchange_security_id();
j["venue"] = req.venue();
j["styp"] = req.security_type();
j["side"] = req.is_buy() ? 'B' : 'S';
j["px"] = req.price();
j["qty"] = req.quantity();
@@ -67,7 +59,6 @@ void from_json(const nlohmann::json &j, PyCancelReplaceOrderRequest &req) {
j.at("eoid").get_to(req.exchange_order_id_);
j.at("esid").get_to(req.exchange_security_id_);
j.at("venue").get_to(req.venue_);
j.at("styp").get_to(req.security_type_);
const auto str = j.at("side").get<string>();
if (!str.empty()) {
switch (str[0]) {
@@ -27,9 +27,6 @@ class PyCancelReplaceOrderRequest {
const std::string &venue() const;
void set_venue(std::string value);
const std::string &security_type() const;
void set_security_type(std::string value);
bool is_buy() const;
void set_is_buy(bool value);
@@ -44,7 +41,6 @@ class PyCancelReplaceOrderRequest {
std::string exchange_order_id,
std::string exchange_security_id,
std::string venue,
std::string security_type,
bool is_buy,
double price,
double quantity);
@@ -56,7 +52,6 @@ class PyCancelReplaceOrderRequest {
std::string exchange_order_id_;
std::string exchange_security_id_;
std::string venue_;
std::string security_type_;
float price_;
float quantity_;
bool is_buy_;
@@ -90,7 +85,6 @@ inline void PyCancelReplaceOrderRequest::set_data(
std::string exchange_order_id,
std::string exchange_security_id,
std::string venue,
std::string security_type,
bool is_buy,
double price,
double quantity) {
@@ -98,7 +92,6 @@ inline void PyCancelReplaceOrderRequest::set_data(
exchange_order_id_ = std::move(exchange_order_id);
exchange_security_id_ = std::move(exchange_security_id);
venue_ = std::move(venue);
security_type_ = std::move(security_type);
is_buy_ = is_buy;
price_ = price;
quantity_ = quantity;
@@ -2,7 +2,6 @@
#include "spark/api/services/native_order_type.h"
#include "spark/api/services/time_in_force.h"
#include "spark/domain_model/Security.pb.h"
#include "spark/domain_model/SecurityType.h"
#include "spark/domain_model/Venue.h"
#include "spark/orderman/CreateOrderMsg.h"
@@ -28,13 +27,6 @@ void PyCreateOrderRequest::set_venue(string value) {
venue_ = std::move(value);
}
const string &PyCreateOrderRequest::security_type() const {
return security_type_;
}
void PyCreateOrderRequest::set_security_type(string value) {
security_type_ = std::move(value);
}
float PyCreateOrderRequest::price() const { return price_; }
void PyCreateOrderRequest::set_price(float value) { price_ = value; }
@@ -58,11 +50,23 @@ void PyCreateOrderRequest::set_is_buy(bool value) {
is_buy_ = std::move(value);
}
void PyCreateOrderRequest::copy_from(const bts::orderman::CreateOrderMsg &msg) {
key_ = msg.key.value();
exchange_security_id_ = msg.security_name();
venue_ = bts::api::venue_name_pretty(msg.venue());
is_buy_ = msg.side == bts::api::SIDE_BUY;
price_ = msg.price.value();
quantity_ = msg.quantity.full_value_float();
order_type_ =
bts::api::orderman::native_order_type_name_pretty(msg.order_type);
time_in_force_ =
bts::api::orderman::time_in_force_name_pretty(msg.time_in_force);
}
void to_json(nlohmann::json &j, const PyCreateOrderRequest &req) {
j["key"] = req.key();
j["esid"] = req.exchange_security_id();
j["venue"] = req.venue();
j["styp"] = req.security_type();
j["side"] = req.is_buy() ? "B" : "S";
j["px"] = req.price();
j["qty"] = req.quantity();
@@ -74,7 +78,6 @@ void from_json(const nlohmann::json &j, PyCreateOrderRequest &req) {
j.at("key").get_to(req.key_);
j.at("eid").get_to(req.exchange_security_id_);
j.at("venue").get_to(req.venue_);
j.at("styp").get_to(req.security_type_);
const auto str = j.at("side").get<string>();
if (!str.empty()) {
switch (str[0]) {
@@ -24,9 +24,6 @@ class PyCreateOrderRequest {
const std::string &venue() const;
void set_venue(std::string value);
const std::string &security_type() const;
void set_security_type(std::string value);
bool is_buy() const;
void set_is_buy(bool value);
@@ -46,13 +43,13 @@ class PyCreateOrderRequest {
std::uint32_t key,
std::string exchange_security_id,
std::string venue,
std::string security_type,
bool is_buy,
double price,
double quantity,
std::string order_type,
std::string time_in_force);
void copy_from(const bts::orderman::CreateOrderMsg &msg);
std::string to_string() const;
private:
@@ -61,7 +58,6 @@ class PyCreateOrderRequest {
float quantity_;
std::string exchange_security_id_;
std::string venue_;
std::string security_type_;
std::string order_type_;
std::string time_in_force_;
bool is_buy_;
@@ -93,7 +89,6 @@ inline void PyCreateOrderRequest::set_data(
std::uint32_t key,
std::string exchange_security_id,
std::string venue,
std::string security_type,
bool is_buy,
double price,
double quantity,
@@ -102,7 +97,6 @@ inline void PyCreateOrderRequest::set_data(
key_ = key;
exchange_security_id_ = std::move(exchange_security_id);
venue_ = std::move(venue);
security_type_ = std::move(security_type);
is_buy_ = is_buy;
price_ = price;
quantity_ = quantity;
@@ -74,9 +74,6 @@ class PyExecutionReport {
std::int64_t transact_time() const;
void set_transact_time(std::int64_t value);
const std::string &error_msg() const;
void set_error_msg(std::string value);
/**
* Sets all data at once (more performant than setting props one at a time
* when calling from python).
@@ -130,7 +127,6 @@ class PyExecutionReport {
double avg_filled_price_;
std::string transact_id_;
std::int64_t transact_time_;
std::string error_msg_;
friend void from_json(const nlohmann::json &j, PyExecutionReport &req);
};
@@ -267,13 +263,6 @@ inline void PyExecutionReport::set_transact_time(std::int64_t value) {
transact_time_ = value;
}
inline const std::string &PyExecutionReport::error_msg() const {
return error_msg_;
}
inline void PyExecutionReport::set_error_msg(std::string value) {
error_msg_ = std::move(value);
}
inline std::string PyExecutionReport::to_string() const {
return fmt::format("{}", *this);
}
@@ -0,0 +1,153 @@
#include "py_cancel_order_request.h"
#include "py_cancel_replace_order_request.h"
#include "py_create_order_request.h"
#include "python_interop/execution/execution_gateway.h"
#include "python_interop/execution/execution_gateway_bridge.h"
#include "python_interop/execution/py_execution_gateway.h"
#include "python_interop/execution/py_execution_report.h"
#include "python_interop/execution/py_position_info.h"
#include "python_interop/md/md_bridge.h"
#include "python_interop/md/md_provider.h"
#include "python_interop/md/py_book.h"
#include "python_interop/md/py_book_level.h"
#include "python_interop/md/py_book_side.h"
#include "python_interop/md/py_md_provider.h"
#include "python_interop/md/py_trade.h"
#include <memory>
#include <pybind11/detail/common.h>
#include <pybind11/pybind11.h>
#include <pybind11/stl_bind.h>
using bts::md::py_interop::MdBridge;
using bts::md::py_interop::MdProvider;
using bts::md::py_interop::PyBook;
using bts::md::py_interop::PyBookLevel;
using bts::md::py_interop::PyBookSide;
using bts::md::py_interop::PyMdProvider;
using bts::md::py_interop::PyTrade;
using python_execution::ExecutionGateway;
using python_execution::ExecutionGatewayBridge;
using python_execution::PyCancelOrderRequest;
using python_execution::PyCancelReplaceOrderRequest;
using python_execution::PyCreateOrderRequest;
using python_execution::PyExecutionGateway;
using python_execution::PyExecutionReport;
using python_execution::PyPositionInfo;
using std::shared_ptr;
using std::shared_ptr;
PYBIND11_DECLARE_HOLDER_TYPE(T, std::shared_ptr<T>)
namespace py = pybind11;
#define PY_ARG(arg) pybind11::arg(#arg)
#define _STR(x) #x
// In general, we bind c++ methods to python methods with the same name verbatim.
// The DEF macro avoids typos in the python method name for this common case.
#define DEF(type, member) def(_STR(member), &type::member)
#define DEF_ARGS(type, member, ...) \
def(_STR(member), &type::member, __VA_ARGS__)
#define EXPLICIT_DEF(type, member, member_type) \
def(_STR(member), (member_type) & type::member)
// The PROP macro defines getter and setter methods according to <name>
// Binds a pseudo property with getter/setter methods.
// Binds getter -> <name>, and setter -> set_<name>.
#define PROP(type, name) \
def(_STR(name), &type::name) \
.def(_STR(set_##name), &type::set_##name, py::arg(#name))
PYBIND11_MODULE(python_execution_module, m) {
py::class_<PyCreateOrderRequest, shared_ptr<PyCreateOrderRequest>>(
m, "PyCreateOrderRequest")
.def(py::init<>())
.PROP(PyCreateOrderRequest, key)
.PROP(PyCreateOrderRequest, exchange_security_id)
.PROP(PyCreateOrderRequest, venue)
.PROP(PyCreateOrderRequest, is_buy)
.PROP(PyCreateOrderRequest, price)
.PROP(PyCreateOrderRequest, quantity)
.PROP(PyCreateOrderRequest, order_type)
.PROP(PyCreateOrderRequest, time_in_force)
.def("__str__", &PyCreateOrderRequest::to_string);
py::class_<PyCancelOrderRequest, shared_ptr<PyCancelOrderRequest>>(
m, "PyCancelOrderRequest")
.def(py::init<>())
.PROP(PyCancelOrderRequest, key)
.PROP(PyCancelOrderRequest, exchange_order_id)
.PROP(PyCancelOrderRequest, exchange_security_id)
.PROP(PyCancelOrderRequest, venue)
.PROP(PyCancelOrderRequest, is_buy)
.def("__str__", &PyCancelOrderRequest::to_string);
py::class_<
PyCancelReplaceOrderRequest, shared_ptr<PyCancelReplaceOrderRequest>>(
m, "PyCancelReplaceOrderRequest")
.def(py::init<>())
.PROP(PyCancelReplaceOrderRequest, key)
.PROP(PyCancelReplaceOrderRequest, exchange_order_id)
.PROP(PyCancelReplaceOrderRequest, exchange_security_id)
.PROP(PyCancelReplaceOrderRequest, venue)
.PROP(PyCancelReplaceOrderRequest, is_buy)
.PROP(PyCancelReplaceOrderRequest, price)
.PROP(PyCancelReplaceOrderRequest, quantity)
.def("__str__", &PyCancelReplaceOrderRequest::to_string);
py::class_<PyExecutionReport, shared_ptr<PyExecutionReport>>(
m, "PyExecutionReport")
.def(py::init<>())
.PROP(PyExecutionReport, key)
.PROP(PyExecutionReport, account)
.PROP(PyExecutionReport, exchange_order_id)
.PROP(PyExecutionReport, exchange_security_id)
.PROP(PyExecutionReport, order_status)
.PROP(PyExecutionReport, time_in_force)
.PROP(PyExecutionReport, exec_type)
.PROP(PyExecutionReport, venue)
.PROP(PyExecutionReport, security_type)
.PROP(PyExecutionReport, is_buy)
.PROP(PyExecutionReport, price)
.PROP(PyExecutionReport, quantity)
.PROP(PyExecutionReport, filled_quantity)
.PROP(PyExecutionReport, last_filled_quantity)
.PROP(PyExecutionReport, last_filled_price)
.PROP(PyExecutionReport, avg_filled_price)
.PROP(PyExecutionReport, order_type)
.PROP(PyExecutionReport, transact_time)
.def(
"set_data", &PyExecutionReport::set_data, py::arg("key"),
py::arg("account"), py::arg("exchange_order_id"),
py::arg("exchange_security_id"), py::arg("venue"),
py::arg("security_type"), py::arg("order_type"),
py::arg("order_status"), py::arg("time_in_force"),
py::arg("exec_type"), py::arg("is_buy"), py::arg("price"),
py::arg("quantity"), py::arg("filled_quantity"),
py::arg("last_filled_quantity"), py::arg("last_filled_price"),
py::arg("avg_filled_price"), py::arg("transact_id"),
py::arg("transact_time"))
.def("__str__", &PyExecutionReport::to_string);
py::class_<PyPositionInfo, shared_ptr<PyPositionInfo>>(m, "PyPositionInfo")
.def(py::init<>())
.PROP(PyPositionInfo, account)
.PROP(PyPositionInfo, exchange_security_id)
.PROP(PyPositionInfo, venue)
.PROP(PyPositionInfo, security_type)
.PROP(PyPositionInfo, long_quantity)
.PROP(PyPositionInfo, long_avg_cost)
.PROP(PyPositionInfo, short_quantity)
.PROP(PyPositionInfo, short_avg_cost)
.PROP(PyPositionInfo, transact_time)
.def("__str__", &PyPositionInfo::to_string);
py::class_<ExecutionGatewayBridge>(m, "ExecutionGatewayBridge")
.def(py::init<>())
.DEF(ExecutionGatewayBridge, on_execution_report)
.DEF(ExecutionGatewayBridge, on_position_info)
.DEF(ExecutionGatewayBridge, on_is_ready);
py::class_<ExecutionGateway, PyExecutionGateway>(m, "ExecutionGateway")
.def(py::init<>())
.DEF_ARGS(ExecutionGateway, start, py::arg("bridge"))
.DEF_ARGS(ExecutionGateway, create_order, py::arg("req"))
.DEF_ARGS(ExecutionGateway, cancel_order, py::arg("req"))
.DEF_ARGS(ExecutionGateway, cancel_replace_order, py::arg("req"))
.DEF(ExecutionGateway, stop);
}
@@ -28,7 +28,7 @@ void MdBridgeImpl::on_securities(
}
void MdBridgeImpl::on_books(const std::vector<std::shared_ptr<PyBook>> &books) {
DIAG_TRACE_PARAMS("on_books", json_formatter(books.size()));
INFO_TRACE_PARAMS("on_books", json_formatter(books.size()));
incoming_books.push_many(books.begin(), books.end());
}
@@ -145,9 +145,6 @@ inline std::string PyBook::to_string() const {
}
inline void to_json(nlohmann::json &j, const PyBook &bk) {
j["sec"] = bk.exchange_security_id();
j["venue"] = bk.venue();
j["typ"] = bk.security_type();
j["bids"] = bk.bids();
j["asks"] = bk.asks();
}
@@ -0,0 +1,97 @@
#include "md_provider.h"
#include "python_interop/md/py_book_level.h"
#include "python_interop/md/py_book_side.h"
#include "python_interop/md/py_md_provider.h"
#include "python_interop/md/py_trade.h"
#include <pybind11/detail/common.h>
#include <pybind11/pybind11.h>
#include <pybind11/stl_bind.h>
using bts::md::py_interop::MdBridge;
using bts::md::py_interop::MdProvider;
using bts::md::py_interop::PyBook;
using bts::md::py_interop::PyBookLevel;
using bts::md::py_interop::PyBookSide;
using bts::md::py_interop::PyMdProvider;
using bts::md::py_interop::PyTrade;
using std::shared_ptr;
PYBIND11_DECLARE_HOLDER_TYPE(T, std::shared_ptr<T>)
namespace py = pybind11;
#define _STR(x) #x
// In general, we bind c++ methods to python methods with the same name verbatim.
// The DEF macro avoids typos in the python method name for this common case.
#define DEF(type, member) def(_STR(member), &type::member)
#define DEF_ARGS(type, member, ...) \
def(_STR(member), &type::member, __VA_ARGS__)
#define EXPLICIT_DEF(type, member, member_type) \
def(_STR(member), (member_type) & type::member)
// The PROP macro defines getter and setter methods according to <name>
// Binds a pseudo property with getter/setter methods.
// Binds getter -> <name>, and setter -> set_<name>.
#define PROP(type, name) \
def(_STR(name), &type::name) \
.def(_STR(set_##name), &type::set_##name, py::arg(#name))
// Define python module with c++ classes.
PYBIND11_MODULE(python_md_module, m) {
using BookSideGetter = PyBookSide &(PyBook::*)();
using BookLevelGetter = PyBookLevel (PyBookSide::*)(int) const;
// book
py::class_<PyBookLevel>(m, "PyBookLevel")
.def(py::init<>())
.PROP(PyBookLevel, price)
.PROP(PyBookLevel, quantity);
py::class_<PyBookSide>(m, "PyBookSide")
.def(py::init<>())
.EXPLICIT_DEF(PyBookSide, get_level, BookLevelGetter)
.DEF(PyBookSide, set_level)
.PROP(PyBookSide, is_ascending)
.def("__len__", &PyBookSide::size)
.def("__str__", &PyBookSide::to_string);
py::class_<PyBook, shared_ptr<PyBook>>(m, "PyBook")
.def(py::init<>())
.PROP(PyBook, venue)
.PROP(PyBook, security_type)
.PROP(PyBook, exchange_security_id)
.EXPLICIT_DEF(PyBook, bids, BookSideGetter)
.DEF(PyBook, set_bids)
.EXPLICIT_DEF(PyBook, asks, BookSideGetter)
.DEF(PyBook, set_asks)
.def(
"set_data", &PyBook::set_data, py::arg("exch_sec_id"),
py::arg("venue"), py::arg("sec_type"), py::arg("bids"),
py::arg("asks"))
.def("__len__", &PyBook::size)
.def("__str__", &PyBook::to_string);
py::bind_vector<std::vector<shared_ptr<PyBook>>>(m, "PyBookList");
// trade
py::class_<PyTrade, shared_ptr<PyTrade>>(m, "PyTrade")
.def(py::init<>())
.PROP(PyTrade, venue)
.PROP(PyTrade, security_type)
.PROP(PyTrade, exchange_security_id)
.PROP(PyTrade, price)
.PROP(PyTrade, quantity)
.PROP(PyTrade, aggressor_side)
.PROP(PyTrade, timestamp);
py::bind_vector<std::vector<shared_ptr<PyTrade>>>(m, "PyTradeList");
// md bridge
py::class_<MdBridge>(m, "MdBridge")
.DEF_ARGS(MdBridge, on_securities, py::arg("securities"))
.DEF_ARGS(MdBridge, on_books, py::arg("books"))
.DEF_ARGS(MdBridge, on_trades, py::arg("trades"));
// md provider
py::class_<MdProvider, PyMdProvider>(m, "MdProvider")
.def(py::init<>())
.DEF_ARGS(MdProvider, start, py::arg("bridge"))
.DEF(MdProvider, stop);
}
#undef PROP
#undef EXPLICIT_DEF
#undef DEF
#undef _STR
@@ -5,281 +5,163 @@
#include "python_interop/md/md_provider.h"
#include "python_interop/plugin_manager_logs.h"
#include "python_interop/python_object.h"
#include "spark/collections/concurrent_queue.h"
#include "spark/collections/concurrent_unordered_map.h"
#include "spark/utils/stopwatch.h"
#include "spark/utils/string_utils.h"
#include <csignal>
#include <fmt/core.h>
#include <functional>
#include <future>
#include <memory>
#include <mutex>
#include <pybind11/embed.h>
#include <pybind11/gil.h>
#include <pybind11/pytypes.h>
#include <string>
#include <thread>
#include <utility>
namespace py = pybind11;
namespace python_interop {
using bts::collections::ConcurrentQueue;
using bts::collections::ConcurrentUnorderedMap;
using bts::md::py_interop::MdBridge;
using bts::utils::Stopwatch;
using python_execution::ExecutionGateway;
using python_execution::ExecutionGatewayBridge;
using std::function;
using std::future;
using std::mutex;
using std::optional;
using std::shared_ptr;
using std::string;
using std::thread;
using std::vector;
static bool parse_target(
const std::string &target,
std::string *module_name,
std::string *type_name) {
auto parts = bts::system::string_utils::split(target, ".");
if (parts.empty()) {
return false;
}
if (parts.size() == 1) {
bts::system::string_utils::snake_to_pascal_case(
parts.emplace_back(parts[0]));
}
*module_name = parts[0];
*type_name = parts[1];
return !module_name->empty() && !type_name->empty();
}
PluginManager::~PluginManager() { stop_work_thread(); }
struct __attribute__((visibility("hidden"))) PluginManager_SessionData {
py::scoped_interpreter interpreter{false};
std::unordered_map<std::string, std::unique_ptr<py::module>> modules;
py::module &import(const std::string &name) {
auto [it, is_new] = modules.try_emplace(name, nullptr);
if (is_new) {
it->second =
std::make_unique<py::module>(py::module::import(name.c_str()));
}
return *it->second;
}
};
struct __attribute__((visibility("hidden"))) PluginManager_Impl {
volatile bool keep_going;
mutex mx;
optional<thread> work_thread;
ConcurrentQueue<function<void()>> work_queue;
ConcurrentUnorderedMap<
string,
shared_ptr<PythonObjectT<bts::md::py_interop::MdProvider>>>
md_providers;
ConcurrentUnorderedMap<
string,
shared_ptr<PythonObjectT<python_execution::ExecutionGateway>>>
execution_gateways;
PluginManager_SessionData *session_data;
void start_work_thread() {
std::lock_guard<std::mutex> lock{mx};
if (work_thread) {
return;
}
keep_going = true;
work_thread.emplace([&]() {
PluginManager_SessionData sd;
mx.lock();
session_data = &sd;
mx.unlock();
try {
sd.import("sys").attr("path").attr("insert")(
0, "python_interop_plugins");
sd.import("spark_core").attr("configure_default_logger")();
sd.import("bybit_md_provider");
sd.import("bybit_execution_gateway");
py::gil_scoped_release gil_rel;
Stopwatch sw;
int n = 0;
while (keep_going) {
auto work_req = work_queue.wait_until_pop();
if (!work_req) {
continue;
}
LOG_INVOKING(n);
sw.restart();
{
py::gil_scoped_acquire gil_acquire;
(*work_req)();
}
LOG_INVOKED(n, sw.elapsed());
n++;
}
} catch (const std::exception &ex) {
bts_assert_msgf(
false, "PythonWorker work_thread threw. {}",
std::string(ex.what()));
return;
} catch (...) {
bts_assert_msg(false, "PythonWorker work_thread threw unknown ex.");
return;
}
mx.lock();
session_data = nullptr;
mx.unlock();
});
}
void stop_work_thread() {
mx.lock();
if (work_thread && work_thread->joinable()) {
keep_going = false;
mx.unlock();
work_thread->join();
work_thread.reset();
} else {
mx.unlock();
}
}
future<shared_ptr<PythonObjectT<bts::md::py_interop::MdProvider>>>
start_md_provider(
string provider_name, string target, shared_ptr<MdBridge> bridge) {
auto prom = std::make_shared<std::promise<
shared_ptr<PythonObjectT<bts::md::py_interop::MdProvider>>>>();
work_queue.push([this, target = std::move(target),
provider_name = std::move(provider_name),
bridge = std::move(bridge), prom = prom]() {
string module_name;
string type_name;
if (!parse_target(target, &module_name, &type_name)) {
LOG_WORK_ERROR(
md,
fmt::format("Failed to parse module/type from '{}'", target));
return;
}
py::object obj = session_data->import(module_name.c_str())
.attr(type_name.c_str())();
auto [md_pro_it, md_pro_is_new] = md_providers.try_emplace(
provider_name,
std::make_shared<PythonObjectT<bts::md::py_interop::MdProvider>>(
std::move(obj)));
bts_assert(md_pro_is_new);
LOG_STARTING("md_provider", provider_name);
Stopwatch sw = Stopwatch::start_new();
shared_ptr<PythonObjectT<bts::md::py_interop::MdProvider>> md_pro =
md_pro_it->second;
(*md_pro)->start(bridge.get());
LOG_STARTED("md_provider", provider_name, sw.elapsed());
prom->set_value(md_pro);
// Don't let python handle the signals (otherwise it'll hang on ctrl+c)
std::signal(SIGINT, SIG_DFL);
});
return prom->get_future();
}
void stop_md_provider(const std::string &provider_name) {
auto it = md_providers.find(provider_name);
if (it != md_providers.end()) {
(*it->second)->stop();
md_providers.erase(it);
}
}
future<shared_ptr<PythonObjectT<ExecutionGateway>>> start_execution_gateway(
string gateway_name,
string target,
shared_ptr<ExecutionGatewayBridge> bridge) {
auto prom = std::make_shared<
std::promise<shared_ptr<PythonObjectT<ExecutionGateway>>>>();
work_queue.push([this, target = std::move(target),
gateway_name = std::move(gateway_name),
bridge = std::move(bridge), prom = prom]() {
string module_name;
string type_name;
if (!parse_target(target, &module_name, &type_name)) {
LOG_WORK_ERROR(
OrderMan,
fmt::format("Failed to parse module/type from '{}'", target));
return;
}
py::object obj = session_data->import(module_name.c_str())
.attr(type_name.c_str())();
auto [exe_gw_it, exe_gw_is_new] = execution_gateways.try_emplace(
gateway_name,
std::make_shared<PythonObjectT<ExecutionGateway>>(std::move(obj)));
bts_assert(exe_gw_is_new);
LOG_STARTING("execution_gateway", gateway_name);
Stopwatch sw = Stopwatch::start_new();
shared_ptr<PythonObjectT<ExecutionGateway>> exe_gw = exe_gw_it->second;
(*exe_gw)->start(bridge.get());
LOG_STARTED("execution_gateway", gateway_name, sw.elapsed());
prom->set_value(exe_gw);
// Don't let python handle the signals (otherwise it'll hang on ctrl+c)
std::signal(SIGINT, SIG_DFL);
});
return prom->get_future();
}
void stop_execution_gateway(const std::string &provider_name) {
auto it = execution_gateways.find(provider_name);
if (it != execution_gateways.end()) {
(*it->second)->stop();
execution_gateways.erase(it);
}
}
};
PluginManager::PluginManager() : me_(new PluginManager_Impl()) {}
PluginManager::~PluginManager() {
stop_work_thread();
delete static_cast<PluginManager_Impl *>(me_);
me_ = nullptr;
}
PythonWorkRequest::~PythonWorkRequest() = default;
void PluginManager::start_work_thread() {
static_cast<PluginManager_Impl *>(me_)->start_work_thread();
mx.lock();
if (work_thread) {
mx.unlock();
return;
}
keep_going = true;
mx.unlock();
work_thread.emplace([&]() {
pybind11::scoped_interpreter interpreter{false};
try {
pybind11::gil_scoped_release gil_rel;
Stopwatch sw;
int n = 0;
while (keep_going) {
auto work_req = work_queue.wait_until_pop();
if (!work_req) {
continue;
}
LOG_INVOKING(n);
sw.restart();
(**work_req).invoke();
LOG_INVOKED(n, sw.elapsed());
n++;
}
} catch (const std::exception &ex) {
bts_assert_msgf(
false, "PythonWorker work_thread threw. {}",
std::string(ex.what()));
}
});
}
void PluginManager::stop_work_thread() {
static_cast<PluginManager_Impl *>(me_)->stop_work_thread();
mx.lock();
if (work_thread && work_thread->joinable()) {
keep_going = false;
mx.unlock();
work_thread->join();
work_thread.reset();
} else {
mx.unlock();
}
}
struct PythonWorkRequestDelegate : PythonWorkRequest {
std::function<void()> callback;
PythonWorkRequestDelegate(std::function<void()> callback)
: callback(callback) {}
void invoke() override { callback(); }
};
void PluginManager::invoke(std::function<void()> callback) {
auto *me = static_cast<PluginManager_Impl *>(me_);
me->work_queue.push(callback);
work_queue.push(std::make_shared<PythonWorkRequestDelegate>(callback));
}
std::future<shared_ptr<PythonObjectT<bts::md::py_interop::MdProvider>>>
PluginManager::start_md_provider(
string provider_name, string target, shared_ptr<MdBridge> bridge) {
return static_cast<PluginManager_Impl *>(me_)->start_md_provider(
std::move(provider_name), std::move(target), std::move(bridge));
string provider_name,
string target,
vector<string> module_paths,
shared_ptr<MdBridge> bridge) {
auto prom = std::make_shared<std::promise<
shared_ptr<PythonObjectT<bts::md::py_interop::MdProvider>>>>();
std::function<void(py::object)> callback{
[&, provider_name = std::move(provider_name), bridge = std::move(bridge),
prom = prom](py::object obj) {
auto [md_pro_it, md_pro_is_new] = md_providers.try_emplace(
provider_name,
std::make_shared<PythonObjectT<bts::md::py_interop::MdProvider>>(
std::move(obj)));
bts_assert(md_pro_is_new);
LOG_STARTING("md_provider", provider_name);
Stopwatch sw = Stopwatch::start_new();
shared_ptr<PythonObjectT<bts::md::py_interop::MdProvider>> md_pro =
md_pro_it->second;
(*md_pro)->start(bridge.get());
LOG_STARTED("md_provider", provider_name, sw.elapsed());
prom->set_value(md_pro);
// Don't let python handle the signals (otherwise it'll hang on ctrl+c)
std::signal(SIGINT, SIG_DFL);
}};
invoke_on_obj(std::move(target), std::move(module_paths), callback);
return prom->get_future();
}
void PluginManager::stop_md_provider(const std::string &provider_name) {
static_cast<PluginManager_Impl *>(me_)->stop_md_provider(provider_name);
auto it = md_providers.find(provider_name);
if (it != md_providers.end()) {
(*it->second)->stop();
md_providers.erase(it);
}
}
std::future<shared_ptr<PythonObjectT<ExecutionGateway>>>
PluginManager::start_execution_gateway(
string gateway_name,
string target,
vector<string> module_paths,
shared_ptr<ExecutionGatewayBridge> bridge) {
return static_cast<PluginManager_Impl *>(me_)->start_execution_gateway(
std::move(gateway_name), std::move(target), std::move(bridge));
auto prom = std::make_shared<
std::promise<shared_ptr<PythonObjectT<ExecutionGateway>>>>();
std::function<void(py::object)> callback{[&, gateway_name = gateway_name,
bridge = std::move(bridge),
prom = prom](py::object obj) {
auto [exe_gw_it, exe_gw_is_new] = execution_gateways.try_emplace(
gateway_name,
std::make_shared<PythonObjectT<ExecutionGateway>>(std::move(obj)));
bts_assert(exe_gw_is_new);
LOG_STARTING("execution_gateway", gateway_name);
Stopwatch sw = Stopwatch::start_new();
shared_ptr<PythonObjectT<ExecutionGateway>> exe_gw = exe_gw_it->second;
(*exe_gw)->start(bridge.get());
LOG_STARTED("execution_gateway", gateway_name, sw.elapsed());
prom->set_value(exe_gw);
// Don't let python handle the signals (otherwise it'll hang on ctrl+c)
std::signal(SIGINT, SIG_DFL);
}};
invoke_on_obj(std::move(target), std::move(module_paths), callback);
return prom->get_future();
}
void PluginManager::stop_execution_gateway(const std::string &provider_name) {
return static_cast<PluginManager_Impl *>(me_)->stop_execution_gateway(
provider_name);
auto it = execution_gateways.find(provider_name);
if (it != execution_gateways.end()) {
(*it->second)->stop();
execution_gateways.erase(it);
}
}
} // namespace python_interop
@@ -5,20 +5,36 @@
#include "python_interop/md/md_bridge.h"
#include "python_interop/md/md_provider.h"
#include "python_interop/python_object.h"
#include "python_interop/python_work_request.h"
#include "spark/collections/concurrent_queue.h"
#include "spark/collections/concurrent_unordered_map.h"
#include <functional>
#include <future>
#include <memory>
#include <pybind11/pytypes.h>
#include <string>
#include <thread>
#include <vector>
namespace python_interop {
class PluginManager {
void *me_;
volatile bool keep_going;
std::mutex mx;
std::optional<std::thread> work_thread;
bts::collections::ConcurrentQueue<std::shared_ptr<PythonWorkRequest>>
work_queue;
bts::collections::ConcurrentUnorderedMap<
std::string,
std::shared_ptr<PythonObjectT<bts::md::py_interop::MdProvider>>>
md_providers;
bts::collections::ConcurrentUnorderedMap<
std::string,
std::shared_ptr<PythonObjectT<python_execution::ExecutionGateway>>>
execution_gateways;
public:
PluginManager();
~PluginManager();
void start_work_thread();
@@ -28,6 +44,7 @@ class PluginManager {
start_md_provider(
std::string provider_name,
std::string target,
std::vector<std::string> module_paths,
std::shared_ptr<bts::md::py_interop::MdBridge> bridge);
void stop_md_provider(const std::string &provider_name);
@@ -36,10 +53,34 @@ class PluginManager {
start_execution_gateway(
std::string gateway_name,
std::string target,
std::vector<std::string> module_paths,
std::shared_ptr<python_execution::ExecutionGatewayBridge> bridge);
void stop_execution_gateway(const std::string &gateway_name);
void invoke(std::function<void()> callback);
private:
template <typename... Args>
__attribute__((visibility("hidden"))) void invoke_on_obj(
std::string target,
std::vector<std::string> module_path,
std::function<void(pybind11::object)> callback,
Args &&...args);
};
//////////////////////////////////////////////////
// Definition
//////////////////////////////////////////////////
template <typename... Args>
inline void PluginManager::invoke_on_obj(
std::string target,
std::vector<std::string> module_path,
std::function<void(pybind11::object)> callback,
Args &&...args) {
work_queue.push(std::make_shared<PythonWorkRequestT<Args...>>(
std::move(target), std::move(module_path), callback,
std::forward<Args>(args)...));
}
} // namespace python_interop
@@ -7,6 +7,8 @@
#include "spark/utils/thread_id.h"
#include <fmt/core.h>
#include <pybind11/embed.h>
#include <pybind11/gil.h>
#include <string>
namespace python_interop::logs {
@@ -14,7 +16,7 @@ namespace python_interop::logs {
struct StartingLog {
std::string instance_name;
std::string thread_name;
std::size_t thread_id;
std::int64_t thread_id;
StartingLog(std::string instance_name)
: instance_name(std::move(instance_name)),
@@ -25,7 +27,7 @@ struct StartingLog {
struct StartedLog {
std::string instance_name;
std::string thread_name;
std::size_t thread_id;
std::int64_t thread_id;
bts::core::TimeSpan duration;
StartedLog(std::string instance_name, bts::core::TimeSpan duration)
@@ -37,7 +39,7 @@ struct StartedLog {
struct InvokingLog {
int count;
std::string thread_name;
std::size_t thread_id;
std::int64_t thread_id;
InvokingLog(int count)
: count(count), thread_name(bts::os::get_thread_name()),
@@ -48,7 +50,7 @@ struct InvokedLog {
int count;
bts::core::TimeSpan duration;
std::string thread_name;
std::size_t thread_id;
std::int64_t thread_id;
InvokedLog(int count, bts::core::TimeSpan dequeue_duration)
: count(count), duration(dequeue_duration),
@@ -111,6 +113,3 @@ FMT_IMPL(
LOG_TRACE_PARAMS( \
Gen, bts::Info, string("py_plugin"), string("work"), \
python_interop::logs::InvokedLog(count, duration))
#define LOG_WORK_ERROR(subsystem, params) \
LOG_TRACE_PARAMS(subsystem, bts::Error, string("py_plugin"), string("work"), params)
@@ -63,24 +63,6 @@ namespace py = pybind11;
def(_STR(name), &type::name) \
.def(_STR(set_##name), &type::set_##name, py::arg(#name))
#define PY_LIST_TO_STRING(type) \
def("__str__", [](const std::vector<shared_ptr<type>> &py_list) { \
std::ostringstream oss; \
oss << "["; \
for (size_t i = 0; i < py_list.size(); ++i) { \
if (i > 0) { \
oss << ", "; \
} \
if (py_list[i]) { \
oss << py_list[i]->to_string(); \
} else { \
oss << "None"; \
} \
} \
oss << "]"; \
return oss.str(); \
})
// Define python module with c++ classes.
PYBIND11_MODULE(python_interop_module, m) {
// price info
@@ -93,8 +75,7 @@ PYBIND11_MODULE(python_interop_module, m) {
.PROP(PyContractPriceInfo, max_price)
.def("__str__", &PyContractPriceInfo::to_string);
py::bind_vector<std::vector<PyContractPriceInfo>>(
m, "PyContractPriceInfoList")
.PY_LIST_TO_STRING(PyContractPriceInfo);
m, "PyContractPriceInfoList");
// quantity info
py::class_<PyContractQuantityInfo>(m, "PyContractQuantityInfo")
.def(py::init<>())
@@ -106,8 +87,7 @@ PYBIND11_MODULE(python_interop_module, m) {
.PROP(PyContractQuantityInfo, max_price)
.def("__str__", &PyContractQuantityInfo::to_string);
py::bind_vector<std::vector<PyContractQuantityInfo>>(
m, "PyContractQuantityInfoList")
.PY_LIST_TO_STRING(PyContractQuantityInfo);
m, "PyContractQuantityInfoList");
// contract spec
py::class_<PyContractSpecification>(m, "PyContractSpecification")
.def(py::init<>())
@@ -144,8 +124,7 @@ PYBIND11_MODULE(python_interop_module, m) {
.PROP(PySecurity, option)
.def("__str__", &PySecurity::to_string)
.def("__format__", &PySecurity::format);
py::bind_vector<std::vector<shared_ptr<PySecurity>>>(m, "PySecurityList")
.PY_LIST_TO_STRING(PySecurity);
py::bind_vector<std::vector<shared_ptr<PySecurity>>>(m, "PySecurityList");
//////////////////////////////////////////////////
// Md
@@ -179,8 +158,7 @@ PYBIND11_MODULE(python_interop_module, m) {
py::arg("asks"))
.def("__len__", &PyBook::size)
.def("__str__", &PyBook::to_string);
py::bind_vector<std::vector<shared_ptr<PyBook>>>(m, "PyBookList")
.PY_LIST_TO_STRING(PyBook);
py::bind_vector<std::vector<shared_ptr<PyBook>>>(m, "PyBookList");
// trade
py::class_<PyTrade, shared_ptr<PyTrade>>(m, "PyTrade")
.def(py::init<>())
@@ -191,8 +169,7 @@ PYBIND11_MODULE(python_interop_module, m) {
.PROP(PyTrade, quantity)
.PROP(PyTrade, aggressor_side)
.PROP(PyTrade, timestamp);
py::bind_vector<std::vector<shared_ptr<PyTrade>>>(m, "PyTradeList")
.PY_LIST_TO_STRING(PyTrade);
py::bind_vector<std::vector<shared_ptr<PyTrade>>>(m, "PyTradeList");
// md bridge
py::class_<MdBridge>(m, "MdBridge")
.DEF_ARGS(MdBridge, on_securities, py::arg("securities"))
@@ -213,7 +190,6 @@ PYBIND11_MODULE(python_interop_module, m) {
.PROP(PyCreateOrderRequest, key)
.PROP(PyCreateOrderRequest, exchange_security_id)
.PROP(PyCreateOrderRequest, venue)
.PROP(PyCreateOrderRequest, security_type)
.PROP(PyCreateOrderRequest, is_buy)
.PROP(PyCreateOrderRequest, price)
.PROP(PyCreateOrderRequest, quantity)
@@ -227,7 +203,6 @@ PYBIND11_MODULE(python_interop_module, m) {
.PROP(PyCancelOrderRequest, exchange_order_id)
.PROP(PyCancelOrderRequest, exchange_security_id)
.PROP(PyCancelOrderRequest, venue)
.PROP(PyCancelOrderRequest, security_type)
.PROP(PyCancelOrderRequest, is_buy)
.def("__str__", &PyCancelOrderRequest::to_string);
py::class_<
@@ -238,7 +213,6 @@ PYBIND11_MODULE(python_interop_module, m) {
.PROP(PyCancelReplaceOrderRequest, exchange_order_id)
.PROP(PyCancelReplaceOrderRequest, exchange_security_id)
.PROP(PyCancelReplaceOrderRequest, venue)
.PROP(PyCancelReplaceOrderRequest, security_type)
.PROP(PyCancelReplaceOrderRequest, is_buy)
.PROP(PyCancelReplaceOrderRequest, price)
.PROP(PyCancelReplaceOrderRequest, quantity)
@@ -264,7 +238,6 @@ PYBIND11_MODULE(python_interop_module, m) {
.PROP(PyExecutionReport, avg_filled_price)
.PROP(PyExecutionReport, order_type)
.PROP(PyExecutionReport, transact_time)
.PROP(PyExecutionReport, error_msg)
.def(
"set_data", &PyExecutionReport::set_data, py::arg("key"),
py::arg("account"), py::arg("exchange_order_id"),
@@ -0,0 +1,96 @@
#pragma once
#include "spark/core/TimeStamp.h"
#include "spark/utils/bts_assert.h"
#include "spark/utils/string_utils.h"
#include <functional>
#include <pybind11/embed.h>
#include <pybind11/gil.h>
#include <pybind11/pybind11.h>
#include <pybind11/pytypes.h>
#include <string>
#include <vector>
namespace python_interop {
struct PythonWorkRequest {
virtual ~PythonWorkRequest();
static bool parse_target(
const std::string &target,
std::string *module_name,
std::string *type_name);
virtual void invoke() = 0;
};
template <typename... Args>
struct __attribute__((visibility("hidden"))) PythonWorkRequestT
: public PythonWorkRequest {
std::string target;
std::vector<std::string> module_paths;
std::function<void(pybind11::object)> callback;
std::tuple<Args...> args;
PythonWorkRequestT(
std::string target,
std::vector<std::string> module_paths,
std::function<void(pybind11::object)> callback,
Args... args)
: target(std::move(target)), module_paths(std::move(module_paths)),
callback(callback), args(std::forward<Args>(args)...) {}
~PythonWorkRequestT() override = default;
void invoke() override;
};
inline bool PythonWorkRequest::parse_target(
const std::string &target,
std::string *module_name,
std::string *type_name) {
auto parts = bts::system::string_utils::split(target, ".");
if (parts.empty()) {
return false;
}
if (parts.size() == 1) {
bts::system::string_utils::snake_to_pascal_case(
parts.emplace_back(parts[0]));
}
*module_name = parts[0];
*type_name = parts[1];
return !module_name->empty() && !type_name->empty();
}
template <typename... Args>
inline void PythonWorkRequestT<Args...>::invoke() {
std::string target_module_name;
std::string target_type_name;
bts_assert(parse_target(target, &target_module_name, &target_type_name));
LOG(Gen, bts::Info,
R"([py_worker.json]{{ "ev": "INVOKE", "mod": "{}", "typ": "{}", "args_t": "{}", "ts": {} }})",
target_module_name, target_type_name,
std::string(typeid(decltype(this->args)).name()),
bts::core::TimeStamp::now_ns());
try {
pybind11::gil_scoped_acquire gil_acquire;
pybind11::module sys = pybind11::module::import("sys");
for (const auto &module_path : module_paths) {
sys.attr("path").attr("append")(module_path.c_str());
}
pybind11::module target_module =
pybind11::module::import(target_module_name.c_str());
std::apply(
[&](auto &&...unpacked_args) {
callback(target_module.attr(target_type_name.c_str())(
std::forward<decltype(unpacked_args)>(unpacked_args)...));
},
std::move(args));
} catch (const std::exception &ex) {
bts_assert_msgf(
false, "PythonWorkRequest::invoke threw. {}", std::string(ex.what()));
}
}
} // namespace python_interop
@@ -0,0 +1,503 @@
import argparse
import uuid
import hmac
import hashlib
import os
import logging
import sys
import time
import websocket
import json
import requests
import python_interop_module as pyinter
from string import Template
from threading import Thread
from typing import Any, Callable, Mapping, MutableMapping, Optional, TextIO
from binance_env_helper import (
required_env_var,
expand_vars,
expand_defaults,
load_env,
)
def hmac_hashing(api_secret: str, payload: str):
m = hmac.new(api_secret.encode("utf-8"), payload.encode("utf-8"), hashlib.sha256)
return m.hexdigest()
def rest_request(base_url: str, api_key: str, api_secret: str, parameters: dict = {}):
parameters["timestamp"] = int(time.time() * 1000)
sorted_keys = list(parameters)
sorted_keys.sort()
query_params = "&".join([f"{k}={parameters[k]}" for k in sorted_keys])
signature = hmac_hashing(api_secret=api_secret, payload=query_params)
parameters["signature"] = signature
query_params += "&signature=" + signature
url = base_url + "/api/v3/userDataStream" + "?" + query_params
res = requests.post(url, headers={"X-MBX-APIKEY": api_key})
return res.text
class CircularBufferWriter:
def __init__(self, buffer_line_count: int, sink: TextIO = sys.stdout):
self.buffer_lines = []
self.buffer_line_count = buffer_line_count
self.sink = sink
def print(self, msg):
if self.buffer_line_count <= 0:
self.sink.write(msg + "\n")
return
for _ in range(0, len(self.buffer_lines)):
self.sink.write("\x1b[1A\x1b[2K") # move up cursor and delete whole line
if len(self.buffer_lines) == self.buffer_line_count:
self.buffer_lines.pop()
self.buffer_lines.insert(0, msg)
lines = ""
num_lines = len(self.buffer_lines)
for i in range(0, num_lines):
lines += self.buffer_lines[num_lines - i - 1] + "\n"
self.sink.write(lines)
def _get_arg_parser():
# Cli Args
arg_parser = argparse.ArgumentParser(description="Request market data at Binance")
arg_parser.add_argument(
"--streams",
help="Streams to subscribe (e.g, btcusdt@depth5@500ms)",
)
arg_parser.add_argument(
"--prod", action="store_const", const="PROD", dest="mode", help="Run at prod"
)
arg_parser.add_argument(
"--dev", action="store_const", const="DEV", dest="mode", help="Run at testnet"
)
arg_parser.add_argument(
"--spot",
action="store_const",
const="BINANCE",
dest="venue",
help="Run at spots",
)
arg_parser.add_argument("--op", help="The op to run")
arg_parser.add_argument(
"--param", help="Parameter(s) formatted as key=val)", nargs="+"
)
arg_parser.add_argument("--create-order", action="store_true", help="Create order")
arg_parser.add_argument(
"--usds-future",
action="store_const",
const="BINANCE_USDS_FUTURES",
dest="venue",
help="Run at usds futures",
)
arg_parser.add_argument("--user-data", action="store_true", help="User data")
arg_parser.add_argument(
"--exch-info",
action="store_const",
const="EXCHANGE_INFO",
dest="op",
help="Get exchangeInfo (REST)",
)
arg_parser.add_argument(
"--list-symbols", action="store_true", help="Get symbols (REST)"
)
arg_parser.add_argument(
"--listen-key",
action="store_const",
const="LISTEN_KEY",
dest="op",
help="Get listen key (needed for streams)",
)
arg_parser.add_argument(
"--list-streams", action="store_true", help="List all streams"
)
arg_parser.add_argument(
"--display-lines",
help="Cap the number of lines to display from incoming messages (applies circular buffer to console output).",
)
arg_parser.add_argument("--url", help="The url to send requests to")
arg_parser.set_defaults(mode="DEV", venue="BINANCE", user_data=False, op=None)
return arg_parser
def _parse_bool(s: Optional[str]):
match s:
case "1":
return True
case "true":
return True
case "on":
return True
case "y":
return True
case "yes":
return True
case _:
return False
class BinanceClient:
def __init__(self, venue="BINANCE", mode="DEV", logger=None):
self.venue = venue
self.mode = mode
self.rest_url = required_env_var(f"{venue}_{mode}_URL")
self.ws_url = required_env_var(f"{venue}_{mode}_WS_URL")
self._user_data_ping_thread: Optional[Thread] = None
self.threads: list[Thread] = []
self.logger = logger or logging.getLogger(__name__)
self.api_key = os.environ.get(f"{self.venue}_{self.mode}_API_KEY")
self.secret_key = os.environ.get(f"{self.venue}_{self.mode}_SECRET_KEY")
self.orders = {}
def _on_open_with_ping(
self, on_open: Optional[Callable[[websocket.WebSocket], None]]
):
def wrapper(ws):
def ping_loop():
time.sleep(10)
# if isinstance(ws, websocket.WebSocketApp):
# wsa: websocket.WebSocketApp = ws
# ws = wsa.sock
while True:
ws.sock.ping()
time.sleep(25)
self._user_data_ping_thread = Thread(target=ping_loop)
self._user_data_ping_thread.start()
if on_open:
on_open(ws)
return wrapper
def _get_base_url(self, op, is_ws: bool = False, vars: Optional[Mapping] = None):
"""Gets the base url for the request (includes host and path parts, but not query params)."""
url_var_suffix = "WS_URL" if is_ws else "URL"
url = os.environ.get(f"{self.venue}_{self.mode}_{url_var_suffix}") or ""
req_path = os.environ.get(f"{self.venue}_{op}_PATH") or ""
if vars:
req_path = expand_vars(req_path, vars)
if req_path:
n = 1 if req_path.startswith("/") else 0
n += 1 if self.rest_url.endswith("/") else 0
match n:
case 0:
url += "/" + req_path
case 1:
url += req_path
case 2:
url += req_path[1:]
return url
def _index_transaction(self, bts_key: int):
exch_key = uuid.uuid1().hex
self.orders[exch_key] = bts_key
return exch_key
def _get_params_str(self, op, **kwargs):
params = os.environ.get(f"{self.venue}_{op}_PARAMS") or ""
params = expand_vars(params, kwargs)
params = expand_defaults(params)
return params
def _get_params_dict(self, op, **kwargs):
params = self._get_params_str(op=op, **kwargs)
if not params:
return {}
return {p[0]: p[1] for p in map(lambda x: x.split("="), params.split("&"))}
def _invoke(self, target: Callable[[], Any], run_in_background: bool):
if run_in_background:
t = Thread(target=target)
self.threads.append(t)
t.start()
else:
target()
def request(self, op, **kwargs):
print(
"Preparing request. op=%s, params=[%s]"
% (op, ", ".join([f"{k}={v}" for k, v in kwargs.items()])),
flush=True,
)
self.logger.info(
"Preparing request. op=%s, params=[%s]"
% (op, ", ".join([f"{k}={v}" for k, v in kwargs.items()]))
)
is_signed = _parse_bool(os.environ.get(f"{self.venue}_{op}_IS_SIGNED"))
is_keyed = _parse_bool(os.environ.get(f"{self.venue}_{op}_IS_KEYED"))
method = os.environ.get(f"{self.venue}_{op}_METHOD")
params = self._get_params_dict(op, **kwargs)
base_url = self._get_base_url(op, is_ws=False)
res = None
if is_keyed or is_signed:
if not self.api_key:
self.logger.error("Unable to make signed request. Api key not defined")
return
headers = {"Content-Type": "application/json", "X-MBX-APIKEY": self.api_key}
if is_signed:
if not self.secret_key:
self.logger.error(
"Unable to make signed request. Secret key not defined"
)
return
params["timestamp"] = str(int(time.time() * 1000))
base_url = self._get_base_url(op)
sorted_keys = list(params)
sorted_keys.sort()
params_str = "&".join([f"{k}={params[k]}" for k in sorted_keys])
signature = hmac_hashing(api_secret=self.secret_key, payload=params_str)
params_str += "&signature=" + signature
url = base_url + "?" + params_str
if not method:
method = "POST"
headers = {"X-MBX-APIKEY": self.api_key}
# print("ABOUT TO SEND.", flush=True)
self.logger.info("Sending %s %s" % (method, url))
try:
print(
"Requesting %s %s (%s)"
% (
method,
url,
", ".join([f"{k}={v}" for k, v in headers.items()]),
),
flush=True,
)
res = requests.request(method, url, headers=headers)
# if res:
print("RESULT. %s", str(res), flush=True)
self.logger.log(
logging.INFO if res.ok else logging.ERROR,
"Received %i %s" % (res.status_code, str(res.text)),
)
except Exception as ex:
print("OH NO %s" % str(ex), flush=True)
self.logger.error("Request threw %s" % str(ex))
else:
params_str = "&".join([f"{k}={v}" for k, v in params.items()])
url = base_url
if params_str:
url += "?" + params_str
if not method:
method = "GET"
headers = {"Content-Type": "application/json", "X-MBX-APIKEY": self.api_key}
self.logger.info(
"Sending request %s %s (headers=[%s])"
% (method, url, ", ".join([f"{k}={v}" for k, v in headers.items()]))
)
try:
res = requests.request(method, url, headers=headers)
self.logger.log(
logging.INFO if res.ok else logging.ERROR,
"Received response. %s" % str(res),
)
except Exception as e:
self.logger.error("Request threw: %s" % str(e))
self.logger.log(
logging.INFO if res and res.ok else logging.ERROR,
"Request failed. op=%s, res=\n%s",
op,
str(res),
)
return res
def create_order(self, req: pyinter.PyCreateOrderRequest):
self.logger.info("Create order. %s" % str(req))
self.request(
op="CREATE_ORDER",
symbol=req.exchange_security_id(),
side="BUY" if req.is_buy() else "SELL",
type=req.order_type(),
price=req.price(),
quantity=f"{req.quantity():.6f}",
new_client_order_id=self._index_transaction(req.key()),
)
def cancel_order(self, req: pyinter.PyCancelOrderRequest):
self.request(
op="CANCEL_ORDER",
symbol=req.exchange_security_id(),
side="BUY" if req.is_buy() else "SELL",
orig_client_order_id=str(req.exchange_order_id()),
new_client_order_id=self._index_transaction(req.key()),
)
def subscribe_user_data(
self,
on_message: Optional[Callable[[websocket.WebSocket, Any], None]] = None,
on_error: Optional[Callable[[websocket.WebSocket, Any], None]] = None,
on_close: Optional[Callable[[websocket.WebSocket, Any, Any], None]] = None,
on_open: Optional[Callable[[websocket.WebSocket], None]] = None,
run_in_background: bool = True,
):
self.logger.info("Requesting listen key")
res = self.request("LISTEN_KEY")
if res is None:
print("'listen_key_res' is none (this is unexpected)")
return
self.logger.log(
logging.INFO if res and res.ok else logging.ERROR,
"Received listen key. %s" % str(res),
)
if not res.ok:
self.logger.error("Failed to get listen key")
lkey = res.json().get("listenKey")
self.logger.info("Received listen key %s" % lkey)
url = self._get_base_url("USER_DATA", is_ws=True, vars={"LISTEN_KEY": lkey})
ws_app = websocket.WebSocketApp(
url=url,
on_message=on_message,
on_error=on_error,
on_close=on_close,
on_open=self._on_open_with_ping(on_open),
)
self.logger.info("Starting user data websocket %s" % url)
self._invoke(
lambda: ws_app.run_forever(ping_interval=600, ping_timeout=180),
run_in_background,
)
def subscribe_streams(
self,
streams,
on_message: Optional[Callable[[websocket.WebSocket, Any], None]] = None,
on_error: Optional[Callable[[websocket.WebSocket, Any], None]] = None,
on_close: Optional[Callable[[websocket.WebSocket, Any, Any], None]] = None,
run_in_background: bool = True,
**kwargs,
):
def on_open(ws: websocket.WebSocket):
self.logger.info("Stream websocket connected")
try:
req = {
"method": "SUBSCRIBE",
"params": streams,
"id": 1,
}
req_str = json.dumps(req)
self.logger.info("Requesting stream. %s", req_str)
ws.send_text(req_str)
except Exception as ex:
self.logger.error("Stream request failed. %s", ex)
self.logger.info("Disconnecting stream websocket")
ws.close()
url = self._get_base_url("STREAMS", is_ws=True, **kwargs)
ws_app = websocket.WebSocketApp(
url=url,
on_open=self._on_open_with_ping(on_open),
on_message=on_message,
on_error=on_error,
on_close=on_close,
)
self.logger.info("StreamSubscription @ %s", url)
self._invoke(
lambda: ws_app.run_forever(ping_interval=600, ping_timeout=180),
run_in_background,
)
def _main():
load_env()
arg_parser = _get_arg_parser()
args = arg_parser.parse_args()
if args.list_streams:
stream_descs = [
"{symbol}@aggTrade",
"{symbol}@markPrice",
"{symbol}@markPrice@{interval} (intervals: 1s)",
"!markPrice@arr",
"!markPrice@arr@1s",
"{symbol}@kline_{interval} (intervals: 1m, 3m, 5m, 15m, 30m, 1h, 2h, 4h, 6h, 8h, 12h, 1d, 3d, 1w, 1M)",
"{pair}_{contractType}@continuousKline_{interval} (intervals: 1m, 3m, 5m, 15m, 30m, 1h, 2h, 4h, 6h, 8h, 12h, 1d, 3d, 1w, 1M)"
"{symbol}@miniTicker",
"!ticker@arr",
"{symbol}@ticker",
"!miniTicker@arr",
"{symbol}@bookTicker",
"!bookTicker",
"{symbol}@forceOrder",
"!forceOrder@arr",
"{symbol}@depth{levels} (levels: 5, 10, 20)",
"{symbol}@depth{levels}@{interval} (levels: 5, 10, 20)",
" (intervals: 100ms, 500ms)",
"{symbol}@depth",
"{symbol}@depth@{interval} (intervals: 100ms, 500ms)",
"{symbol}@compositeIndex",
"!contractInfo",
"!assetIndex@arr",
"{assetSymbol}@assetIndex",
]
print("\n".join(stream_descs))
return
logger = logging.getLogger()
cli = BinanceClient(args.venue, args.mode, logger=logger)
params = {kv[0]: kv[1] for kv in map(lambda x: x.split("="), args.param or [])}
if args.op:
res = cli.request(args.op, **params)
if res and res.ok:
print(res.text)
elif args.list_symbols:
# Get symbols
res = cli.request("EXCHANGE_INFO")
if not res or not res.ok:
return 1
syms = {}
res_j = json.loads(res.text)
for sym in res_j["symbols"]:
syms[sym["symbol"]] = sym
sym_list = list(syms)
sym_list.sort()
for sym in sym_list:
print(sym)
elif args.user_data:
# Subscribe user data
writer = CircularBufferWriter(buffer_line_count=int(args.display_lines or -1))
def on_message(ws, msg):
logger.debug("on_message: %s", msg)
writer.print(msg)
def on_error(ws, err):
logger.error("on_error: %s", err)
def on_close(ws, code, msg):
logger.info("on_close (%d): %s", code, msg)
cli.subscribe_user_data(on_message, on_error, on_close)
elif args.streams:
# Subscribe streams
writer = CircularBufferWriter(buffer_line_count=int(args.display_lines or -1))
def on_message(ws, msg):
logger.debug("Recv: %s", msg)
writer.print(msg)
def on_error(ws, err):
logger.error("Error: %s", err)
def on_close(ws, code, msg):
logger.info("Closed (%d): %s", code, msg)
cli.subscribe_streams(
streams=args.streams if isinstance(args.streams, list) else [args.streams],
on_message=on_message,
on_error=on_error,
on_close=on_close,
)
else:
sys.stderr.write(
"No data requested (run with --streams <stream1> <stream2> ... or --exch-info or --list-symbols)"
)
return 1
return 0
if __name__ == "__main__":
code = _main()
exit(code)
@@ -0,0 +1,100 @@
from typing import Any, Mapping, Optional
import dotenv
import os
def required_env_var(key):
val = os.environ.get(key)
if val is None:
raise ValueError(f"{key} env var not defined")
return val
def expand_env(s: str, vars=os.environ):
if not s:
return ""
beg = 0
while True:
beg = s.find("${", beg)
if beg == -1 or beg > len(s) - 4:
break
end = s.find("}", beg + 2)
if end == -1:
break
k = s[beg:end]
v = vars.get(k) or ""
s = s[:beg] + v + s[end + 1 :]
beg += len(v) + 1
return s
def expand_var(req_params: str, k: str, v: Any):
ku = str(k).upper()
substitution = f"${{{ku}}}"
pos = req_params.find(substitution)
end_pos = -1
if pos == -1:
substitution = f"${{{ku}:"
pos = req_params.find(substitution)
if pos == -1:
return req_params
end_pos = req_params.find("}", pos + len(substitution))
if end_pos == -1:
return req_params
end_pos += 1
else:
end_pos = pos + len(substitution)
if end_pos == -1:
raise ValueError(f"Invalid request params template '{req_params}'")
req_params = req_params[:pos] + str(v) + req_params[end_pos:]
return req_params
def expand_vars(req_params: str, key_vals: Mapping):
for k, v in key_vals.items():
req_params = expand_var(req_params, k, v)
return req_params
def expand_defaults(req_params: str):
i = 0
prefix = ":-"
while i < len(req_params):
start = req_params.find("${", i)
if start == -1:
break
stop = req_params.find("}", start)
if stop == -1:
raise ValueError(f"Invalid param str {req_params}")
prefix_pos = req_params.find(prefix, start, stop)
if prefix_pos < start:
req_params = req_params[:start] + req_params[stop + 1 :]
i = start
continue
default_val = req_params[prefix_pos + len(prefix) : stop]
if default_val == "OMIT":
i = req_params.rfind("&", 0, prefix_pos - 1)
if i == -1:
i = 0
req_params = req_params[:i] + req_params[stop + 1 :]
else:
req_params = req_params[:start] + default_val + req_params[stop + 1 :]
i = start + len(default_val) + 1
if req_params.startswith("&"):
req_params = req_params[1:]
return req_params
def load_env(env_path: Optional[str] = None):
if not env_path:
env_path = os.path.join(
os.path.dirname(os.path.realpath(__file__)), "binance.env"
)
dotenv.load_dotenv(env_path, interpolate=False)
if __name__ == "__main__":
req_params = required_env_var("BINANCE_CREATE_ORDER_PARAMS")
res = expand_var(req_params, "SYMBOL", "BTCUSDT")
res = expand_defaults(res)
print(res)
@@ -0,0 +1,242 @@
import dotenv
import json
import logging
import math
import sys
import traceback
from typing import Any, Optional
import uuid
import websocket
from binance_client import BinanceClient
import python_interop_module as pyinter
BINANCE_API_KEY = "2RJ9H02ztFGWxeY2oRpnFYZNU8mki4V1yMvKrDU08HCiEhO4VLe2ePSUFqEYTNxc"
BINANCE_USDS_FUTURES_API_KEY = (
"b838c108cd2648e34a4f088a7fc635a9cbccf3c2faa6bcaabcb532ffacc087b0"
)
BINANCE_USDS_FUTURES_SECRET_KEY = (
"a68f0ab27403c42fa0306b6341b2ddd51d99ac42916f849404155449457d0492"
)
def to_bts_exec_type(exec_type: str, ord_status: str):
match exec_type:
case "TRADE_PREVENTION":
return "REJECTED"
case "NONE":
return "UNDEFINED"
case "TRADE":
return "FILL" if ord_status == "FILLED" else "PARTIAL_FILL"
case _:
return exec_type
def to_bts_order_status(ord_status: str):
match ord_status:
case "EXPIRED_IN_MATCH":
return "EXPIRED"
case _:
return ord_status
class DummyExecutionGateway(pyinter.ExecutionGateway):
def __init__(self):
super(DummyExecutionGateway, self).__init__()
self.bridge = None
self.client = None
self.start_th = None
self.ws = None
self.orders = {}
self.logger = logging.getLogger(__name__)
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.formatter = logging.Formatter(
"%(asctime)s | %(levelname)7s | %(filename)s:%(lineno)s | %(message)s"
)
self.logger.addHandler(stdout_handler)
self.logger.addHandler(
logging.FileHandler(
"/home/bts/sparkdata/data/logging.python_execution/dummy_log"
)
)
self.logger.setLevel(logging.INFO)
env_path = "/workspaces/spark/tools/binance/binance.env"
self.logger.info("DummyExecutionGateway ctr. Loading env")
dotenv.load_dotenv(env_path, interpolate=False)
def _index_transaction(self, bts_key: int):
exch_key = uuid.uuid1().hex
self.orders[exch_key] = bts_key
return exch_key
def _on_user_data_msg_core(self, ws: websocket.WebSocket, msg: Any):
self.logger.info("Recv usr data: %s", str(msg))
if not msg:
self.logger.error("Recv empty usr data msg")
return
if not self.bridge:
self.logger.error("Cannot process usr data. Bridge not created")
return
upd = json.loads(str(msg))
if "e" in upd:
if upd["e"] == "executionReport":
# Binance spots update
bts_ord_id = self.orders.get(upd["c"])
is_old_ord = False
if not bts_ord_id:
bts_ord_id = self.orders.get(upd["C"])
if not bts_ord_id:
self.logger.warning(
"Received update with unknown order id: %s", msg
)
return
is_old_ord = True
er = pyinter.PyExecutionReport()
er.set_key(bts_ord_id)
er.set_venue("BINANCE")
er.set_exchange_security_id(upd["s"])
er.set_security_type("SPOT")
er.set_exec_type(to_bts_exec_type(upd["x"], upd["X"]))
er.set_is_buy(upd["S"] == "BUY")
er.set_price(float(upd["p"]))
er.set_quantity(float(upd["q"]))
er.set_last_filled_price(upd["L"])
er.set_last_filled_quantity(upd["l"])
total_filled_qty = float(upd["z"])
er.set_filled_quantity(total_filled_qty)
total_filled_notional = float(upd["Z"])
avg_filled_price = (
total_filled_notional / total_filled_qty
if total_filled_qty > 0
else math.nan
)
er.set_avg_filled_price(avg_filled_price)
er.set_order_status(to_bts_order_status(upd["X"]))
er.set_order_type(upd["o"])
match upd["X"]:
case "CANCELED":
if not is_old_ord:
del self.orders[bts_ord_id]
case "FILLED":
del self.orders[bts_ord_id]
case "REJECTED":
del self.orders[bts_ord_id]
self.bridge.on_execution_report(er)
elif upd["e"] == "ORDER_TRADE_UPDATE":
data = upd["o"]
# Binance Usds Futures order update
bts_ord_id = self.orders.get(data["c"])
is_old_ord = False
if not bts_ord_id:
bts_ord_id = self.orders.get(data["C"])
if not bts_ord_id:
self.logger.warning(
"Received dataate with unknown order id: %s", msg
)
return
is_old_ord = True
def parse_float(s: Optional[str], default_val=math.nan):
if s is not None:
try:
return float(s)
except Exception:
pass
return default_val
er = pyinter.PyExecutionReport()
er.set_key(bts_ord_id)
er.set_venue("BINANCE_USDS_FUTURES")
er.set_security_type("FUTURE")
er.set_time_in_force("GTC")
er.set_exchange_order_id(str(data.get("i") or 0))
# er.set_exchange_order_id(data["i"])
er.set_exchange_security_id(data["s"])
er.set_exec_type(to_bts_exec_type(data["x"], data["X"]))
er.set_is_buy(data["S"] == "BUY")
er.set_price(parse_float(data["p"]))
er.set_quantity(parse_float(data["q"]))
er.set_last_filled_price(parse_float(data.get("L")))
er.set_last_filled_quantity(parse_float(data["l"]))
er.set_filled_quantity(parse_float(data["z"]))
er.set_avg_filled_price(parse_float(data["ap"]))
er.set_order_status(to_bts_order_status(data["X"]))
er.set_order_type(data["o"])
match data["X"]:
case "CANCELED":
if not is_old_ord:
del self.orders[bts_ord_id]
case "FILLED":
del self.orders[bts_ord_id]
case "REJECTED":
del self.orders[bts_ord_id]
print("Firing this exe report: %s" % str(er), flush=True)
self.bridge.on_execution_report(er)
def _on_user_data_msg(self, ws: websocket.WebSocket, msg: Any):
try:
self._on_user_data_msg_core(ws, msg)
except Exception:
self.logger.error("on_user_data_msg. %s" % traceback.format_exc())
def _on_user_data_open(self, ws: websocket.WebSocket):
self.ws = ws
if self.bridge:
self.bridge.on_is_ready(True)
def _on_user_data_err(self, ws: websocket.WebSocket, err: str):
self.logger.error("on_user_data_err: %s", err)
def _on_user_data_close(self, ws: websocket.WebSocket, code: int, msg: str):
self.logger.warning("on_user_data_close: %s %s", str(code), msg)
if self.bridge:
self.bridge.on_is_ready(False)
def start(self, bridge: pyinter.ExecutionGatewayBridge):
print("DummyExecutionGateway.start")
self.bridge = bridge
self.client = BinanceClient(venue="BINANCE_USDS_FUTURES")
print("DummyExecutionGateway.start. Subscribing to binance user data")
self.client.subscribe_user_data(
on_message=self._on_user_data_msg,
on_open=self._on_user_data_open,
on_close=self._on_user_data_close,
)
print("DummyExecutionGateway.start. Subscribed to binance user data")
def cancel_order(self, req: pyinter.PyCancelOrderRequest):
if not self.client:
self.logger.error("Cannot cancel order. BinanceClient not created.")
return
self.logger.info("[cancel_order.json]%s", str(req))
try:
self.client.request(
"CANCEL_ORDER",
symbol=req.exchange_security_id(),
orig_client_order_id=req.exchange_order_id(),
new_client_order_id=self._index_transaction(req.key()),
)
except Exception as ex:
self.logger.error("Create order request threw. %s" % str(ex))
def create_order(self, req: pyinter.PyCreateOrderRequest):
if not self.client:
self.logger.error("Cannot create order. BinanceClient not created.")
return
print("hey bud")
self.logger.info("[create_order.json]%s", str(req))
side = "BUY" if req.is_buy() else "SELL"
try:
self.client.request(
op="CREATE_ORDER",
symbol=req.exchange_security_id(),
price=req.price(),
quantity=f"{req.quantity():.6f}",
side=side,
time_in_force="GTC",
type="LIMIT",
new_client_order_id=self._index_transaction(req.key()),
)
except Exception as ex:
self.logger.error("Create order request threw. %s" % str(ex))
@@ -0,0 +1,49 @@
import json
from typing import Optional
from pathlib import Path
test_data_dir = Path("/").joinpath(
"home",
"bts",
".local",
"share",
"bts",
"sparkd",
"data",
"test_data",
"binance",
)
def load_securities() -> list:
with open(test_data_dir.joinpath("exch_info.json")) as f:
return json.load(f)["symbols"]
return []
def load_books() -> list:
with open(test_data_dir.joinpath("books.json")) as f:
return json.load(f)["books"]
return []
if __name__ == "__main__":
raw_secs = load_securities()
for s in raw_secs:
obj = {
"venue": "Binance",
"name": s["symbol"],
"asset": s["baseAsset"],
"group": s["baseAsset"],
"currency": s["quoteAsset"],
}
print(json.dumps(obj))
raw_books = load_books()
for b in raw_books:
obj = {
"name": b["name"],
"bids": b["bids"],
"asks": b["asks"],
}
print(json.dumps(obj))
@@ -0,0 +1,161 @@
import python_interop_module as pyinter
import json
from typing import Optional
from pathlib import Path
test_data_dir = Path("/").joinpath(
"home",
"bts",
".local",
"share",
"bts",
"sparkd",
"data",
"test_data",
"binance",
)
def load_securities() -> list:
with open(test_data_dir.joinpath("exch_info.json")) as f:
return json.load(f)["symbols"]
return []
def load_books() -> list:
with open(test_data_dir.joinpath("books.json")) as f:
return json.load(f)["books"]
return []
def _get_security(
venue: str,
name: str,
asset: str,
group: str,
currency: str,
expiration: Optional[str] = None,
option_kind: Optional[str] = None,
strike: Optional[float] = None,
):
sec = pyinter.PySecurity()
sec.set_venue(venue)
sec.set_exchange_id(name)
sec.set_name(name)
sec.set_currency(currency)
sec.set_asset(asset)
sec.set_group(group)
is_spot = expiration is None and option_kind is None and strike is None
is_future = expiration is not None and option_kind is None and strike is None
is_option = (
expiration is not None and option_kind is not None and strike is not None
)
if is_spot:
sec.set_spot(pyinter.PySpot())
elif is_future:
fut = pyinter.PyFuture()
fut.set_expiration(expiration)
sec.set_future(fut)
elif is_option:
opt = pyinter.PyOption()
opt.set_expiration(expiration)
opt.set_is_call(option_kind.lower().startswith("c"))
opt.set_strike(strike)
sec.set_option(opt)
else:
raise ValueError("Invalid parameters (unable to deduce security type)")
return sec
def _get_book_side(levels: list[(float, float)], is_bid_side: bool):
book_side = pyinter.PyBookSide()
book_side.set_is_ascending(not is_bid_side)
for price, qty in levels:
book_level = pyinter.PyBookLevel()
book_level.set_price(price)
book_level.set_quantity(qty)
book_side.set_level(book_level)
return book_side
def _get_trade(
venue: str,
security_type: str,
exchange_security_id: str,
price: float,
quantity: float,
aggressor_side: str,
):
trade = pyinter.PyTrade()
trade.set_venue(venue)
trade.set_security_type(security_type)
trade.set_exchange_security_id(exchange_security_id)
trade.set_price(price)
trade.set_quantity(quantity)
trade.set_aggressor_side(aggressor_side)
return trade
class BinanceTestMdProvider(pyinter.MdProvider):
def __init__(self):
super(BinanceTestMdProvider, self).__init__()
pass
def start(self, bridge: pyinter.MdBridge):
# start subscriptions, bind callbacks etc.
def add_securities():
sec_list = pyinter.PySecurityList()
raw_secs = load_securities()
for s in raw_secs:
sec_list.append(
_get_security(
venue="Binance",
name=s["symbol"],
asset=s["baseAsset"],
group=s["baseAsset"],
currency=s["quoteAsset"],
)
)
bridge.on_securities(sec_list)
def add_books():
books = load_books()
for b in books:
bids = _get_book_side(
[(lvl["price"], lvl["quantity"]) for lvl in b["bids"]],
is_bid_side=True,
)
asks = _get_book_side(
[(lvl["price"], lvl["quantity"]) for lvl in b["asks"]],
is_bid_side=False,
)
book = pyinter.PyBook()
book.set_venue("Binance")
book.set_security_type("SPOT")
book.set_exchange_security_id(b["name"])
book.set_bids(bids)
book.set_asks(asks)
# pass book to bridge
book_list = pyinter.PyBookList()
book_list.append(book)
bridge.on_books(book_list)
add_securities()
add_books()
def stop(self):
print("Stopping python DummyMdProvider")
if __name__ == "__main__":
raw_secs = load_securities()
for s in raw_secs:
obj = {
"venue": "Binance",
"name": s["symbol"],
"asset": s["baseAsset"],
"group": s["baseAsset"],
"currency": s["quoteAsset"],
}
print(json.dump(obj))
@@ -0,0 +1,484 @@
# Bybit API configuration
BYBIT_API_KEY = 'api_key'
BYBIT_API_SECRET = 'api_secret'
BYBIT_TESTNET = True # Set to False for mainnet
WS_URL = "wss://stream-testnet.bybit.com/v5/public/linear"
CONTRACTS = ["BTCUSDT", "ETHUSDT", "BNBUSDT", "XRPUSDT", "ADAUSDT", "PRIMEUSDT"]
# CONTRACTS = [
# '1000000BABYDOGEUSDT',
# '1000000CHEEMSUSDT',
# '1000000MOGUSDT',
# '1000000PEIPEIUSDT',
# '10000COQUSDT',
# '10000ELONUSDT',
# '10000LADYSUSDT',
# '10000QUBICUSDT',
# '10000SATSUSDT',
# '10000WENUSDT',
# '10000WHYUSDT',
# '1000APUUSDT',
# '1000BONKPERP',
# '1000BONKUSDT',
# '1000BTTUSDT',
# '1000CATSUSDT',
# '1000CATUSDT',
# '1000FLOKIUSDT',
# '1000LUNCUSDT',
# '1000MUMUUSDT',
# '1000NEIROCTOUSDT',
# '1000PEPEPERP',
# '1000PEPEUSDT',
# '1000RATSUSDT',
# '1000TOSHIUSDT',
# '1000TURBOUSDT',
# '1000XECUSDT',
# '1000XUSDT',
# '1INCHUSDT',
# 'A8USDT',
# 'AAVEPERP',
# 'AAVEUSDT',
# 'ACEUSDT',
# 'ACHUSDT',
# 'ACTUSDT',
# 'ACXUSDT',
# 'ADAUSDT',
# 'AERGOUSDT',
# 'AEROUSDT',
# 'AEVOPERP',
# 'AEVOUSDT',
# 'AGIUSDT',
# 'AGLDUSDT',
# 'AI16ZUSDT',
# 'AIOZUSDT',
# 'AIUSDT',
# 'AIXBTUSDT',
# 'AKTUSDT',
# 'ALCHUSDT',
# 'ALEOUSDT',
# 'ALICEUSDT',
# 'ALPHAUSDT',
# 'ALTUSDT',
# 'ALUUSDT',
# 'ANIMEUSDT',
# 'ANKRUSDT',
# 'APEUSDT',
# 'APEXUSDT',
# 'API3USDT',
# 'APTUSDT',
# 'ARBPERP',
# 'ARCUSDT',
# 'ARKMUSDT',
# 'ARKUSDT',
# 'ARPAUSDT',
# 'ARUSDT',
# 'ASTRUSDT',
# 'ATAUSDT',
# 'ATHUSDT',
# 'AUCTIONUSDT',
# 'AUDIOUSDT',
# 'AVAAIUSDT',
# 'AVAILUSDT',
# 'AVAUSDT',
# 'AVLUSDT',
# 'AXLUSDT',
# 'AXSUSDT',
# 'B3USDT',
# 'BABYUSDT',
# 'BADGERUSDT',
# 'BAKEUSDT',
# 'BALUSDT',
# 'BANANAS31USDT',
# 'BANANAUSDT',
# 'BANDUSDT',
# 'BANKUSDT',
# 'BANUSDT',
# 'BATUSDT',
# 'BBUSDT',
# 'BCHPERP',
# 'BCHUSDT',
# 'BEAMUSDT',
# 'BELUSDT',
# 'BERAUSDT',
# 'BIGTIMEUSDT',
# 'BIOUSDT',
# 'BLASTUSDT',
# 'BLURUSDT',
# 'BMTUSDT',
# 'BNBUSDT',
# 'BNTUSDT',
# 'BOBAUSDT',
# 'BOMEUSDT',
# 'BRETTUSDT',
# 'BROCCOLIUSDT',
# 'BRUSDT',
# 'BSVUSDT',
# 'BTCPERP',
# 'BTCUSDT',
# 'C98USDT',
# 'CAKEUSDT',
# 'CARVUSDT',
# 'CATIUSDT',
# 'CELOUSDT',
# 'CELRUSDT',
# 'CETUSUSDT',
# 'CFXUSDT',
# 'CGPTUSDT',
# 'CHESSUSDT',
# 'CHILLGUYUSDT',
# 'CHZUSDT',
# 'CLANKERUSDT',
# 'CLOUDUSDT',
# 'COMPUSDT',
# 'COOKIEUSDT',
# 'COOKUSDT',
# 'COREUSDT',
# 'COSUSDT',
# 'COTIUSDT',
# 'COWUSDT',
# 'CPOOLUSDT',
# 'CROUSDT',
# 'CRVPERP',
# 'CRVUSDT',
# 'CTKUSDT',
# 'CTSIUSDT',
# 'CVCUSDT',
# 'CVXUSDT',
# 'DARKUSDT',
# 'DASHUSDT',
# 'DATAUSDT',
# 'DBRUSDT',
# 'DEEPUSDT',
# 'DEGENUSDT',
# 'DENTUSDT',
# 'DEXEUSDT',
# 'DGBUSDT',
# 'DODOUSDT',
# 'DOGEPERP',
# 'DOGEUSDT',
# 'DOGSUSDT',
# 'DOGUSDT',
# 'DOODUSDT',
# 'DOTPERP',
# 'DOTUSDT',
# 'DRIFTUSDT',
# 'DUCKUSDT',
# 'DUSKUSDT',
# 'DYDXUSDT',
# 'DYMUSDT',
# 'EDUUSDT',
# 'EGLDUSDT',
# 'ELXUSDT',
# 'ENAPERP',
# 'ENAUSDT',
# 'ENJUSDT',
# 'ENSUSDT',
# 'EOSUSDT',
# 'EPICUSDT',
# 'EPTUSDT',
# 'ETCUSDT',
# 'ETHBTCUSDT',
# 'ETHFIPERP',
# 'ETHFIUSDT',
# 'ETHPERP',
# 'ETHUSDT',
# 'ETHWUSDT',
# 'FARTCOINUSDT',
# 'FBUSDT',
# 'FHEUSDT',
# 'FIDAUSDT',
# 'FILUSDT',
# 'FIOUSDT',
# 'FLMUSDT',
# 'FLOCKUSDT',
# 'FLOWUSDT',
# 'FLRUSDT',
# 'FLUXUSDT',
# 'FORMUSDT',
# 'FORTHUSDT',
# 'FOXYUSDT',
# 'FTNUSDT',
# 'FUELUSDT',
# 'FUSDT',
# 'FWOGUSDT',
# 'FXSUSDT',
# 'GALAUSDT',
# 'GASUSDT',
# 'GIGAUSDT',
# 'GLMRUSDT',
# 'GLMUSDT',
# 'GMTUSDT',
# 'GMXUSDT',
# 'GNOUSDT',
# 'GOATUSDT',
# 'GODSUSDT',
# 'GOMININGUSDT',
# 'GORKUSDT',
# 'GPSUSDT',
# 'GRIFFAINUSDT',
# 'GRTUSDT',
# 'GUNUSDT',
# 'GUSDT',
# 'HAEDALUSDT',
# 'HBARUSDT',
# 'HEIUSDT',
# 'HFTUSDT',
# 'HIFIUSDT',
# 'HIGHUSDT',
# 'HIPPOUSDT',
# 'HIVEUSDT',
# 'HMSTRUSDT',
# 'HNTUSDT',
# 'HOOKUSDT',
# 'HOTUSDT',
# 'HPOS10IUSDT',
# 'HYPEPERP',
# 'HYPERUSDT',
# 'HYPEUSDT',
# 'ICPUSDT',
# 'ICXUSDT',
# 'IDEXUSDT',
# 'IDUSDT',
# 'ILVUSDT',
# 'IMXUSDT',
# 'INITUSDT',
# 'INJUSDT',
# 'IOSTUSDT',
# 'IOTAUSDT',
# 'IOTXUSDT',
# 'IOUSDT',
# 'IPUSDT',
# 'JASMYUSDT',
# 'JELLYJELLYUSDT',
# 'JOEUSDT',
# 'JSTUSDT',
# 'JTOUSDT',
# 'JUPUSDT',
# 'JUSDT',
# 'KAIAUSDT',
# 'KAITOUSDT',
# 'KASUSDT',
# 'KAVAUSDT',
# 'KDAUSDT',
# 'KERNELUSDT',
# 'KMNOUSDT',
# 'KNCUSDT',
# 'KOMAUSDT',
# 'KSMUSDT',
# 'L3USDT',
# 'LAUNCHCOINUSDT',
# 'LDOUSDT',
# 'LEVERUSDT',
# 'LINKPERP',
# 'LINKUSDT',
# 'LISTAUSDT',
# 'LOOKSUSDT',
# 'LPTUSDT',
# 'LQTYUSDT',
# 'LRCUSDT',
# 'LSKUSDT',
# 'LTCPERP',
# 'LUMIAUSDT',
# 'LUNA2USDT',
# 'MAGICUSDT',
# 'MAJORUSDT',
# 'MANAUSDT',
# 'MANTAUSDT',
# 'MASAUSDT',
# 'MASKUSDT',
# 'MAVIAUSDT',
# 'MAVUSDT',
# 'MBLUSDT',
# 'MBOXUSDT',
# 'MDTUSDT',
# 'MELANIAUSDT',
# 'MEMEUSDT',
# 'MERLUSDT',
# 'METISUSDT',
# 'MEUSDT',
# 'MEWUSDT',
# 'MICHIUSDT',
# 'MILKUSDT',
# 'MINAUSDT',
# 'MKRUSDT',
# 'MLNUSDT',
# 'MNTPERP',
# 'MNTUSDT',
# 'MOBILEUSDT',
# 'MOCAUSDT',
# 'MOODENGUSDT',
# 'MORPHOUSDT',
# # 'MOTHER1USDT',
# # 'MOTHER2USDT',
# # 'MOTHER3USDT',
# 'MOVEUSDT',
# 'MOVRUSDT',
# 'MTLUSDT',
# 'MUBARAKUSDT',
# 'MVLUSDT',
# 'MYRIAUSDT',
# 'NCUSDT',
# 'NEARUSDT',
# 'NEIROETHUSDT',
# 'NEOUSDT',
# 'NFPUSDT',
# 'NILUSDT',
# 'NKNUSDT',
# 'NMRUSDT',
# 'NOTPERP',
# 'NOTUSDT',
# 'NSUSDT',
# 'NTRNUSDT',
# 'NXPCUSDT',
# 'OBOLUSDT',
# 'OBTUSDT',
# 'OGUSDT',
# # 'OKBUSDT',
# 'OLUSDT',
# 'OMGUSDT',
# 'OMUSDT',
# 'ONDOPERP',
# 'ONDOUSDT',
# 'ONEUSDT',
# 'ONGUSDT',
# 'ONTUSDT',
# 'OPUSDT',
# 'ORBSUSDT',
# 'ORCAUSDT',
# 'ORDERUSDT',
# 'ORDIPERP',
# 'ORDIUSDT',
# 'OSMOUSDT',
# 'PARTIUSDT',
# 'PAXGUSDT',
# 'PEAQUSDT',
# 'PENDLEUSDT',
# 'PEOPLEUSDT',
# # 'PERPUSDT',
# 'PHAUSDT',
# 'PHBUSDT',
# 'PIPPINUSDT',
# 'PIXELUSDT',
# 'PLUMEUSDT',
# 'PNUTUSDT',
# 'POLPERP',
# 'POLUSDT',
# 'POLYXUSDT',
# 'PONKEUSDT',
# 'POPCATPERP',
# 'POPCATUSDT',
# 'PORTALUSDT',
# 'POWRUSDT',
# 'PRAIUSDT',
# 'PRCLUSDT',
# # 'PRE05EOSUSDT',
# # 'PRE08EOSUSDT',
# 'PRIMEUSDT',
# 'PROMPTUSDT',
# 'PROMUSDT',
# 'PUFFERUSDT',
# 'PUMPUSDT',
# 'PUNDIXUSDT',
# 'PYRUSDT',
# 'PYTHUSDT',
# 'QIUSDT',
# 'QNTUSDT',
# 'QTUMUSDT',
# 'QUICKUSDT',
# 'RADUSDT',
# 'RAREUSDT',
# 'RAYDIUMUSDT',
# 'RAYUSDT',
# 'RDNTUSDT',
# 'REDUSDT',
# 'RENDERUSDT',
# 'REXUSDT',
# 'REZUSDT',
# 'RFCUSDT',
# 'RIFUSDT',
# 'RLCUSDT',
# 'ROAMUSDT',
# 'RONINUSDT',
# 'ROSEUSDT',
# 'RPLUSDT',
# 'RSRUSDT',
# 'RUNEUSDT',
# 'SAGAUSDT',
# 'SANDUSDT',
# 'SCAUSDT',
# 'SDUSDT',
# 'SENDUSDT',
# 'SERAPHUSDT',
# 'SFPUSDT',
# 'SHELLUSDT',
# 'SHIB1000PERP',
# 'SHIB1000USDT',
# 'SIGNUSDT',
# 'SIRENUSDT',
# 'SKLUSDT',
# 'SKYAIUSDT',
# 'SLERFUSDT',
# 'SLFUSDT',
# 'SLPUSDT',
# 'SNTUSDT',
# 'SNXUSDT',
# 'SOLAYERUSDT',
# 'SOLOUSDT',
# 'SOLPERP',
# 'SOLUSDT',
# 'SOLVUSDT',
# 'SONICUSDT',
# 'SPECUSDT',
# 'SPXUSDT',
# 'SSVUSDT',
# 'STEEMUSDT',
# 'STGUSDT',
# 'STORJUSDT',
# 'STOUSDT',
# 'STRKPERP',
# 'STRKUSDT',
# 'SUIPERP',
# 'SUIUSDT',
# 'SUNDOGUSDT',
# 'SUNUSDT',
# 'SUPERUSDT',
# 'SUSDT',
# 'SUSHIUSDT',
# 'SWARMSUSDT',
# 'SWEATUSDT',
# 'SWELLUSDT',
# 'SXPUSDT',
# 'SYNUSDT',
# 'SYRUPUSDT',
# 'SYSUSDT',
# 'TAIKOUSDT',
# 'TAIUSDT',
# 'TAOUSDT',
# 'THETAUSDT',
# 'THEUSDT',
# 'TIAPERP',
# 'TIAUSDT',
# 'TLMUSDT',
# 'TOKENUSDT',
# 'TONPERP',
# 'TRBUSDT',
# 'TRUMPPERP',
# 'TRUMPUSDT',
# 'TRUUSDT',
# 'TRXUSDT',
# 'TSTBSCUSDT',
# 'TUSDT',
# 'TUTUSDT',
# 'TWTUSDT',
# 'UMAUSDT',
# 'UNIPERP',
# 'UNIUSDT',
# 'USDCUSDT',
# 'USDEUSDT',
# 'USTCUSDT',
# 'USUALUSDT',
# 'UXLINKUSDT',
# 'VANAUSDT'
# ]
@@ -2,9 +2,8 @@ import json
import time
from typing import Optional
import uuid
from pybit_unified_trading import WebSocket, WebSocketTrading
from pybit.unified_trading import WebSocket, WebSocketTrading
from spark_core import (
JsonElement,
as_bool,
configure_logger,
format_price,
@@ -24,32 +23,14 @@ from python_interop_module import (
PySecurity,
PySecurityList,
)
from spark_enums import OrderStatus, ExecType, OrderType, SecurityType, TimeInForce
from spark_enums import OrderStatus, ExecType
import threading
class OrderData:
def __init__(
self,
account: str,
bts_id: int,
exch_id: str,
exch_sec_id: str,
sec_type: SecurityType,
is_buy: bool,
order_type: OrderType,
time_in_force: TimeInForce,
price: float,
quantity: float,
):
self.account = account
def __init__(self, bts_id: int, exch_id: str, price: float, quantity: float):
self.bts_id = bts_id
self.exch_id = exch_id
self.exch_sec_id = exch_sec_id
self.sec_type = sec_type
self.is_buy = is_buy
self.order_type = order_type
self.time_in_force = time_in_force
self.price = price
self.quantity = quantity
self.filled_quantity: float = 0
@@ -59,53 +40,6 @@ class OrderData:
self.exec_type: ExecType = ExecType.PENDING_NEW
self.status: OrderStatus = OrderStatus.PENDING_NEW
def to_json(self):
return {
"account": self.account,
"bts_id": self.bts_id,
"exch_id": self.exch_id,
"exch_sec_id": self.exch_sec_id,
"sec_type": self.sec_type.name,
"is_buy": self.is_buy,
"order_type": self.order_type.name,
"time_in_force": self.time_in_force.name,
"price": self.price,
"quantity": self.quantity,
"filled_quantity": self.filled_quantity,
"last_filled_quantity": self.last_filled_quantity,
"avg_filled_price": self.avg_filled_price,
"last_filled_price": self.last_filled_price,
"exec_type": self.exec_type.name,
"status": self.status.name,
}
def to_execution_report(self):
transact_time = time.time_ns()
er = PyExecutionReport()
er.set_data(
key=self.bts_id,
exchange_order_id=self.exch_id,
venue="BYBIT",
security_type=self.sec_type.name,
exchange_security_id=self.exch_sec_id,
account=self.account,
order_type=self.order_type.name,
is_buy=self.is_buy,
price=self.price,
quantity=self.quantity,
time_in_force=self.time_in_force.name,
exec_type=self.exec_type.name,
order_status=self.status.name,
last_filled_price=self.last_filled_price,
last_filled_quantity=self.last_filled_quantity,
filled_quantity=self.filled_quantity,
avg_filled_price=self.avg_filled_price,
# There is no transact_id in Bybit, so we use updatedTime
transact_id=str(transact_time),
transact_time=transact_time,
)
return er
def on_update(
self,
price: float,
@@ -148,7 +82,6 @@ class OrderData:
self.status = OrderStatus.REJECTED
case _:
raise ValueError(f"Unexpected order status '{status}'")
return self.to_execution_report()
class BybitExecutionGateway(ExecutionGateway):
@@ -156,7 +89,7 @@ class BybitExecutionGateway(ExecutionGateway):
super(BybitExecutionGateway, self).__init__()
name = "bybit_1"
self.name = name
cfg = load_config([f"py_{name}"])
cfg = load_config(name)
self.logger = configure_logger(
logging.getLogger(name), "OrderMan", **(cfg.get("logging") or {})
)
@@ -179,53 +112,10 @@ class BybitExecutionGateway(ExecutionGateway):
return v
return None
def _order_handlers(self, typ: str, exch_id: str):
def get_error_desc(msg: dict):
return "%s (code %s)" % (
(msg.get("retMsg") or "Unknown", msg.get("retCode") or "Unknown")
)
def on_err(msg: dict):
self.logger.error(
'[bybit_gw_ord_ack_err.json]{ "typ": "%s", "exch_id": "%s", "err": %s, "known_orders": %s }'
% (
typ,
exch_id,
JsonElement(msg),
JsonElement({k: v.to_json() for k, v in self.orders.items()}),
)
)
req_id = msg.get("reqId")
if not req_id:
self.logger.error(
'[bybit_gw_ord_ack_err.json]{ "typ": "%s", "exch_id": "%s", "err": "%s" }'
% (typ, exch_id, msg)
)
ord_data = self._get_order_data(exch_id)
if ord_data:
er = ord_data.to_execution_report()
match er.order_status():
case OrderStatus.PENDING_NEW.name:
er.set_order_status(OrderStatus.REJECTED.name)
er.set_exec_type(ExecType.REJECTED.name)
er.set_error_msg(get_error_desc(msg))
self.bridge.on_execution_report(er)
case _:
pass
else:
self.logger.error(
'[bybit_gw_ord_ack_err.json]{"typ": "%s", "exch_id": "%s", "err": "Unknown exch id" }'
)
def on_ack(msg: dict):
ord_data = self._get_order_data(exch_id)
if ord_data is None:
self.logger.error(
'[bybit_gw_ord_ack_err.json]{ "typ": "%s", "exch_id": "%s", "err": "No order data found with exch id", "msg": "%s" }'
% (typ, exch_id, msg)
)
return (on_ack, on_err)
def _on_order_ack(self, typ, msg: object):
self.logger.info(
'[bybit_gw_ord_ack.json]{ "typ": "%s", "msg": %s}' % (typ, msg)
)
def _on_abort(
self,
@@ -303,10 +193,9 @@ class BybitExecutionGateway(ExecutionGateway):
"api_secret": self.secret_key,
"testnet": self.use_testnet,
"channel_type": "private",
"private_auth_expire": 5,
"trace_logging": False,
}
self.logger.info("[bybit_gw_stream_sub.json]%s" % JsonElement(params))
self.logger.info("[bybit_gw_stream_sub.json]%s" % json.dumps(params))
self.stream_cli = WebSocket(**params)
self.stream_cli.order_stream(on_order_update)
self.stream_cli.execution_stream(on_exe_report)
@@ -315,30 +204,17 @@ class BybitExecutionGateway(ExecutionGateway):
"api_key": self.api_key,
"api_secret": self.secret_key,
"testnet": self.use_testnet,
"private_auth_expire": 5,
}
self.logger.info("[bybit_gw_start_trd_cli.json]%s" % JsonElement(params))
self.logger.info("[bybit_gw_start_trd_cli.json]%s" % json.dumps(params))
self.trade_cli = WebSocketTrading(**params)
self.bridge.on_is_ready(True)
# Sleep loop to keep thread alive and detect connection status changes
conn_statuses = {
"stream": self.stream_cli.is_connected(),
"trade": self.trade_cli.is_connected(),
params = {
"category": "linear",
}
while not self.stop_event.is_set():
time.sleep(1)
new_conn_statuses = {
"stream": self.stream_cli.is_connected(),
"trade": self.trade_cli.is_connected(),
}
if conn_statuses != new_conn_statuses:
conn_statuses = new_conn_statuses
all_connected = all(map(lambda kv: kv[1], conn_statuses.items()))
self.logger.log(
logging.INFO if all_connected else logging.WARNING,
"[bybit_gw_connection_status.json]%s" % SLog(**conn_statuses),
)
self.bridge.on_is_ready(all_connected)
self.bridge.on_is_ready(True)
while True:
if not self.stream_cli.is_connected():
# try reconnecting
pass
def start(self, bridge: ExecutionGatewayBridge):
def start_wrapper():
@@ -349,7 +225,7 @@ class BybitExecutionGateway(ExecutionGateway):
'[bybit_gw_start_err.json]{ "err": "%s" }' % str(e), exc_info=True
)
self.logger.info("[bybit_gw_start.json]%s" % SLog())
self.logger.info("[bybit_gw_start.json]%s" % json.dumps({"ts": time.time_ns()}))
self.bridge = bridge
threading.Thread(
target=start_wrapper,
@@ -358,11 +234,11 @@ class BybitExecutionGateway(ExecutionGateway):
).start()
def stop(self):
self.logger.info("[bybit_gw_stop.json]%s" % SLog())
self.logger.info("[bybit_gw_stop.json]%s" % json.dumps({"ts": time.time_ns()}))
self.stop_event.set()
def create_order(self, req: PyCreateOrderRequest):
self.logger.info("[bybit_gw_crt_ord.json]%s" % req)
self.logger.info("[bybit_gw_crt_ord.json]%s" % str(req))
if not self.stream_cli or not self.trade_cli or not self.bridge:
self._on_abort(
req.key(),
@@ -375,35 +251,24 @@ class BybitExecutionGateway(ExecutionGateway):
bts_id = req.key()
px = req.price()
qty = req.quantity()
sec_name = req.exchange_security_id()
self.orders[req.key()] = OrderData(
account=self.account,
bts_id=bts_id,
exch_id=exch_ord_id,
exch_sec_id=req.exchange_security_id(),
sec_type=SecurityType.parse(req.security_type()),
is_buy=req.is_buy(),
order_type=OrderType.parse(req.order_type()),
time_in_force=TimeInForce.parse(req.time_in_force()),
price=px,
quantity=qty,
bts_id=bts_id, exch_id=exch_ord_id, price=px, quantity=qty
)
sec_name = req.exchange_security_id()
sec = self.securities.get(sec_name)
if not sec:
raise ValueError(f"Unknown security '{sec_name}'")
return
params = {
"orderLinkId": exch_ord_id,
"category": "linear",
"symbol": sec_name,
"side": "Buy" if req.is_buy() else "Sell",
"orderType": req.order_type().title(),
"timeInForce": req.time_in_force().upper(),
"price": format_price(sec, px),
"qty": format_quantity(sec, qty),
}
on_ack, on_err = self._order_handlers("crt", exch_ord_id)
self.logger.info("[bybit_gw_crt_ord_send.json]%s" % SLog(params=params))
self.trade_cli.place_order(on_ack, **params, error_callback=on_err)
self.logger.info("[bybit_gw_crt_ord_send.json]%s" % json.dumps(params))
self.trade_cli.place_order(lambda msg: self._on_order_ack("crt", msg), **params)
def cancel_replace_order(self, req: PyCancelReplaceOrderRequest):
self.logger.info("[bybit_gw_cxlrep_ord.json]%s" % req)
@@ -426,12 +291,12 @@ class BybitExecutionGateway(ExecutionGateway):
"price": format_price(sec, req.price()),
"qty": format_quantity(sec, req.quantity()),
}
on_ack, on_err = self._order_handlers("cxlrep", ord_data.exch_id)
self.logger.info("[bybit_gw_cxlrep_ord_send.json]%s" % SLog(params=params))
self.trade_cli.amend_order(on_ack, **params, error_callback=on_err)
self.trade_cli.amend_order(
lambda msg: self._on_order_ack("cxlrep", msg), **params
)
def cancel_order(self, req: PyCancelOrderRequest):
self.logger.info("[bybit_gw_cxl_ord.json]%s" % req)
self.logger.info("[bybit_gw_cxl_ord.json]%s" % str(req))
if not self.stream_cli or not self.trade_cli:
self._on_abort(req.key(), "cxl", "WebSocket client not created.")
return
@@ -445,10 +310,12 @@ class BybitExecutionGateway(ExecutionGateway):
"category": "linear",
"symbol": req.exchange_security_id(),
}
on_ack, on_err = self._order_handlers("cxl", ord_data.exch_id)
self.logger.info("[bybit_gw_cxl_ord_send.json]%s" % SLog(params=params))
self.trade_cli.cancel_order(on_ack, **params, error_callback=on_err)
self.trade_cli.cancel_order(
lambda msg: self._on_order_ack("cxl", msg), **params
)
def on_securities(self, securities: PySecurityList):
self.logger.info("[bybit_gw_sec.json]%s" % securities)
self.logger.info(
"[bybit_gw_sec.json][%s]" % ", ".join([str(sec) for sec in securities])
)
self.securities = {sec.name(): sec for sec in securities}
@@ -1,10 +1,11 @@
import json
import numpy as np
import time
import logging
import sys
import threading
from itertools import chain
from typing import Iterable, Optional
import signal
from typing import Optional
from pybit.unified_trading import WebSocket
from python_interop_module import (
MdBridge,
@@ -14,18 +15,15 @@ from python_interop_module import (
PySecurityList,
)
from spark_core import (
JsonElement,
configure_logger,
get_contract_spec,
get_price_info,
get_quantity_info,
get_security,
get_tick_factors,
load_config,
)
from py_md_utils import PERP_EXP_STRING
import requests
import numpy as np
logger = logging.getLogger("bybit_md_provider")
@@ -34,123 +32,79 @@ def normalize_contract_name(contract: str) -> str:
return contract[:-4] + "_" + contract[-4:]
return contract
def fetch_futures_specs(contracts: Iterable[str], use_testnet: bool = True) -> dict:
# Fetch specs for futures only, explicitly, handling pagination
specs_dict = {}
def fetch_contract_specs(contracts: list[str]) -> dict:
default_specs = {
"tick_unit": 1,
"tick_base": 10,
"cash_per_point": 0.1,
"qty_tick_unit": 1,
"qty_tick_base": 10000
}
specs_dict = {contract: default_specs for contract in contracts}
try:
host = "api-testnet.bybit.com" if use_testnet else "api.bybit.com"
base_url = f"https://{host}/v5/market/instruments-info?category=linear&limit=500"
logger.info(f"Querying Bybit API for futures specs: {base_url}")
url = "https://api.bybit.com/v5/market/instruments-info?category=linear"
response = requests.get(url, timeout=5)
response.raise_for_status()
data = response.json()
found_contracts = set()
cursor = ""
while True:
url = f"{base_url}&cursor={cursor}" if cursor else base_url
logger.info(f"Fetching page with cursor: {cursor or 'initial'}")
response = requests.get(url, timeout=5)
response.raise_for_status()
data = response.json()
if data.get("retCode") != 0:
logger.error(f"Futures API error: {data.get('retMsg', 'Unknown error')}")
return specs_dict
result = data.get("result", {})
symbols = result.get("list", [])
logger.info(f"Loaded page with cursor: {cursor or 'initial'}, symbols: {len(symbols)}")
for item in symbols:
contract = item.get("symbol")
if contract in contracts:
try:
price_tick = float(item.get("priceFilter", {}).get("tickSize", 0.1))
lot_size = item.get("lotSizeFilter", {})
qty_step = float(lot_size.get("qtyStep", 0.001))
if price_tick == 0 or qty_step == 0:
logger.error(f"Invalid tick or lot size for {contract}: tickSize={price_tick}, qtyStep={qty_step}")
continue
tick_unit, tick_base = get_tick_factors(price_tick)
qty_tick_unit, qty_tick_base = get_tick_factors(qty_step)
specs_dict[contract] = {
"tick_unit": tick_unit,
"tick_base": tick_base,
"cash_per_point": 1,
"qty_tick_unit": qty_tick_unit,
"qty_tick_base": qty_tick_base,
"min_notional_value": float(
lot_size.get("minNotionalValue", tick_unit / tick_base)
),
}
found_contracts.add(contract)
logger.info(f"Fetched futures specs for {contract}: {specs_dict[contract]}")
except (ValueError, TypeError) as e:
logger.error(f"Failed to parse futures specs for {contract}: {e}")
continue
cursor = result.get("nextPageCursor", "")
if not cursor:
logger.info("No more pages to fetch; pagination complete")
break
logger.info(f"Returning specs for contracts: {list(specs_dict.keys())}")
logger.debug(f"Final specs: {specs_dict}")
for item in data.get("result", {}).get("list", []):
contract = item.get("symbol")
if contract in contracts:
price_tick = float(item.get("priceFilter", {}).get("tickSize", "0.1"))
lot_size = float(item.get("lotSizeFilter", {}).get("qtyStep", "0.001"))
tick_base = int(1 / price_tick) if price_tick != 0 else 10
if lot_size >= 1:
qty_tick_base = int(lot_size)
else:
qty_tick_base = int(1 / lot_size) if lot_size != 0 else 10000
specs_dict[contract] = {
"tick_unit": 1,
"tick_base": tick_base,
"cash_per_point": price_tick,
"qty_tick_unit": 1,
"qty_tick_base": qty_tick_base
}
found_contracts.add(contract)
for contract in contracts:
if contract not in found_contracts:
logger.warning(f"Contract {contract} not found in API response, using default specs")
return specs_dict
except Exception as e:
logger.error(f"Failed to fetch futures contract specs: {e}", exc_info=True)
logger.error(f"Failed to fetch contract specs: {e}", exc_info=True)
return specs_dict
class BybitMdProvider(MdProvider):
def __init__(self):
super().__init__()
self._config: Optional[dict] = None
self.ws = None
self.ws_thread: Optional[threading.Thread] = None
self.bridge: Optional[MdBridge] = None
self.stop_event = threading.Event()
self._setup_signal_handlers()
@property
def config(self):
if self._config is None:
self._config = load_config(["py_bybit_md_provider", "bybit_overrides"])
return self._config
@property
def use_testnet(self):
return self.config.get("use_testnet", True)
@property
def currencies(self):
return self.config.get("currencies") or ["USDT"]
@property
def futures(self):
return self.config.get("futures", [])
@property
def spots(self):
return self.config.get("spots", [])
def _setup_signal_handlers(self):
"""Set up signal handlers for SIGINT, SIGTERM, and SIGTSTP."""
def signal_handler(sig, frame):
logger.info(f"Received signal {sig}, stopping...")
self.stop()
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
signal.signal(signal.SIGTSTP, signal_handler)
def start_impl(self, bridge: MdBridge):
configure_logger(logger, "md", **self.config.get("logging", {}))
config = load_config("bybit_md_provider")
if not config:
logger.error("bybit_md_provider not configured")
return
configure_logger(logger, "md", **config.get("logging", {}))
self.bridge = bridge
logger.info("Starting BybitMdProvider")
contracts = list(
chain(*[map(lambda f: f"{f}{c}", self.futures) for c in self.currencies])
)
contract_specs = fetch_futures_specs(contracts, use_testnet=self.use_testnet)
contracts: list[str] = config["futures"]
contract_specs = fetch_contract_specs(contracts)
sec_list = PySecurityList()
missing_contracts = list(filter(lambda c: c not in contract_specs, contracts))
if missing_contracts:
logger.error(
"Did not find instrument info for %s" % JsonElement(missing_contracts)
)
contracts = list(filter(lambda c: c in contract_specs, contracts))
for contract in contracts:
normalized_contract = normalize_contract_name(contract)
asset, quote = normalized_contract.split("_")
@@ -166,20 +120,13 @@ class BybitMdProvider(MdProvider):
continue
try:
price_tick_unit = specs.get("tick_unit", 1)
price_tick_base=specs.get("tick_base", 10)
spec = get_contract_spec(
price_info=get_price_info(
tick_unit=price_tick_unit,
tick_base=price_tick_base,
cash_per_point=specs.get("cash_per_point", 0.1),
),
qty_info=qty_info,
notional_qty_info=get_quantity_info(
tick_unit=price_tick_unit,
tick_base=price_tick_base,
min_value=specs.get("min_notional_value"),
tick_unit=specs.get("tick_unit", 1),
tick_base=specs.get("tick_base", 10),
cash_per_point=specs.get("cash_per_point", 0.1)
),
qty_info=qty_info
)
except Exception as e:
logger.error(f"Failed to get contract spec for contract: {contract}, error: {e}")
@@ -208,7 +155,7 @@ class BybitMdProvider(MdProvider):
try:
self.ws = WebSocket(
testnet=self.use_testnet,
testnet=True,
channel_type="linear",
ping_interval=20,
)
@@ -231,7 +178,7 @@ class BybitMdProvider(MdProvider):
logger.debug(f"Thread (start): Name={t.name}, Alive={t.is_alive()}, Daemon={t.daemon}, Target={t._target}")
while not self.stop_event.is_set():
time.sleep(1)
time.sleep(0.05)
except KeyboardInterrupt:
logger.info("KeyboardInterrupt received — shutting down.")
@@ -267,7 +214,7 @@ class BybitMdProvider(MdProvider):
sorted_asks = sorted(raw_asks, key=lambda x: x[0])[:5]
sorted_asks += [(np.float64('nan'), np.float64('nan'))]
logger.debug(f"📘 BOOK DATA for {contract}: bids={sorted_bids} asks={sorted_asks}")
logger.info(f"📘 BOOK DATA for {contract}: bids={sorted_bids} asks={sorted_asks}")
book = PyBook()
book.set_data(
@@ -319,47 +266,3 @@ class BybitMdProvider(MdProvider):
import os
os._exit(0)
if __name__ == "__main__":
import argparse
from fake_md_bridge import FakeMdBridge
ap = argparse.ArgumentParser()
ap.add_argument("--currencies", nargs="+")
ap.add_argument("--futures", nargs="+")
log_levels = list(logging._nameToLevel)
ap.add_argument("--console-level", choices=log_levels)
ap.add_argument("--file-level", choices=log_levels)
ap.add_argument("--dev", action="store_true", dest="use_testnet")
ap.add_argument("--prod", action="store_false", dest="use_testnet")
ap.set_defaults(
currencies=None,
futures=None,
console_level=None,
file_level=None,
use_testnet=None,
)
args = ap.parse_args()
print(args)
logger = logging.getLogger("bybit_md_provider_app")
md_pro = BybitMdProvider()
config = md_pro.config
if args.currencies:
config["currencies"] = args.currencies
if args.futures:
config["futures"] = args.futures
logging_overrides = {}
if args.console_level:
logging_overrides["console_level"] = args.console_level
if args.file_level:
logging_overrides["file_level"] = args.file_level
if logging_overrides:
config["logging"].update(logging_overrides)
if args.use_testnet is not None:
config["use_testnet"] = args.use_testnet
# print(json.dumps(config))
md_bridge = FakeMdBridge(logger)
md_pro.start(md_bridge) # type: ignore
while True:
time.sleep(1)
@@ -1,102 +0,0 @@
import sys
import threading
import time
import traceback
class ConsoleTableColumn:
def __init__(self, id, header: str, data_format=None, width=None):
self.id = id
self.header = header
self.data_format = data_format
self.width = width or len(header)
class ConsoleTable:
ESC = "\033["
def __init__(self, columns: list[ConsoleTableColumn], col_separator=" "):
self.columns = columns
self.col_separator = col_separator
self.header_row = col_separator.join(map(lambda c: c.header, columns))
self._pending_data = None
self._is_running = False
def move_cursor(self, row, col):
sys.stdout.write(f"{ConsoleTable.ESC}{row};{col}H")
def clear_screen(self):
sys.stdout.write(f"{ConsoleTable.ESC}2J")
def hide_cursor(self):
sys.stdout.write(f"{ConsoleTable.ESC}?25l")
def show_cursor(self):
sys.stdout.write(f"{ConsoleTable.ESC}?25h")
def print_table(self, rows):
print(self.header_row)
print("-" * len(self.header_row))
for row in rows:
cells = []
for col in self.columns:
val = row[col.id]
cell = f"{val:{col.data_format}}" if col.data_format else str(val)
len_diff = col.width - len(cell)
if len_diff > 0:
cell += " " * len_diff
else:
leading_ws_count = 0
while (
leading_ws_count < -len_diff and cell[leading_ws_count] == " "
):
leading_ws_count += 1
if leading_ws_count > 0:
cell = cell[leading_ws_count:]
len_diff = col.width - len(cell)
if len_diff < 0:
cell = cell[:len_diff]
cells.append(cell)
print(self.col_separator.join(cells))
def set_data(self, data):
self._pending_data = data
def run(self):
try:
self.clear_screen()
self.hide_cursor()
while self._is_running:
if self._pending_data:
# move to bottom of screen
self.move_cursor(1, 1)
self.print_table(self._pending_data)
self._pending_data = None
sys.stdout.flush()
time.sleep(0.5)
finally:
self.show_cursor()
def run_in_background(self):
def run_wrapper():
self.run()
self._is_running = True
threading.Thread(target=run_wrapper, daemon=True)
if __name__ == "__main__":
cols = [
ConsoleTableColumn("symbol", f"{'Symbol':<10}"),
ConsoleTableColumn("bid", f"{'Bid':>10}", data_format=">10"),
ConsoleTableColumn("ask", f"{'Ask':>10}", data_format=">10"),
]
bbo_data = [
{"symbol": "BTCUSDT", "bid": 104500.00, "ask": 105000.00},
{"symbol": "BNBUSDT", "bid": 645.00, "ask": 650.00},
]
try:
table = ConsoleTable(cols)
table.print_table(bbo_data)
except Exception:
traceback.print_exc()
@@ -1,19 +0,0 @@
import logging
from typing import Optional
class FakeMdBridge:
def __init__(self, logger: Optional[logging.Logger] = logging.getLogger()):
self.logger = logger
def on_books(self, v):
if self.logger:
self.logger.info("[books.json]%s" % v)
def on_securities(self, v):
if self.logger:
self.logger.info("[securities.json]%s" % v)
def on_trades(self, v):
if self.logger:
self.logger.info("[trades.json]%s" % v)
@@ -5,9 +5,8 @@ import logging
import sys
import threading
import requests
from itertools import chain
from websocket import create_connection, WebSocketConnectionClosedException
from typing import Iterable, Optional, Any
from typing import Optional, Any
from python_interop_module import (
MdBridge,
MdProvider,
@@ -15,22 +14,19 @@ from python_interop_module import (
PyBookList,
PySecurityList,
)
from config import WS_URL, CONTRACTS
from spark_core import (
configure_logger,
get_contract_spec,
get_price_info,
get_quantity_info,
get_security,
load_config,
)
# Urls:
# prod perps (ws) - "wss://fx-ws.gateio.ws/v4/ws/usdt"
# testnet perps (ws) - "wss://fx-ws-testnet.gateio.ws/v4/ws/usdt"
logger = logging.getLogger("gateio_md_provider")
configure_logger(logger, "md")
def fetch_contract_specs(contracts: Iterable[str]) -> dict:
def fetch_contract_specs() -> dict:
"""Fetch specs for all contracts in one request."""
default_specs = {
"tick_unit": 1,
@@ -39,7 +35,7 @@ def fetch_contract_specs(contracts: Iterable[str]) -> dict:
"qty_tick_unit": 1,
"qty_tick_base": 10000
}
specs_dict = {contract: default_specs for contract in contracts}
specs_dict = {contract: default_specs for contract in CONTRACTS}
try:
url = "https://api.gateio.ws/api/v4/futures/usdt/contracts"
response = requests.get(url, timeout=5)
@@ -47,7 +43,7 @@ def fetch_contract_specs(contracts: Iterable[str]) -> dict:
data = response.json()
for item in data:
contract = item.get("name")
if contract in contracts:
if contract in CONTRACTS:
price_tick = float(item.get("price_tick", "0.1"))
contract_size = float(item.get("contract_size", "0.0001"))
tick_base = int(1 / price_tick) if price_tick != 0 else 10
@@ -91,12 +87,9 @@ class GateIoMdProvider(MdProvider):
time.sleep(15)
def start_impl(self, bridge: MdBridge):
cfg = load_config(["gateio_md_provider", "gateio_overrides"])
configure_logger(logger, "md", **cfg.get("logging", {}))
self.bridge = bridge
ws_url = cfg["ws_url"]
logger.info(f"Connecting to {ws_url}")
self.ws = create_connection(ws_url)
logger.info(f"Connecting to {WS_URL}")
self.ws = create_connection(WS_URL)
self.ws.settimeout(5)
logger.info("Connected.")
@@ -106,14 +99,11 @@ class GateIoMdProvider(MdProvider):
logger.info("Started ping thread")
# Fetch all contract specs at once
spots: list[str] = cfg["spots"]
currencies: list[str] = cfg.get("currencies") or ["USDT"]
contracts = list(chain(*[map(lambda s: f"{s}{c}", spots) for c in currencies]))
contract_specs = fetch_contract_specs(contracts)
contract_specs = fetch_contract_specs()
# 1) Send security info
sec_list = PySecurityList()
for contract in contracts:
for contract in CONTRACTS:
asset, quote = contract.split("_")
specs = contract_specs[contract]
spec = get_contract_spec(
@@ -144,7 +134,7 @@ class GateIoMdProvider(MdProvider):
logger.info("Sent securities list--sink.on_securities called.")
# 2) Subscribe to order books
for contract in contracts:
for contract in CONTRACTS:
sub_msg = {
"time": int(time.time() * 1000),
"channel": "futures.order_book",
@@ -1,463 +0,0 @@
import logging
from dataclasses import dataclass
from typing import Any
from pybit.exceptions import (
InvalidChannelTypeError,
TopicMismatchError,
UnauthorizedExceptionError,
)
from pybit._v5_misc import MiscHTTP
from pybit._v5_market import MarketHTTP
from pybit._v5_trade import TradeHTTP
from pybit._v5_account import AccountHTTP
from pybit._v5_asset import AssetHTTP
from pybit._v5_position import PositionHTTP
from pybit._v5_pre_upgrade import PreUpgradeHTTP
from pybit._v5_spot_leverage_token import SpotLeverageHTTP
from pybit._v5_spot_margin_trade import SpotMarginTradeHTTP
from pybit._v5_user import UserHTTP
from pybit._v5_broker import BrokerHTTP
from pybit._v5_institutional_loan import InstitutionalLoanHTTP
from pybit._v5_crypto_loan import CryptoLoanHTTP
from pybit._v5_earn import EarnHTTP
from pybit_websocket_stream import _V5WebSocketManager
from pybit_websocket_trading import _V5TradeWebSocketManager
from pybit._v5_spread import (
SpreadHTTP,
_V5WebSocketSpreadTrading,
)
logger = logging.getLogger(__name__)
WSS_NAME = "Unified V5"
PRIVATE_WSS = "wss://{SUBDOMAIN}.{DOMAIN}.com/v5/private"
PUBLIC_WSS = "wss://{SUBDOMAIN}.{DOMAIN}.com/v5/public/{CHANNEL_TYPE}"
AVAILABLE_CHANNEL_TYPES = [
"inverse",
"linear",
"spot",
"option",
"private",
]
@dataclass
class HTTP(
MiscHTTP,
MarketHTTP,
TradeHTTP,
AccountHTTP,
AssetHTTP,
PositionHTTP,
PreUpgradeHTTP,
SpotLeverageHTTP,
SpotMarginTradeHTTP,
UserHTTP,
BrokerHTTP,
InstitutionalLoanHTTP,
CryptoLoanHTTP,
EarnHTTP,
):
def __init__(self, **args):
super().__init__(**args)
class WebSocket(_V5WebSocketManager):
def _validate_public_topic(self):
if "/v5/public" not in self.WS_URL:
raise TopicMismatchError(
"Requested topic does not match channel_type"
)
def _validate_private_topic(self):
if not self.WS_URL.endswith("/private"):
raise TopicMismatchError(
"Requested topic does not match channel_type"
)
def __init__(
self,
channel_type: str,
**kwargs,
):
super().__init__(WSS_NAME, **kwargs)
if channel_type not in AVAILABLE_CHANNEL_TYPES:
raise InvalidChannelTypeError(
f"Channel type is not correct. Available: {AVAILABLE_CHANNEL_TYPES}"
)
if channel_type == "private":
self.WS_URL = PRIVATE_WSS
else:
self.WS_URL = PUBLIC_WSS.replace("{CHANNEL_TYPE}", channel_type)
# Do not pass keys and attempt authentication on a public connection
self.api_key = None
self.api_secret = None
if (
self.api_key is None or self.api_secret is None
) and channel_type == "private":
raise UnauthorizedExceptionError(
"API_KEY or API_SECRET is not set. They both are needed in order to access private topics"
)
self._connect(self.WS_URL)
# Private topics
def position_stream(self, callback):
"""Subscribe to the position stream to see changes to your position data in real-time.
Push frequency: real-time
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/private/position
"""
self._validate_private_topic()
topic = "position"
self.subscribe(topic, callback)
def order_stream(self, callback):
"""Subscribe to the order stream to see changes to your orders in real-time.
Push frequency: real-time
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/private/order
"""
self._validate_private_topic()
topic = "order"
self.subscribe(topic, callback)
def execution_stream(self, callback):
"""Subscribe to the execution stream to see your executions in real-time.
Push frequency: real-time
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/private/execution
"""
self._validate_private_topic()
topic = "execution"
self.subscribe(topic, callback)
def fast_execution_stream(self, callback, categorised_topic=""):
"""Fast execution stream significantly reduces data latency compared
original "execution" stream. However, it pushes limited execution type
of trades, and fewer data fields.
Use categorised_topic as a filter for a certain `category`. See docs.
Push frequency: real-time
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/private/fast-execution
"""
self._validate_private_topic()
topic = "execution.fast"
if categorised_topic:
topic += "." + categorised_topic
self.subscribe(topic, callback)
def wallet_stream(self, callback):
"""Subscribe to the wallet stream to see changes to your wallet in real-time.
Push frequency: real-time
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/private/wallet
"""
self._validate_private_topic()
topic = "wallet"
self.subscribe(topic, callback)
def greek_stream(self, callback):
"""Subscribe to the greeks stream to see changes to your greeks data in real-time. option only.
Push frequency: real-time
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/private/greek
"""
self._validate_private_topic()
topic = "greeks"
self.subscribe(topic, callback)
def spread_order_stream(self, callback):
"""Subscribe to the spread trading order stream to see changes to your orders in real-time.
Push frequency: real-time
Additional information:
https://bybit-exchange.github.io/docs/v5/spread/websocket/private/order
"""
self._validate_private_topic()
topic = "spread.order"
self.subscribe(topic, callback)
def spread_execution_stream(self, callback):
"""Subscribe to the spread trading execution stream to see your executions in real-time.
Push frequency: real-time
Additional information:
https://bybit-exchange.github.io/docs/v5/spread/websocket/private/execution
"""
self._validate_private_topic()
topic = "spread.execution"
self.subscribe(topic, callback)
# Public topics
def orderbook_stream(self, depth: int, symbol: (str, list), callback):
"""Subscribe to the orderbook stream. Supports different depths.
Linear & inverse:
Level 1 data, push frequency: 10ms
Level 50 data, push frequency: 20ms
Level 200 data, push frequency: 100ms
Level 500 data, push frequency: 100ms
Spot:
Level 1 data, push frequency: 10ms
Level 50 data, push frequency: 20ms
Option:
Level 25 data, push frequency: 20ms
Level 100 data, push frequency: 100ms
Required args:
symbol (string/list): Symbol name(s)
depth (int): Orderbook depth
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/public/orderbook
"""
self._validate_public_topic()
topic = f"orderbook.{depth}." + "{symbol}"
self.subscribe(topic, callback, symbol)
def trade_stream(self, symbol: (str, list), callback):
"""
Subscribe to the recent trades stream.
After subscription, you will be pushed trade messages in real-time.
Push frequency: real-time
Required args:
symbol (string/list): Symbol name(s)
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/public/trade
"""
self._validate_public_topic()
topic = f"publicTrade." + "{symbol}"
self.subscribe(topic, callback, symbol)
def ticker_stream(self, symbol: (str, list), callback):
"""Subscribe to the ticker stream.
Push frequency: 100ms
Required args:
symbol (string/list): Symbol name(s)
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/public/ticker
"""
self._validate_public_topic()
topic = "tickers.{symbol}"
self.subscribe(topic, callback, symbol)
def kline_stream(self, interval: int, symbol: (str, list), callback):
"""Subscribe to the klines stream.
Push frequency: 1-60s
Required args:
symbol (string/list): Symbol name(s)
interval (int): Kline interval
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/public/kline
"""
self._validate_public_topic()
topic = f"kline.{interval}." + "{symbol}"
self.subscribe(topic, callback, symbol)
def liquidation_stream(self, symbol: (str, list), callback):
"""
Pushes at most one order per second per symbol.
As such, this feed does not push all liquidations that occur on Bybit.
Push frequency: 1s
Required args:
symbol (string/list): Symbol name(s)
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/public/liquidation
"""
logger.warning("liquidation_stream() is deprecated. Please use "
"all_liquidation_stream().")
self._validate_public_topic()
topic = "liquidation.{symbol}"
self.subscribe(topic, callback, symbol)
def all_liquidation_stream(self, symbol: (str, list), callback):
"""Subscribe to the liquidation stream, push all liquidations that
occur on Bybit.
Push frequency: 500ms
Required args:
symbol (string/list): Symbol name(s)
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/public/all-liquidation
"""
self._validate_public_topic()
topic = "allLiquidation.{symbol}"
self.subscribe(topic, callback, symbol)
def lt_kline_stream(self, interval: int, symbol: (str, list), callback):
"""Subscribe to the leveraged token kline stream.
Push frequency: 1-60s
Required args:
symbol (string/list): Symbol name(s)
interval (int): Leveraged token Kline stream interval
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/public/etp-kline
"""
self._validate_public_topic()
topic = f"kline_lt.{interval}." + "{symbol}"
self.subscribe(topic, callback, symbol)
def lt_ticker_stream(self, symbol: (str, list), callback):
"""Subscribe to the leveraged token ticker stream.
Push frequency: 300ms
Required args:
symbol (string/list): Symbol name(s)
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/public/etp-ticker
"""
self._validate_public_topic()
topic = "tickers_lt.{symbol}"
self.subscribe(topic, callback, symbol)
def lt_nav_stream(self, symbol: (str, list), callback):
"""Subscribe to the leveraged token nav stream.
Push frequency: 300ms
Required args:
symbol (string/list): Symbol name(s)
Additional information:
https://bybit-exchange.github.io/docs/v5/websocket/public/etp-nav
"""
self._validate_public_topic()
topic = "lt.{symbol}"
self.subscribe(topic, callback, symbol)
class WebSocketTrading(_V5TradeWebSocketManager):
def __init__(self, recv_window=0, referral_id="", **kwargs):
super().__init__(recv_window, referral_id, **kwargs)
@staticmethod
def _extract_error_callback(req_params: dict[str, Any]):
err_cb = req_params.pop("error_callback") if "error_callback" in req_params else None
return (req_params, err_cb)
def place_order(self, callback, **kwargs):
operation = "order.create"
self._send_order_operation(
operation, callback, *self._extract_error_callback(kwargs)
)
def amend_order(self, callback, **kwargs):
operation = "order.amend"
self._send_order_operation(
operation, callback, *self._extract_error_callback(kwargs)
)
def cancel_order(self, callback, **kwargs):
operation = "order.cancel"
self._send_order_operation(
operation, callback, *self._extract_error_callback(kwargs)
)
def place_batch_order(self, callback, **kwargs):
operation = "order.create-batch"
self._send_order_operation(
operation, callback, *self._extract_error_callback(kwargs)
)
def amend_batch_order(self, callback, **kwargs):
operation = "order.amend-batch"
self._send_order_operation(
operation, callback, *self._extract_error_callback(kwargs)
)
def cancel_batch_order(self, callback, **kwargs):
operation = "order.cancel-batch"
self._send_order_operation(
operation, callback, *self._extract_error_callback(kwargs)
)
class WebsocketSpreadTrading(_V5WebSocketSpreadTrading):
def __init__(self, **kwargs):
super().__init__(**kwargs)
def orderbook_stream(self, depth: int, symbol: (str, list), callback):
"""Subscribe to the orderbook stream. Supports different depths.
Level 25 data, push frequency: 20ms
Required args:
symbol (string/list): Symbol name(s)
depth (int): Orderbook depth
Additional information:
https://bybit-exchange.github.io/docs/v5/spread/websocket/public/orderbook
"""
topic = f"orderbook.{depth}." + "{symbol}"
self.subscribe(topic, callback, symbol)
def trade_stream(self, symbol: (str, list), callback):
"""
Subscribe to the recent trades stream.
After subscription, you will be pushed trade messages in real-time.
Push frequency: real-time
Required args:
symbol (string/list): Symbol name(s)
Additional information:
https://bybit-exchange.github.io/docs/v5/spread/websocket/public/trade
"""
topic = f"publicTrade." + "{symbol}"
self.subscribe(topic, callback, symbol)
def ticker_stream(self, symbol: (str, list), callback):
"""Subscribe to the ticker stream.
Push frequency: 100ms
Required args:
symbol (string/list): Symbol name(s)
Additional information:
https://bybit-exchange.github.io/docs/v5/spread/websocket/public/ticker
"""
topic = "tickers.{symbol}"
self.subscribe(topic, callback, symbol)
Loaded 100 of 130 files, more files were not shown because too many files have changed in this diff. Show more