Compare commits

..
Author SHA1 Message Date
John Colvin bef84d6dc3 build: Install boost via vcpkg on linux, update ccapi
Old ccapi wasn't compatible with new boost
2026-03-20 17:02:44 +00:00
John Colvin b061509126 build: Compile spark-windows with clang
Made a couple minor changes to cpp to address new werrors
2026-03-20 17:02:44 +00:00
John Colvin 5c0487358e build: Install boost via vcpkg on windows 2026-03-20 17:02:44 +00:00
John Colvin 5c7eedbf7b build: Don't activate venv when generating individual protos
We activate the venv for the whole build
2026-03-20 17:02:44 +00:00
John Colvin 74475cb68f deploy: Resolve profile dirs with regex patterns 2026-03-20 17:02:44 +00:00
John Colvin 24978f76b5 improve: Wait until py work requested before loading py modules
So we don't crash due to missing py modules when no py plugins are running.
2026-03-20 17:02:44 +00:00
John Colvin 0698567f53 deploy: Add sparkd-shell service to compose.yaml for troubleshooting 2026-03-20 17:02:44 +00:00
John Colvin ec78e35733 cleanup: Remove unused code 2026-03-20 17:02:44 +00:00
John Colvin 4801ce6b83 refactor: Wrap md run logic in class MarketDataSession
Start in the constructor, gracefully exit in the destructor.
2026-03-20 17:02:44 +00:00
John Colvin 7eb8cd5ead Allow md recording dir to be mounted as volume in docker compose 2026-03-20 17:02:44 +00:00
John Colvin d7b42e3cee refactor: Split MDRelay and QueryServer up into smaller files
This code is inherently complicated. It's hard to follow when everything
is in one giant file.
2026-03-20 17:02:44 +00:00
John Colvin 35d65f107b deploy: Configure timezone in entrypoint script 2026-03-20 17:02:44 +00:00
John Colvin e335a2d827 deploy: Move compose entrypoint to dedicated script 2026-03-20 17:02:44 +00:00
John Colvin abc7d44ce2 deploy: Fix typo in sparkd conf dir path in compose.yaml 2026-03-20 17:02:44 +00:00
John Colvin 85ff7f412a cleanup: Fix a few more build warnings 2026-03-20 17:02:44 +00:00
John Colvin 8a3ad08456 refactor: Replace DataPacket with IDataPacket in more places 2026-03-20 17:02:44 +00:00
John Colvin f181d34cbe build: Check if venv already active before activating 2026-03-20 17:02:44 +00:00
John Colvin 002087f62a refactor: Replace TableFormatter with table.h
Also move to "format" dir
2026-03-20 17:02:44 +00:00
John Colvin e37ae056c0 refactor: Replace StringFinder/Joiner.h with find/join.h
Also move to "algorithm" dir
2026-03-20 17:02:44 +00:00
John Colvin b8a7f398c9 refactor: Rename a few TitleCase .h files snake_case
* EnumMacros.h -> enum_macros.h
* TimeSpan.h ---> time_span.h
* TimeStamp.h --> time_stamp.h
2026-03-20 17:02:44 +00:00
John Colvin 2038d747d8 diag: Use structured logs in MDP3Listener::onOption 2026-03-20 17:02:44 +00:00
John Colvin edf127069f deploy: Install tzdata in linux_dotnet8 Dockerfile
So we can configure timezone in compose services. Specifically, we need this
to support live cme md (so onixs doesn't think messages are "too old").
2026-03-20 17:02:44 +00:00
John Colvin 07448cede9 cleanup: Remove unused code 2026-03-20 17:02:44 +00:00
John Colvin 9aef42cb21 refactor: Add new glbx variable ticks to security (shm and proto) 2026-03-20 17:02:44 +00:00
John Colvin baa43cbecf refactor: Enable nullability in Bts.Spark.Api 2026-03-08 16:00:50 +00:00
John Colvin de3e491843 build: Upgrade cs LangVersion 11 -> 12 2026-03-08 16:00:50 +00:00
John Colvin 7cc09350a0 refactor: Rename SparkApiCredentials -> SparkCredentials 2026-03-08 16:00:50 +00:00
John Colvin 059e1202e1 refactor: Rename SparkdLoginService -> SparkLoginService
This makes it consistent with other type names (we don't use "Sparkd"
anywhere else).
2026-03-08 16:00:50 +00:00
John Colvin 9093f56498 refactor: Rename SparkdCredentials.cs -> SparkApiCredentials.cs
Matches the class name
2026-03-08 16:00:50 +00:00
John Colvin 649b4560ff cleanup: Remove unnecessary subscription wrapper in cs sdk 2026-03-08 16:00:50 +00:00
John Colvin 6c060f6171 cleanup: Fix indentation in TopStream
Whitespace changes
2026-03-08 16:00:50 +00:00
John Colvin 581b9259b8 cleanup: Fix build warnings in EqualityComparer
Nullability issues
2026-03-08 16:00:50 +00:00
John Colvin e7bd57964e refactor: Rename Optional members PascalCase
Also reformat code according to new style guidlines
2026-03-08 16:00:50 +00:00
John Colvin 6ea955aacd refactor: Replace ToDictionaryCrushed/Dupes w/ DuplicateKeyPolicy enum
Overload ToDictionary with this enum instead.
2026-03-08 16:00:49 +00:00
John Colvin 29d028fee2 refactor: Replace find_first_index with FindIndex 2026-03-08 16:00:49 +00:00
John Colvin dc5da87a65 cleanup: Fix cs build warnings 2026-03-08 16:00:49 +00:00
John Colvin 7de98a313f refactor: Break up Bts.Spark.Core into a few diff projs
This is in anticipation of decoupling BlueOrb from Spark.

* Bts.Collections
* Bts.Core
* Bts.Disposables
* Bts.Numerics
2026-03-08 16:00:49 +00:00
John Colvin 7873e76ba0 build: Clean up spark_client.sln
* Remove unused csprojs related to old sdk
* Remove erroneous build type config
2026-03-08 16:00:49 +00:00
354 changed files with 3199 additions and 2639 deletions

No files matched your search

+1 -2
View File
@@ -51,8 +51,7 @@ SPARKD__QUICKBUS_PORT=8001
SPARKD__SIM_FIX_PORT=8002 SPARKD__SIM_FIX_PORT=8002
SPARKD__SHM_SIZE=4gb SPARKD__SHM_SIZE=4gb
SPARKD__PYTHON_REQUIREMENTS=requirements.txt SPARKD__PYTHON_REQUIREMENTS=requirements.txt
SPARKD__SPARKD_ENTRYPOINT=src/app_sparkd/sparkd-entrypoint.sh SPARKD__ENTRYPOINT=src/app_sparkd/sparkd-entrypoint.sh
SPARKD__MAIN_ENTRYPOINT=./sparkd-entrypoint.sh
### PubSubServer ### ### PubSubServer ###
PUBSUB__CONFIG_DIR=src/DataStreamSuite/PubSubServer/Configurations/Development PUBSUB__CONFIG_DIR=src/DataStreamSuite/PubSubServer/Configurations/Development
+9 -9
View File
@@ -183,10 +183,11 @@
"windows": { "windows": {
"program": "PubSubServer.exe", "program": "PubSubServer.exe",
}, },
"args": [
"--config",
"${workspaceFolder}/src/DataStreamSuite/PubSubServer/Configurations/${input:pubSubServerConfig}.json"
],
"cwd": "build/Debug/bin/net8.0", "cwd": "build/Debug/bin/net8.0",
"env": {
"DOTNET_ENVIRONMENT": "${input:pubSubServerEnvironment}",
},
"stopAtEntry": false, "stopAtEntry": false,
"console": "integratedTerminal" "console": "integratedTerminal"
}, },
@@ -448,15 +449,14 @@
"default": "Local", "default": "Local",
}, },
{ {
"id": "pubSubServerEnvironment", "id": "pubSubServerConfig",
"type": "pickString", "type": "pickString",
"description": "PubSubServer Configuration", "description": "PubSub Configuration",
"options": [ "options": [
"Development", "PubSubServer",
"EchoServer", "SimpleEchoClient",
"Local",
], ],
"default": "Development", "default": "PubSubServer",
}, },
{ {
"id": "sdkDemoEnvironment", "id": "sdkDemoEnvironment",
+60
View File
@@ -17,6 +17,14 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Spark.Sdk", "src\cs\Bts
EndProject EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Spark.Sdk.Demo", "src\cs\Bts.Spark.Sdk.Demo\Bts.Spark.Sdk.Demo.csproj", "{FCFDBDBE-74B0-41C4-9011-C651B21056D4}" Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Spark.Sdk.Demo", "src\cs\Bts.Spark.Sdk.Demo\Bts.Spark.Sdk.Demo.csproj", "{FCFDBDBE-74B0-41C4-9011-C651B21056D4}"
EndProject EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Collections", "src\cs\Bts.Collections\Bts.Collections.csproj", "{3BF4DF96-5AFF-4674-866A-71A8090627FF}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Core", "src\cs\Bts.Core\Bts.Core.csproj", "{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Disposables", "src\cs\Bts.Disposables\Bts.Disposables.csproj", "{62067659-AA2E-4CB5-B7CE-0B38EAA92080}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Numerics", "src\cs\Bts.Numerics\Bts.Numerics.csproj", "{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE}"
EndProject
Global Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU Debug|Any CPU = Debug|Any CPU
@@ -87,6 +95,54 @@ Global
{FCFDBDBE-74B0-41C4-9011-C651B21056D4}.Release|x64.Build.0 = Release|Any CPU {FCFDBDBE-74B0-41C4-9011-C651B21056D4}.Release|x64.Build.0 = Release|Any CPU
{FCFDBDBE-74B0-41C4-9011-C651B21056D4}.Release|x86.ActiveCfg = Release|Any CPU {FCFDBDBE-74B0-41C4-9011-C651B21056D4}.Release|x86.ActiveCfg = Release|Any CPU
{FCFDBDBE-74B0-41C4-9011-C651B21056D4}.Release|x86.Build.0 = Release|Any CPU {FCFDBDBE-74B0-41C4-9011-C651B21056D4}.Release|x86.Build.0 = Release|Any CPU
{3BF4DF96-5AFF-4674-866A-71A8090627FF}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{3BF4DF96-5AFF-4674-866A-71A8090627FF}.Debug|Any CPU.Build.0 = Debug|Any CPU
{3BF4DF96-5AFF-4674-866A-71A8090627FF}.Debug|x64.ActiveCfg = Debug|Any CPU
{3BF4DF96-5AFF-4674-866A-71A8090627FF}.Debug|x64.Build.0 = Debug|Any CPU
{3BF4DF96-5AFF-4674-866A-71A8090627FF}.Debug|x86.ActiveCfg = Debug|Any CPU
{3BF4DF96-5AFF-4674-866A-71A8090627FF}.Debug|x86.Build.0 = Debug|Any CPU
{3BF4DF96-5AFF-4674-866A-71A8090627FF}.Release|Any CPU.ActiveCfg = Release|Any CPU
{3BF4DF96-5AFF-4674-866A-71A8090627FF}.Release|Any CPU.Build.0 = Release|Any CPU
{3BF4DF96-5AFF-4674-866A-71A8090627FF}.Release|x64.ActiveCfg = Release|Any CPU
{3BF4DF96-5AFF-4674-866A-71A8090627FF}.Release|x64.Build.0 = Release|Any CPU
{3BF4DF96-5AFF-4674-866A-71A8090627FF}.Release|x86.ActiveCfg = Release|Any CPU
{3BF4DF96-5AFF-4674-866A-71A8090627FF}.Release|x86.Build.0 = Release|Any CPU
{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B}.Debug|Any CPU.Build.0 = Debug|Any CPU
{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B}.Debug|x64.ActiveCfg = Debug|Any CPU
{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B}.Debug|x64.Build.0 = Debug|Any CPU
{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B}.Debug|x86.ActiveCfg = Debug|Any CPU
{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B}.Debug|x86.Build.0 = Debug|Any CPU
{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B}.Release|Any CPU.ActiveCfg = Release|Any CPU
{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B}.Release|Any CPU.Build.0 = Release|Any CPU
{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B}.Release|x64.ActiveCfg = Release|Any CPU
{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B}.Release|x64.Build.0 = Release|Any CPU
{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B}.Release|x86.ActiveCfg = Release|Any CPU
{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B}.Release|x86.Build.0 = Release|Any CPU
{62067659-AA2E-4CB5-B7CE-0B38EAA92080}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{62067659-AA2E-4CB5-B7CE-0B38EAA92080}.Debug|Any CPU.Build.0 = Debug|Any CPU
{62067659-AA2E-4CB5-B7CE-0B38EAA92080}.Debug|x64.ActiveCfg = Debug|Any CPU
{62067659-AA2E-4CB5-B7CE-0B38EAA92080}.Debug|x64.Build.0 = Debug|Any CPU
{62067659-AA2E-4CB5-B7CE-0B38EAA92080}.Debug|x86.ActiveCfg = Debug|Any CPU
{62067659-AA2E-4CB5-B7CE-0B38EAA92080}.Debug|x86.Build.0 = Debug|Any CPU
{62067659-AA2E-4CB5-B7CE-0B38EAA92080}.Release|Any CPU.ActiveCfg = Release|Any CPU
{62067659-AA2E-4CB5-B7CE-0B38EAA92080}.Release|Any CPU.Build.0 = Release|Any CPU
{62067659-AA2E-4CB5-B7CE-0B38EAA92080}.Release|x64.ActiveCfg = Release|Any CPU
{62067659-AA2E-4CB5-B7CE-0B38EAA92080}.Release|x64.Build.0 = Release|Any CPU
{62067659-AA2E-4CB5-B7CE-0B38EAA92080}.Release|x86.ActiveCfg = Release|Any CPU
{62067659-AA2E-4CB5-B7CE-0B38EAA92080}.Release|x86.Build.0 = Release|Any CPU
{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE}.Debug|Any CPU.Build.0 = Debug|Any CPU
{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE}.Debug|x64.ActiveCfg = Debug|Any CPU
{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE}.Debug|x64.Build.0 = Debug|Any CPU
{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE}.Debug|x86.ActiveCfg = Debug|Any CPU
{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE}.Debug|x86.Build.0 = Debug|Any CPU
{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE}.Release|Any CPU.ActiveCfg = Release|Any CPU
{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE}.Release|Any CPU.Build.0 = Release|Any CPU
{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE}.Release|x64.ActiveCfg = Release|Any CPU
{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE}.Release|x64.Build.0 = Release|Any CPU
{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE}.Release|x86.ActiveCfg = Release|Any CPU
{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE}.Release|x86.Build.0 = Release|Any CPU
EndGlobalSection EndGlobalSection
GlobalSection(SolutionProperties) = preSolution GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE HideSolutionNode = FALSE
@@ -98,5 +154,9 @@ Global
{37E4E890-D025-4729-90F9-6F95862F47BC} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295} {37E4E890-D025-4729-90F9-6F95862F47BC} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295}
{A5FF3FAF-43CB-496B-AF4B-F61EFD645579} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295} {A5FF3FAF-43CB-496B-AF4B-F61EFD645579} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295}
{FCFDBDBE-74B0-41C4-9011-C651B21056D4} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295} {FCFDBDBE-74B0-41C4-9011-C651B21056D4} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295}
{3BF4DF96-5AFF-4674-866A-71A8090627FF} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295}
{CD0A3EEF-303A-44EE-85F2-76F4CE632A3B} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295}
{62067659-AA2E-4CB5-B7CE-0B38EAA92080} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295}
{2FE1E5D0-168A-4D83-9B18-0D48250B4BFE} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295}
EndGlobalSection EndGlobalSection
EndGlobal EndGlobal
+8 -2
View File
@@ -32,14 +32,20 @@
"lhs": "${hostSystemName}", "lhs": "${hostSystemName}",
"rhs": "Windows" "rhs": "Windows"
}, },
"generator": "Visual Studio 17 2022", "generator": "Ninja Multi-Config",
"environment": { "environment": {
"SPARK_CMAKE_CXX_FLAGS": "/MP /WX /wd4250 /wd4251 /wd4244 /wd4267 /wd4355 /wd4275 /wd4345 /bigobj /EHsc /DCURL_STATICLIB /D_WINSOCK_DEPRECATED_NO_WARNINGS /D__TBB_NO_IMPLICIT_LINKAGE /DBOOST_ALL_NO_LIB /std:c++20 /D_SCL_SECURE_NO_WARNINGS /D_CRT_SECURE_NO_WARNINGS /D_WIN32_WINNT=0x0600" "SPARK_CMAKE_CXX_FLAGS": "/WX /wd4250 /wd4251 /wd4244 /wd4267 /wd4355 /wd4275 /wd4345 /bigobj /EHsc /DCURL_STATICLIB /D_WINSOCK_DEPRECATED_NO_WARNINGS /D__TBB_NO_IMPLICIT_LINKAGE /DBOOST_ALL_NO_LIB /std:c++20 /D_SCL_SECURE_NO_WARNINGS /D_CRT_SECURE_NO_WARNINGS /D_WIN32_WINNT=0x0600"
}, },
"cacheVariables": { "cacheVariables": {
"CMAKE_C_COMPILER": "clang-cl",
"CMAKE_CXX_COMPILER": "clang-cl",
"CMAKE_CXX_FLAGS": "$env{SPARK_CMAKE_CXX_FLAGS}", "CMAKE_CXX_FLAGS": "$env{SPARK_CMAKE_CXX_FLAGS}",
"CMAKE_CXX_FLAGS_DEBUG": "$env{SPARK_CMAKE_CXX_FLAGS} /MTd", "CMAKE_CXX_FLAGS_DEBUG": "$env{SPARK_CMAKE_CXX_FLAGS} /MTd",
"CMAKE_CXX_FLAGS_RELEASE": "$env{SPARK_CMAKE_CXX_FLAGS} /MT", "CMAKE_CXX_FLAGS_RELEASE": "$env{SPARK_CMAKE_CXX_FLAGS} /MT",
"CMAKE_EXPORT_COMPILE_COMMANDS": {
"type": "BOOL",
"value": "ON"
},
"CMAKE_SHARED_LINKER_FLAGS_DEBUG": "/DEBUG", "CMAKE_SHARED_LINKER_FLAGS_DEBUG": "/DEBUG",
"CMAKE_SHARED_LINKER_FLAGS_RELEASE": "/OPT:REF /OPT:ICF /LTCG", "CMAKE_SHARED_LINKER_FLAGS_RELEASE": "/OPT:REF /OPT:ICF /LTCG",
"EXTERNALS_ROOT": "${sourceDir}/externals/windows" "EXTERNALS_ROOT": "${sourceDir}/externals/windows"
+108
View File
@@ -51,6 +51,14 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Net", "src\cs\Bts.Net\B
EndProject EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.IO.Compression", "src\cs\Bts.IO.Compression\Bts.IO.Compression.csproj", "{BC277499-8215-4F9F-B9AB-07B6391110BF}" Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.IO.Compression", "src\cs\Bts.IO.Compression\Bts.IO.Compression.csproj", "{BC277499-8215-4F9F-B9AB-07B6391110BF}"
EndProject EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Collections", "src\cs\Bts.Collections\Bts.Collections.csproj", "{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Core", "src\cs\Bts.Core\Bts.Core.csproj", "{802ECF0B-7264-4168-95D1-158E47FB65CC}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Disposables", "src\cs\Bts.Disposables\Bts.Disposables.csproj", "{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Numerics", "src\cs\Bts.Numerics\Bts.Numerics.csproj", "{0354655D-2394-42BB-B1D4-2D75B9BCABF1}"
EndProject
Global Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|x64 = Debug|x64 Debug|x64 = Debug|x64
@@ -571,6 +579,102 @@ Global
{BC277499-8215-4F9F-B9AB-07B6391110BF}.RelWithDebInfo|Any CPU.Build.0 = Debug|Any CPU {BC277499-8215-4F9F-B9AB-07B6391110BF}.RelWithDebInfo|Any CPU.Build.0 = Debug|Any CPU
{BC277499-8215-4F9F-B9AB-07B6391110BF}.RelWithDebInfo|x86.ActiveCfg = Debug|Any CPU {BC277499-8215-4F9F-B9AB-07B6391110BF}.RelWithDebInfo|x86.ActiveCfg = Debug|Any CPU
{BC277499-8215-4F9F-B9AB-07B6391110BF}.RelWithDebInfo|x86.Build.0 = Debug|Any CPU {BC277499-8215-4F9F-B9AB-07B6391110BF}.RelWithDebInfo|x86.Build.0 = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.Debug|x64.ActiveCfg = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.Debug|x64.Build.0 = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.Debug|Any CPU.Build.0 = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.Debug|x86.ActiveCfg = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.Debug|x86.Build.0 = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.MinSizeRel|x64.ActiveCfg = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.MinSizeRel|x64.Build.0 = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.MinSizeRel|Any CPU.ActiveCfg = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.MinSizeRel|Any CPU.Build.0 = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.MinSizeRel|x86.ActiveCfg = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.MinSizeRel|x86.Build.0 = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.Release|x64.ActiveCfg = Release|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.Release|x64.Build.0 = Release|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.Release|Any CPU.ActiveCfg = Release|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.Release|Any CPU.Build.0 = Release|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.Release|x86.ActiveCfg = Release|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.Release|x86.Build.0 = Release|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.RelWithDebInfo|x64.ActiveCfg = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.RelWithDebInfo|x64.Build.0 = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.RelWithDebInfo|Any CPU.ActiveCfg = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.RelWithDebInfo|Any CPU.Build.0 = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.RelWithDebInfo|x86.ActiveCfg = Debug|Any CPU
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6}.RelWithDebInfo|x86.Build.0 = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.Debug|x64.ActiveCfg = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.Debug|x64.Build.0 = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.Debug|Any CPU.Build.0 = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.Debug|x86.ActiveCfg = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.Debug|x86.Build.0 = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.MinSizeRel|x64.ActiveCfg = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.MinSizeRel|x64.Build.0 = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.MinSizeRel|Any CPU.ActiveCfg = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.MinSizeRel|Any CPU.Build.0 = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.MinSizeRel|x86.ActiveCfg = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.MinSizeRel|x86.Build.0 = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.Release|x64.ActiveCfg = Release|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.Release|x64.Build.0 = Release|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.Release|Any CPU.ActiveCfg = Release|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.Release|Any CPU.Build.0 = Release|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.Release|x86.ActiveCfg = Release|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.Release|x86.Build.0 = Release|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.RelWithDebInfo|x64.ActiveCfg = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.RelWithDebInfo|x64.Build.0 = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.RelWithDebInfo|Any CPU.ActiveCfg = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.RelWithDebInfo|Any CPU.Build.0 = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.RelWithDebInfo|x86.ActiveCfg = Debug|Any CPU
{802ECF0B-7264-4168-95D1-158E47FB65CC}.RelWithDebInfo|x86.Build.0 = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.Debug|x64.ActiveCfg = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.Debug|x64.Build.0 = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.Debug|Any CPU.Build.0 = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.Debug|x86.ActiveCfg = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.Debug|x86.Build.0 = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.MinSizeRel|x64.ActiveCfg = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.MinSizeRel|x64.Build.0 = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.MinSizeRel|Any CPU.ActiveCfg = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.MinSizeRel|Any CPU.Build.0 = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.MinSizeRel|x86.ActiveCfg = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.MinSizeRel|x86.Build.0 = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.Release|x64.ActiveCfg = Release|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.Release|x64.Build.0 = Release|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.Release|Any CPU.ActiveCfg = Release|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.Release|Any CPU.Build.0 = Release|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.Release|x86.ActiveCfg = Release|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.Release|x86.Build.0 = Release|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.RelWithDebInfo|x64.ActiveCfg = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.RelWithDebInfo|x64.Build.0 = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.RelWithDebInfo|Any CPU.ActiveCfg = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.RelWithDebInfo|Any CPU.Build.0 = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.RelWithDebInfo|x86.ActiveCfg = Debug|Any CPU
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0}.RelWithDebInfo|x86.Build.0 = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.Debug|x64.ActiveCfg = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.Debug|x64.Build.0 = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.Debug|Any CPU.Build.0 = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.Debug|x86.ActiveCfg = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.Debug|x86.Build.0 = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.MinSizeRel|x64.ActiveCfg = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.MinSizeRel|x64.Build.0 = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.MinSizeRel|Any CPU.ActiveCfg = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.MinSizeRel|Any CPU.Build.0 = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.MinSizeRel|x86.ActiveCfg = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.MinSizeRel|x86.Build.0 = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.Release|x64.ActiveCfg = Release|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.Release|x64.Build.0 = Release|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.Release|Any CPU.ActiveCfg = Release|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.Release|Any CPU.Build.0 = Release|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.Release|x86.ActiveCfg = Release|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.Release|x86.Build.0 = Release|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.RelWithDebInfo|x64.ActiveCfg = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.RelWithDebInfo|x64.Build.0 = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.RelWithDebInfo|Any CPU.ActiveCfg = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.RelWithDebInfo|Any CPU.Build.0 = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.RelWithDebInfo|x86.ActiveCfg = Debug|Any CPU
{0354655D-2394-42BB-B1D4-2D75B9BCABF1}.RelWithDebInfo|x86.Build.0 = Debug|Any CPU
EndGlobalSection EndGlobalSection
GlobalSection(SolutionProperties) = preSolution GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE HideSolutionNode = FALSE
@@ -592,6 +696,10 @@ Global
{385C4948-E080-56A6-E29E-8B0F4913175C} = {267F81EF-A8AB-49FD-9F56-D7EC3C1F7C7C} {385C4948-E080-56A6-E29E-8B0F4913175C} = {267F81EF-A8AB-49FD-9F56-D7EC3C1F7C7C}
{43FB297B-7A4D-4835-A878-E6DF345B580C} = {385C4948-E080-56A6-E29E-8B0F4913175C} {43FB297B-7A4D-4835-A878-E6DF345B580C} = {385C4948-E080-56A6-E29E-8B0F4913175C}
{BC277499-8215-4F9F-B9AB-07B6391110BF} = {385C4948-E080-56A6-E29E-8B0F4913175C} {BC277499-8215-4F9F-B9AB-07B6391110BF} = {385C4948-E080-56A6-E29E-8B0F4913175C}
{C42F4CCB-9E65-4370-939A-527ACDBC4DA6} = {385C4948-E080-56A6-E29E-8B0F4913175C}
{802ECF0B-7264-4168-95D1-158E47FB65CC} = {385C4948-E080-56A6-E29E-8B0F4913175C}
{95C5F6C0-88E5-4338-9A4F-614CF6C03CC0} = {385C4948-E080-56A6-E29E-8B0F4913175C}
{0354655D-2394-42BB-B1D4-2D75B9BCABF1} = {385C4948-E080-56A6-E29E-8B0F4913175C}
EndGlobalSection EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {422D1620-0166-4C26-9A81-CA8A2EB1AB58} SolutionGuid = {422D1620-0166-4C26-9A81-CA8A2EB1AB58}
+4 -32
View File
@@ -10,7 +10,7 @@ services:
- SOURCE_DIR=${SPARKD__SOURCE_DIR} - SOURCE_DIR=${SPARKD__SOURCE_DIR}
- CONFIG_DIR=${SPARKD__CONFIG_DIR} - CONFIG_DIR=${SPARKD__CONFIG_DIR}
- PYTHON_REQUIREMENTS=${SPARKD__PYTHON_REQUIREMENTS} - PYTHON_REQUIREMENTS=${SPARKD__PYTHON_REQUIREMENTS}
- SPARKD_ENTRYPOINT=${SPARKD__SPARKD_ENTRYPOINT} - ENTRYPOINT_SCRIPT=${SPARKD__ENTRYPOINT}
shm_size: "${SPARKD__SHM_SIZE:-512mb}" shm_size: "${SPARKD__SHM_SIZE:-512mb}"
env_file: env_file:
- .env - .env
@@ -19,7 +19,6 @@ services:
- "${SPARKD__CONFIG_PORT}:${SPARKD__CONFIG_PORT}" - "${SPARKD__CONFIG_PORT}:${SPARKD__CONFIG_PORT}"
- "${SPARKD__QUICKBUS_PORT}:${SPARKD__QUICKBUS_PORT}" - "${SPARKD__QUICKBUS_PORT}:${SPARKD__QUICKBUS_PORT}"
- "${SPARKD__SIM_FIX_PORT}:${SPARKD__SIM_FIX_PORT}" - "${SPARKD__SIM_FIX_PORT}:${SPARKD__SIM_FIX_PORT}"
entrypoint: ${SPARKD__MAIN_ENTRYPOINT}
deploy: deploy:
resources: resources:
limits: limits:
@@ -36,7 +35,7 @@ services:
- SOURCE_DIR=${SPARKD__SOURCE_DIR} - SOURCE_DIR=${SPARKD__SOURCE_DIR}
- CONFIG_DIR=${SPARKD__CONFIG_DIR} - CONFIG_DIR=${SPARKD__CONFIG_DIR}
- PYTHON_REQUIREMENTS=${SPARKD__PYTHON_REQUIREMENTS} - PYTHON_REQUIREMENTS=${SPARKD__PYTHON_REQUIREMENTS}
- SPARKD_ENTRYPOINT=${SPARKD__SPARKD_ENTRYPOINT} - ENTRYPOINT_SCRIPT=${SPARKD__ENTRYPOINT}
shm_size: "${SPARKD__SHM_SIZE:-512mb}" shm_size: "${SPARKD__SHM_SIZE:-512mb}"
command: "do sleep 1; done" command: "do sleep 1; done"
env_file: env_file:
@@ -54,7 +53,6 @@ services:
sparkd-symbols: sparkd-symbols:
volumes: volumes:
- ${SPARKD__DATA_DIR:-sparkd_data}:/home/bts/.local/share/bts/sparkd/data - ${SPARKD__DATA_DIR:-sparkd_data}:/home/bts/.local/share/bts/sparkd/data
- ${SPARKD__MD_RECORDING_DIR:-md_recording}:/md_recording
- ${SPARKD__RUNTIME_CONF_DIR:-sparkd_conf}:/home/bts/.local/share/bts/sparkd/conf - ${SPARKD__RUNTIME_CONF_DIR:-sparkd_conf}:/home/bts/.local/share/bts/sparkd/conf
build: build:
dockerfile: src/app_sparkd/Dockerfile dockerfile: src/app_sparkd/Dockerfile
@@ -63,7 +61,7 @@ services:
- SOURCE_DIR=${SPARKD__SOURCE_DIR} - SOURCE_DIR=${SPARKD__SOURCE_DIR}
- CONFIG_DIR=${SPARKD__CONFIG_DIR} - CONFIG_DIR=${SPARKD__CONFIG_DIR}
- PYTHON_REQUIREMENTS=${SPARKD__PYTHON_REQUIREMENTS} - PYTHON_REQUIREMENTS=${SPARKD__PYTHON_REQUIREMENTS}
- SPARKD_ENTRYPOINT=${SPARKD__SPARKD_ENTRYPOINT} - ENTRYPOINT_SCRIPT=${SPARKD__ENTRYPOINT}
shm_size: "${SPARKD__SHM_SIZE:-512mb}" shm_size: "${SPARKD__SHM_SIZE:-512mb}"
env_file: env_file:
- .env - .env
@@ -72,33 +70,6 @@ services:
- "${SPARKD__CONFIG_PORT}:${SPARKD__CONFIG_PORT}" - "${SPARKD__CONFIG_PORT}:${SPARKD__CONFIG_PORT}"
- "${SPARKD__QUICKBUS_PORT}:${SPARKD__QUICKBUS_PORT}" - "${SPARKD__QUICKBUS_PORT}:${SPARKD__QUICKBUS_PORT}"
- "${SPARKD__SIM_FIX_PORT}:${SPARKD__SIM_FIX_PORT}" - "${SPARKD__SIM_FIX_PORT}:${SPARKD__SIM_FIX_PORT}"
entrypoint: ${SPARKD__MAIN_ENTRYPOINT}
deploy:
restart_policy:
condition: any
delay: 3s
resources:
limits:
memory: "${SPARKD__MEMORY:-4gb}"
sparkd-multicast:
volumes:
- ${SPARKD__DATA_DIR:-sparkd_data}:/home/bts/.local/share/bts/sparkd/data
- ${SPARKD__MD_RECORDING_DIR:-md_recording}:/md_recording
- ${SPARKD__RUNTIME_CONF_DIR:-sparkd_conf}:/home/bts/.local/share/bts/sparkd/conf
build:
dockerfile: src/app_sparkd/Dockerfile
args:
- IMAGE=${SPARKD__IMAGE}
- SOURCE_DIR=${SPARKD__SOURCE_DIR}
- CONFIG_DIR=${SPARKD__CONFIG_DIR}
- PYTHON_REQUIREMENTS=${SPARKD__PYTHON_REQUIREMENTS}
- SPARKD_ENTRYPOINT=${SPARKD__SPARKD_ENTRYPOINT}
shm_size: "${SPARKD__SHM_SIZE:-2048mb}"
env_file:
- .env
- ${SPARKD__CONFIG_DIR}/.env
network_mode: "host"
entrypoint: ${SPARKD__MAIN_ENTRYPOINT}
deploy: deploy:
restart_policy: restart_policy:
condition: any condition: any
@@ -106,6 +77,7 @@ services:
resources: resources:
limits: limits:
memory: "${SPARKD__MEMORY:-4gb}" memory: "${SPARKD__MEMORY:-4gb}"
sparkd-tail: sparkd-tail:
image: bluetradingsystems/linux_dotnet8 image: bluetradingsystems/linux_dotnet8
volumes: volumes:
+108
View File
@@ -49,6 +49,14 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "spark_test", "src\client\sp
EndProject EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Spark.Console", "src\client\Spark.Console\Spark.Console.csproj", "{2B8243FF-60C6-4607-8AAE-7762D29B0A48}" Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Spark.Console", "src\client\Spark.Console\Spark.Console.csproj", "{2B8243FF-60C6-4607-8AAE-7762D29B0A48}"
EndProject EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Collections", "src\cs\Bts.Collections\Bts.Collections.csproj", "{00D21F82-61F4-467F-B125-268018A185E5}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Core", "src\cs\Bts.Core\Bts.Core.csproj", "{513A3F0D-46B5-4AC2-B700-8671933F6347}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Disposables", "src\cs\Bts.Disposables\Bts.Disposables.csproj", "{D419D996-9870-4FF5-82AD-08370D1C945F}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Numerics", "src\cs\Bts.Numerics\Bts.Numerics.csproj", "{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}"
EndProject
Global Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU Debug|Any CPU = Debug|Any CPU
@@ -275,6 +283,102 @@ Global
{2B8243FF-60C6-4607-8AAE-7762D29B0A48}.Release|x64.Build.0 = Release|x64 {2B8243FF-60C6-4607-8AAE-7762D29B0A48}.Release|x64.Build.0 = Release|x64
{2B8243FF-60C6-4607-8AAE-7762D29B0A48}.Release|x86.ActiveCfg = Release|x64 {2B8243FF-60C6-4607-8AAE-7762D29B0A48}.Release|x86.ActiveCfg = Release|x64
{2B8243FF-60C6-4607-8AAE-7762D29B0A48}.Release|x86.Build.0 = Release|x64 {2B8243FF-60C6-4607-8AAE-7762D29B0A48}.Release|x86.Build.0 = Release|x64
{00D21F82-61F4-467F-B125-268018A185E5}.Debug|x64.ActiveCfg = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.Debug|x64.Build.0 = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.Debug|Any CPU.Build.0 = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.Debug|x86.ActiveCfg = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.Debug|x86.Build.0 = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.MinSizeRel|x64.ActiveCfg = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.MinSizeRel|x64.Build.0 = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.MinSizeRel|Any CPU.ActiveCfg = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.MinSizeRel|Any CPU.Build.0 = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.MinSizeRel|x86.ActiveCfg = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.MinSizeRel|x86.Build.0 = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.Release|x64.ActiveCfg = Release|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.Release|x64.Build.0 = Release|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.Release|Any CPU.ActiveCfg = Release|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.Release|Any CPU.Build.0 = Release|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.Release|x86.ActiveCfg = Release|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.Release|x86.Build.0 = Release|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.RelWithDebInfo|x64.ActiveCfg = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.RelWithDebInfo|x64.Build.0 = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.RelWithDebInfo|Any CPU.ActiveCfg = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.RelWithDebInfo|Any CPU.Build.0 = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.RelWithDebInfo|x86.ActiveCfg = Debug|Any CPU
{00D21F82-61F4-467F-B125-268018A185E5}.RelWithDebInfo|x86.Build.0 = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.Debug|x64.ActiveCfg = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.Debug|x64.Build.0 = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.Debug|Any CPU.Build.0 = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.Debug|x86.ActiveCfg = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.Debug|x86.Build.0 = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.MinSizeRel|x64.ActiveCfg = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.MinSizeRel|x64.Build.0 = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.MinSizeRel|Any CPU.ActiveCfg = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.MinSizeRel|Any CPU.Build.0 = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.MinSizeRel|x86.ActiveCfg = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.MinSizeRel|x86.Build.0 = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.Release|x64.ActiveCfg = Release|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.Release|x64.Build.0 = Release|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.Release|Any CPU.ActiveCfg = Release|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.Release|Any CPU.Build.0 = Release|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.Release|x86.ActiveCfg = Release|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.Release|x86.Build.0 = Release|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.RelWithDebInfo|x64.ActiveCfg = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.RelWithDebInfo|x64.Build.0 = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.RelWithDebInfo|Any CPU.ActiveCfg = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.RelWithDebInfo|Any CPU.Build.0 = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.RelWithDebInfo|x86.ActiveCfg = Debug|Any CPU
{513A3F0D-46B5-4AC2-B700-8671933F6347}.RelWithDebInfo|x86.Build.0 = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.Debug|x64.ActiveCfg = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.Debug|x64.Build.0 = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.Debug|Any CPU.Build.0 = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.Debug|x86.ActiveCfg = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.Debug|x86.Build.0 = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.MinSizeRel|x64.ActiveCfg = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.MinSizeRel|x64.Build.0 = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.MinSizeRel|Any CPU.ActiveCfg = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.MinSizeRel|Any CPU.Build.0 = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.MinSizeRel|x86.ActiveCfg = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.MinSizeRel|x86.Build.0 = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.Release|x64.ActiveCfg = Release|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.Release|x64.Build.0 = Release|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.Release|Any CPU.ActiveCfg = Release|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.Release|Any CPU.Build.0 = Release|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.Release|x86.ActiveCfg = Release|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.Release|x86.Build.0 = Release|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.RelWithDebInfo|x64.ActiveCfg = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.RelWithDebInfo|x64.Build.0 = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.RelWithDebInfo|Any CPU.ActiveCfg = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.RelWithDebInfo|Any CPU.Build.0 = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.RelWithDebInfo|x86.ActiveCfg = Debug|Any CPU
{D419D996-9870-4FF5-82AD-08370D1C945F}.RelWithDebInfo|x86.Build.0 = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.Debug|x64.ActiveCfg = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.Debug|x64.Build.0 = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.Debug|Any CPU.Build.0 = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.Debug|x86.ActiveCfg = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.Debug|x86.Build.0 = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.MinSizeRel|x64.ActiveCfg = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.MinSizeRel|x64.Build.0 = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.MinSizeRel|Any CPU.ActiveCfg = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.MinSizeRel|Any CPU.Build.0 = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.MinSizeRel|x86.ActiveCfg = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.MinSizeRel|x86.Build.0 = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.Release|x64.ActiveCfg = Release|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.Release|x64.Build.0 = Release|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.Release|Any CPU.ActiveCfg = Release|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.Release|Any CPU.Build.0 = Release|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.Release|x86.ActiveCfg = Release|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.Release|x86.Build.0 = Release|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.RelWithDebInfo|x64.ActiveCfg = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.RelWithDebInfo|x64.Build.0 = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.RelWithDebInfo|Any CPU.ActiveCfg = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.RelWithDebInfo|Any CPU.Build.0 = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.RelWithDebInfo|x86.ActiveCfg = Debug|Any CPU
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049}.RelWithDebInfo|x86.Build.0 = Debug|Any CPU
EndGlobalSection EndGlobalSection
GlobalSection(SolutionProperties) = preSolution GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE HideSolutionNode = FALSE
@@ -302,5 +406,9 @@ Global
{D432F3E9-3D2F-4623-B7B9-CEBA6701FFC3} = {42C014A0-00C5-0620-8552-38912E891A7F} {D432F3E9-3D2F-4623-B7B9-CEBA6701FFC3} = {42C014A0-00C5-0620-8552-38912E891A7F}
{DD00F5D5-6C2F-4FDA-AE3D-25CF01A9A224} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295} {DD00F5D5-6C2F-4FDA-AE3D-25CF01A9A224} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295}
{2B8243FF-60C6-4607-8AAE-7762D29B0A48} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295} {2B8243FF-60C6-4607-8AAE-7762D29B0A48} = {C5A3B5EA-BE9F-76E5-E9B8-8A22B289B295}
{00D21F82-61F4-467F-B125-268018A185E5} = {D7C6B9C7-1058-B1EA-FEE6-34C56B05AEDB}
{513A3F0D-46B5-4AC2-B700-8671933F6347} = {D7C6B9C7-1058-B1EA-FEE6-34C56B05AEDB}
{D419D996-9870-4FF5-82AD-08370D1C945F} = {D7C6B9C7-1058-B1EA-FEE6-34C56B05AEDB}
{697BC3A3-1A98-43D3-A6E8-BB39A05EA049} = {D7C6B9C7-1058-B1EA-FEE6-34C56B05AEDB}
EndGlobalSection EndGlobalSection
EndGlobal EndGlobal
@@ -102,10 +102,10 @@ namespace BCG
List<ColumnInfo> col_info = new List<ColumnInfo>(); List<ColumnInfo> col_info = new List<ColumnInfo>();
col_info.Add(new ColumnInfo("id", col_index++, typeof(int))); col_info.Add(new ColumnInfo("id", col_index++, typeof(int)));
foreach (Pair<string, Type> p in group_by_col_info) foreach (Pair<string, Type> p in group_by_col_info)
col_info.Add(new ColumnInfo(p.first, col_index++, p.second)); col_info.Add(new ColumnInfo(p.first ?? string.Empty, col_index++, ExceptionAssert.NotNull(p.second)));
foreach (string tc_prefix in tc_prefixes) foreach (string tc_prefix in tc_prefixes)
foreach (Pair<string, Type> p in _cols_per_transposed) foreach (Pair<string, Type> p in _cols_per_transposed)
col_info.Add(new ColumnInfo(_get_full_column_name(tc_prefix, p.first ?? string.Empty), col_index++, p.second)); col_info.Add(new ColumnInfo(_get_full_column_name(tc_prefix, p.first ?? string.Empty), col_index++, ExceptionAssert.NotNull(p.second)));
IBlueTableSchema schema = new BlueTableSchema(col_info.ToArray(), "xy_transposed_" + table_name); IBlueTableSchema schema = new BlueTableSchema(col_info.ToArray(), "xy_transposed_" + table_name);
if (_schema == null) if (_schema == null)
_schema = schema; _schema = schema;
@@ -182,7 +182,7 @@ namespace BCG
if (_transpsed_col_value_comparer_wrapper != null) if (_transpsed_col_value_comparer_wrapper != null)
tc_values.Sort(_transpsed_col_value_comparer_wrapper); tc_values.Sort(_transpsed_col_value_comparer_wrapper);
List<string> prefixes = new List<string>(); List<string> prefixes = new List<string>();
foreach (Pair<int, object[]> p in tc_values) foreach (Pair<int, object?[]> p in tc_values)
prefixes.Add(tc_prefixes[p.first]); prefixes.Add(tc_prefixes[p.first]);
_set_schema(prefixes, group_by_col_info, table_name); _set_schema(prefixes, group_by_col_info, table_name);
BlueTable result = new BlueTable(_schema); BlueTable result = new BlueTable(_schema);
@@ -1,17 +1,17 @@
using Spark.Core.Diagnostics;
using System; using System;
using System.Collections; using System.Collections;
using System.Diagnostics.CodeAnalysis;
using System.Text; using System.Text;
using BCG;
using BCG.Utilities;
namespace BCG namespace BCG
{ {
public class ConcatenateColumnsMapper : IStreamMapper public class ConcatenateColumnsMapper : IStreamMapper
{ {
protected IDictionary myTempColMap = new SortedList (); protected IDictionary myTempColMap = new SortedList ();
protected ArrayList myColMap; protected ArrayList? myColMap;
protected BlueTableSchema mySchema; protected BlueTableSchema? mySchema;
public ConcatenateColumnsMapper(params string [] columnDefs) public ConcatenateColumnsMapper(params string [] columnDefs)
{ {
@@ -55,11 +55,7 @@ namespace BCG
myColMap = new ArrayList (myTempColMap.Count); myColMap = new ArrayList (myTempColMap.Count);
foreach (DictionaryEntry de in myTempColMap) foreach (DictionaryEntry de in myTempColMap)
{ {
string[] cols = de.Value as string[]; myColMap.Add(Array.ConvertAll((string[])de.Value!, c => inputSchema[c].colNo));
ArrayList temp = new ArrayList ();
foreach (string col in cols)
temp.Add (inputSchema[col].colNo);
myColMap.Add (temp.ToArray(typeof(int)) as int[]);
} }
myTempColMap.Clear (); myTempColMap.Clear ();
} }
@@ -72,6 +68,8 @@ namespace BCG
public virtual IBlueRow map (IBlueRow row) public virtual IBlueRow map (IBlueRow row)
{ {
requireSchema();
requireColMap();
object[] retVal = new object[mySchema.columnInfo.Length]; object[] retVal = new object[mySchema.columnInfo.Length];
row.CopyTo (retVal, myColMap.Count); row.CopyTo (retVal, myColMap.Count);
for (int colNo = 0; colNo < myColMap.Count; colNo ++) for (int colNo = 0; colNo < myColMap.Count; colNo ++)
@@ -90,11 +88,21 @@ namespace BCG
/// <returns></returns> /// <returns></returns>
protected virtual object myGetConcatValue (IBlueRow row, int colNo) protected virtual object myGetConcatValue (IBlueRow row, int colNo)
{ {
requireColMap();
StringBuilder b = new StringBuilder(); StringBuilder b = new StringBuilder();
foreach (int cn in myColMap[colNo] as int[]) foreach (int cn in (int[])myColMap[colNo]!) {
b.Append(row[cn].ToString()); b.Append(row[cn]?.ToString());
}
return b.ToString (); return b.ToString ();
} }
[MemberNotNull(nameof(myColMap))]
protected void requireColMap() =>
ExceptionAssert.NotNull(myColMap);
[MemberNotNull(nameof(mySchema))]
void requireSchema() =>
ExceptionAssert.NotNull(mySchema);
} }
public class ConcatenateColumnsMapperForSameDataSet : ConcatenateColumnsMapper public class ConcatenateColumnsMapperForSameDataSet : ConcatenateColumnsMapper
@@ -107,23 +115,21 @@ namespace BCG
myDicts = new HashTableND [columnDefs.Length]; myDicts = new HashTableND [columnDefs.Length];
int i = 0; int i = 0;
foreach (DictionaryEntry de in myTempColMap) foreach (DictionaryEntry de in myTempColMap)
myDicts[i++] = new HashTableND ((de.Value as string[]).Length); myDicts[i++] = new HashTableND(((string[])de.Value!).Length);
} }
protected override object myGetConcatValue (IBlueRow row, int colNo) protected override object myGetConcatValue (IBlueRow row, int colNo)
{ {
int[] cols = myColMap[colNo] as int[]; requireColMap();
object[] vals = new object[cols.Length];
for (int i = 0; i < vals.Length; i++)
vals[i] = row[cols[i]];
HashTableND ht = myDicts[colNo]; HashTableND ht = myDicts[colNo];
object rv = ht.getValue (vals); var vals = Array.ConvertAll((int[])myColMap[colNo]!, c => row[c]);
var rv = ht.getValue(vals);
if (rv != null) if (rv != null)
return rv; return rv;
else else
{ {
rv = base.myGetConcatValue (row, colNo); rv = base.myGetConcatValue (row, colNo);
object[] args = new object[vals.Length + 1]; var args = new object?[vals.Length + 1];
vals.CopyTo (args, 0); vals.CopyTo (args, 0);
args[args.Length-1] = rv; args[args.Length-1] = rv;
ht.add (args); ht.add (args);
@@ -20,7 +20,7 @@ namespace BCG
{ {
IDictionary<string, double> columnSums = new Dictionary<string, double>(); IDictionary<string, double> columnSums = new Dictionary<string, double>();
IBlueRow resultRow = null; IBlueRow? resultRow = null;
foreach (IBlueRow row in rows) foreach (IBlueRow row in rows)
{ {
resultRow = row; resultRow = row;
@@ -29,7 +29,7 @@ namespace BCG
{ {
double sum = 0.0f; double sum = 0.0f;
columnSums.TryGetValue(colName, out sum); columnSums.TryGetValue(colName, out sum);
sum += (double) row[colName]; sum += (double)row[colName]!;
columnSums[colName] = sum; columnSums[colName] = sum;
} }
} }
@@ -1,8 +1,3 @@
using System;
using System.Collections;
using BCG.Utilities;
namespace BCG namespace BCG
{ {
public class DistinctRowEvaluator : GroupByEvaluator public class DistinctRowEvaluator : GroupByEvaluator
@@ -67,7 +67,7 @@ namespace BCG {
} }
#endregion #endregion
public event FilterChangeDelegate onFilterChange; public event FilterChangeDelegate? onFilterChange;
#region IStreamFilter #region IStreamFilter
public bool filter(IBlueRow row) { public bool filter(IBlueRow row) {
@@ -43,7 +43,7 @@ namespace bts.SilverPine {
operations = operations_; operations = operations_;
} }
public static IBlueRow group_by(List<IBlueRow> rows, MultiRowOpHelper parameters) { public static IBlueRow? group_by(List<IBlueRow> rows, MultiRowOpHelper parameters) {
if (rows.Count() == 0) { if (rows.Count() == 0) {
return null; return null;
} }
@@ -73,7 +73,7 @@ namespace bts.SilverPine {
// rolling_rows are the rolling window of rows. the 0th row is the one thats gettin // rolling_rows are the rolling window of rows. the 0th row is the one thats gettin
// calculated. rows 1 - n-1 are its window rows. the schema is based on the operations. // calculated. rows 1 - n-1 are its window rows. the schema is based on the operations.
public IBlueRow rolling_calc(List<IBlueRow> rolling_rows) { public IBlueRow? rolling_calc(List<IBlueRow> rolling_rows) {
if (rolling_rows.Count() == 0) { if (rolling_rows.Count() == 0) {
return null; return null;
} }
@@ -123,4 +123,4 @@ namespace bts.SilverPine {
} }
} }
} }
@@ -1,4 +1,4 @@
using System; using Bts.DataStreamCore;
namespace BCG namespace BCG
{ {
@@ -9,18 +9,18 @@ namespace BCG
public abstract class GroupByEvaluator : IGroupByEvaluator public abstract class GroupByEvaluator : IGroupByEvaluator
{ {
protected string[] myGroupByCols; protected string[] myGroupByCols;
protected string myOutputColumn; protected string? myOutputColumn;
protected GroupByOutputTable myOutputTable; protected GroupByOutputTable? myOutputTable;
protected string myColMapExpression; protected string? myColMapExpression;
#region constructors #region constructors
public GroupByEvaluator (string outputCol, string[] groupByCols) public GroupByEvaluator (string? outputCol, string[] groupByCols)
{ {
myOutputColumn = outputCol; myOutputColumn = outputCol;
myGroupByCols = groupByCols; myGroupByCols = groupByCols;
} }
public GroupByEvaluator (string colMapExpression, string outputCol, string[] groupByCols) public GroupByEvaluator (string colMapExpression, string? outputCol, string[] groupByCols)
: this (outputCol, groupByCols) {myColMapExpression = colMapExpression;} : this (outputCol, groupByCols) {myColMapExpression = colMapExpression;}
#endregion #endregion
@@ -52,21 +52,20 @@ namespace BCG
/// <param name="r"></param> /// <param name="r"></param>
/// <param name="id"></param> /// <param name="id"></param>
/// <returns></returns> /// <returns></returns>
protected IBlueRow myProject (IBlueRow r, int id) protected IBlueRow? myProject(IBlueRow? r, int id) =>
{ r is null
return r == null ? null : ? null
myColMapExpression == null || myColMapExpression.Length < 1 ? : myColMapExpression is null || myColMapExpression.Length < 1
BlueDataUtility.getNewReplacedRow (r, id, "id") : ? BlueDataUtility.getNewReplacedRow(r, id, "id")
BlueDataUtility.getNewReplacedRow(myProjectRow (r, myColMapExpression), : BlueDataUtility.getNewReplacedRow(myProjectRow(r, myColMapExpression), id, "id");
id, "id");
}
/// <summary> /// <summary>
/// Projects the id, conserving the id /// Projects the id, conserving the id
/// </summary> /// </summary>
/// <param name="r"></param> /// <param name="r"></param>
/// <returns></returns> /// <returns></returns>
protected IBlueRow myProject (IBlueRow r) {return myProject (r, r == null ? Int32.MinValue : (int)r["id"]);} protected IBlueRow? myProject(IBlueRow? r) =>
myProject(r, r is null ? int.MinValue : r.Id());
#endregion #endregion
IBlueRow myProjectRow (IBlueRow r, string colMapExpression) => IBlueRow myProjectRow (IBlueRow r, string colMapExpression) =>
@@ -1,5 +1,6 @@
using Bts.DataStreamCore;
using System; using System;
using BCG.Utilities;
using System.Collections; using System.Collections;
namespace BCG namespace BCG
@@ -22,10 +23,12 @@ namespace BCG
/// <param name="groupRepString"></param> /// <param name="groupRepString"></param>
public void addRow (IBlueRow row, string groupRepString) public void addRow (IBlueRow row, string groupRepString)
{ {
if (myDict[groupRepString] != null) var grp = myDict[groupRepString];
myTable.deleteRow ((IBlueRow)myDict[groupRepString]); if (grp is not null) {
myTable.deleteRow((IBlueRow)grp);
}
myTable.addRow (row); myTable.addRow (row);
myDict[groupRepString] = myTable[(int)row["id"]]; myDict[groupRepString] = myTable[row.Id()];
} }
/// <summary> /// <summary>
@@ -35,7 +38,7 @@ namespace BCG
public void removeRow (string groupRepString) public void removeRow (string groupRepString)
{ {
// BlueEventLog.logAssert (myDict[groupRepString] != null); // BlueEventLog.logAssert (myDict[groupRepString] != null);
myTable.deleteRow ((IBlueRow)myDict[groupRepString]); myTable.deleteRow ((IBlueRow)myDict[groupRepString]!);
myDict.Remove (groupRepString); myDict.Remove (groupRepString);
} }
@@ -48,11 +51,8 @@ namespace BCG
myDict.Clear (); myDict.Clear ();
} }
public IBlueRow getRow (string groupRepString) public IBlueRow? getRow(string groupRepString) =>
{ myDict[groupRepString] as IBlueRow;
return myDict[groupRepString] == null ? null :
myDict[groupRepString] as IBlueRow;
}
#region IBlueRowCollection members #region IBlueRowCollection members
public IBlueRow this[int id] {get{return myTable[id];}} public IBlueRow this[int id] {get{return myTable[id];}}
@@ -1,7 +1,9 @@
using Spark.Core.Diagnostics;
using System; using System;
using System.Collections; using System.Collections;
using BCG.Utilities;
using System.Collections.Generic; using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.Linq; using System.Linq;
namespace BCG { namespace BCG {
@@ -26,11 +28,12 @@ namespace BCG {
/// Serves as a quick way to access all rows belonging to the same group /// Serves as a quick way to access all rows belonging to the same group
/// </summary> /// </summary>
private bool myIdGuard = true; private bool myIdGuard = true;
private RowDictionaryTable myBookKeepingTable; RowDictionaryTable? myBookKeepingTable;
private IGroupByEvaluator myGroupByEvaluator; private IGroupByEvaluator myGroupByEvaluator;
private IBlueTableSchema mySchema; //input schema, as output schema can differ (IGroupByEvaluator's // input schema, as output schema can differ (IGroupByEvaluator's
IBlueTableSchema? mySchema;
//can map a schema to another //can map a schema to another
private IBlueTableSchema myOutputSchema; IBlueTableSchema? myOutputSchema;
private int myIdGenerator = 0; private int myIdGenerator = 0;
public GroupByStream(ViewStream stream, IEnumerable<string> groupby_cols, Func<IDataPacket, IBlueRow> groupby_func) public GroupByStream(ViewStream stream, IEnumerable<string> groupby_cols, Func<IDataPacket, IBlueRow> groupby_func)
@@ -46,14 +49,14 @@ namespace BCG {
} }
protected override void myHandleFirstCall() { protected override void myHandleFirstCall() {
requireLastArrivedPacket();
mySchema = myLastArrivedPacket.schema; mySchema = myLastArrivedPacket.schema;
myOutputSchema = myGroupByEvaluator.getOutputSchema(mySchema); myOutputSchema = myGroupByEvaluator.getOutputSchema(mySchema);
myBookKeepingTable = new RowDictionaryTable(myLastArrivedPacket.newRows, myBookKeepingTable = new RowDictionaryTable(myLastArrivedPacket.newRows,
myGroupByEvaluator.groupByColumns); myGroupByEvaluator.groupByColumns);
GroupByOutputTable temp = new GroupByOutputTable(myOutputSchema); GroupByOutputTable temp = new GroupByOutputTable(myOutputSchema);
foreach (IBlueRowCollection rowGroup in myBookKeepingTable.getGroups()) { foreach (IBlueRowCollection rowGroup in myBookKeepingTable.getGroups()) {
IBlueRow newRow = myEvaluate(new DataPacket(rowGroup, var newRow = myEvaluate(new DataPacket(rowGroup, rowGroup, new BlueTable(mySchema)));
rowGroup, new BlueTable(mySchema)));
if (newRow != null) if (newRow != null)
temp.addRow(newRow, myGetGroupString(rowGroup)); temp.addRow(newRow, myGetGroupString(rowGroup));
} }
@@ -79,15 +82,14 @@ namespace BCG {
ArrayList handledAddGroups = new ArrayList(); ArrayList handledAddGroups = new ArrayList();
string groupRepString; string groupRepString;
requireBookKeeping();
foreach (IBlueRowCollection delGroup in delGroups.getGroups()) { foreach (IBlueRowCollection delGroup in delGroups.getGroups()) {
// BlueEventLog.logAssert (delGroup.Count > 0); // BlueEventLog.logAssert (delGroup.Count > 0);
groupRepString = myGetGroupString(delGroup); groupRepString = myGetGroupString(delGroup);
IBlueRowCollection addGroup = addGroups.getGroup(groupRepString) == null ? IBlueRowCollection addGroup = addGroups.getGroup(groupRepString) ?? new BlueTable(mySchema);
new BlueTable(mySchema) : addGroups.getGroup(groupRepString);
handledAddGroups.Add(addGroup); handledAddGroups.Add(addGroup);
IBlueRowCollection aggGroup = myBookKeepingTable.getGroup(groupRepString) == null ? IBlueRowCollection aggGroup = myBookKeepingTable.getGroup(groupRepString) ?? new BlueTable(mySchema);
new BlueTable(mySchema) : myBookKeepingTable.getGroup(groupRepString); var newRow = myEvaluate(new DataPacket(aggGroup, addGroup, delGroup));
IBlueRow newRow = myEvaluate(new DataPacket(aggGroup, addGroup, delGroup));
myFireSwitch |= myMergeRow(newRow, groupRepString); myFireSwitch |= myMergeRow(newRow, groupRepString);
} }
@@ -99,16 +101,16 @@ namespace BCG {
//the loop above //the loop above
groupRepString = myGetGroupString(addGroup); groupRepString = myGetGroupString(addGroup);
IBlueRowCollection aggGroup = myBookKeepingTable.getGroup(groupRepString); var aggGroup = ExceptionAssert.NotNull(myBookKeepingTable.getGroup(groupRepString));
// BlueEventLog.logAssert (aggGroup != null && aggGroup.Count > 0); // BlueEventLog.logAssert (aggGroup != null && aggGroup.Count > 0);
IBlueRow newRow = myEvaluate(new DataPacket(aggGroup, addGroup, var newRow = myEvaluate(new DataPacket(aggGroup, addGroup, new BlueTable(aggGroup.schema)));
new BlueTable(aggGroup.schema)));
myFireSwitch |= myMergeRow(newRow, groupRepString); myFireSwitch |= myMergeRow(newRow, groupRepString);
} }
} }
} }
private bool myMergeRow(IBlueRow newRow, string groupRepString) { private bool myMergeRow(IBlueRow? newRow, string groupRepString) {
requireLastFiredPacket();
IBlueRow existingRow = ((GroupByOutputTable)myLastFiredPacket.newRows).getRow(groupRepString); IBlueRow existingRow = ((GroupByOutputTable)myLastFiredPacket.newRows).getRow(groupRepString);
//BlueEventLog.logAssert (newRow != null || existingRow != null); //BlueEventLog.logAssert (newRow != null || existingRow != null);
if (existingRow == null && newRow == null) if (existingRow == null && newRow == null)
@@ -116,6 +118,7 @@ namespace BCG {
else if (existingRow != null && newRow != null && BlueDataUtility.compute_equality(existingRow, newRow)) else if (existingRow != null && newRow != null && BlueDataUtility.compute_equality(existingRow, newRow))
return false; return false;
else if (existingRow == null) { else if (existingRow == null) {
ArgumentNullException.ThrowIfNull(newRow);
//row added for the first time //row added for the first time
((GroupByOutputTable)myLastFiredPacket.newRows).addRow(newRow, groupRepString); ((GroupByOutputTable)myLastFiredPacket.newRows).addRow(newRow, groupRepString);
((BlueTable)myLastFiredPacket.addedRows).addRow(newRow); ((BlueTable)myLastFiredPacket.addedRows).addRow(newRow);
@@ -138,6 +141,9 @@ namespace BCG {
/// <param name="delGroups"></param> /// <param name="delGroups"></param>
/// <param name="addGroups"></param> /// <param name="addGroups"></param>
private void myInitIncrementalChangeData(out RowDictionaryTable delGroups, out RowDictionaryTable addGroups) { private void myInitIncrementalChangeData(out RowDictionaryTable delGroups, out RowDictionaryTable addGroups) {
ExceptionAssert.NotNull(myBookKeepingTable);
requireLastFiredPacket();
requireLastArrivedPacket();
delGroups = delGroups =
new RowDictionaryTable(myLastArrivedPacket.deletedRows, myGroupByEvaluator.groupByColumns); new RowDictionaryTable(myLastArrivedPacket.deletedRows, myGroupByEvaluator.groupByColumns);
addGroups = addGroups =
@@ -151,16 +157,19 @@ namespace BCG {
} }
protected override void myOnTableClear() { protected override void myOnTableClear() {
if (myBookKeepingTable.Count == 0) if (ExceptionAssert.NotNull(myBookKeepingTable).Count == 0) {
return; //ie there arent any rows, so no need to optimize the 'cleansing' process // ie there arent any rows, so no need to optimize the 'cleansing' process
return;
}
requireLastFiredPacket();
myLastFiredPacket.deletedRows = new BlueTable(myLastFiredPacket.newRows); myLastFiredPacket.deletedRows = new BlueTable(myLastFiredPacket.newRows);
myLastFiredPacket.addedRows = new BlueTable(myLastFiredPacket.deletedRows.schema); myLastFiredPacket.addedRows = new BlueTable(myLastFiredPacket.deletedRows.schema);
(myLastFiredPacket.newRows as GroupByOutputTable).clear(); ((GroupByOutputTable)myLastFiredPacket.newRows).clear();
myBookKeepingTable.clear(); myBookKeepingTable.clear();
myGroupByEvaluator.clear(); myGroupByEvaluator.clear();
} }
private IBlueRow myEvaluate(DataPacket dp) { private IBlueRow? myEvaluate(DataPacket dp) {
IBlueRow evalRow = myGroupByEvaluator.evaluate(dp); IBlueRow evalRow = myGroupByEvaluator.evaluate(dp);
return evalRow == null ? null : return evalRow == null ? null :
myIdGuard ? myIdGuard ?
@@ -192,7 +201,7 @@ namespace BCG {
return BlueDataUtility.rowToString(getRepRow(dp), colNames); return BlueDataUtility.rowToString(getRepRow(dp), colNames);
} }
public static IBlueRow getRepRow(IDataPacket dp) { public static IBlueRow? getRepRow(IDataPacket dp) {
if (dp.newRows.Count > 0) if (dp.newRows.Count > 0)
return BlueDataUtility.extractRow(dp.newRows, 0); return BlueDataUtility.extractRow(dp.newRows, 0);
else if (dp.addedRows.Count > 0) else if (dp.addedRows.Count > 0)
@@ -211,13 +220,13 @@ namespace BCG {
/// <param name="newValue"></param> /// <param name="newValue"></param>
/// <param name="isAdd"></param> /// <param name="isAdd"></param>
/// <returns></returns> /// <returns></returns>
public static object incrementalAdd(object lastValue, object newValue, bool isAdd) { public static object? incrementalAdd(object lastValue, object newValue, bool isAdd) {
if (lastValue == null || lastValue.Equals(DBNull.Value)) if (lastValue == null || lastValue.Equals(DBNull.Value))
return newValue; return newValue;
// BlueEventLog.logAssert (lastValue.GetType().Equals (typeof(Int32)) && newValue.GetType().Equals (typeof(Int32)) || // BlueEventLog.logAssert (lastValue.GetType().Equals (typeof(Int32)) && newValue.GetType().Equals (typeof(Int32)) ||
// lastValue.GetType().Equals (typeof(Double)) && newValue.GetType().Equals (typeof(Double)) || // lastValue.GetType().Equals (typeof(Double)) && newValue.GetType().Equals (typeof(Double)) ||
// lastValue.GetType().Equals (typeof(Decimal)) && newValue.GetType().Equals (typeof(Decimal))); // lastValue.GetType().Equals (typeof(Decimal)) && newValue.GetType().Equals (typeof(Decimal)));
Object retVal = null; object? retVal = null;
if (newValue.GetType().Equals(typeof(int))) if (newValue.GetType().Equals(typeof(int)))
retVal = isAdd ? (int)lastValue + (int)newValue : (int)lastValue - (int)newValue; retVal = isAdd ? (int)lastValue + (int)newValue : (int)lastValue - (int)newValue;
else if (newValue.GetType().Equals(typeof(Double))) else if (newValue.GetType().Equals(typeof(Double)))
@@ -229,6 +238,9 @@ namespace BCG {
return retVal; return retVal;
} }
[MemberNotNull(nameof(myBookKeepingTable))]
void requireBookKeeping() =>
ExceptionAssert.NotNull(myBookKeepingTable);
} }
} }
@@ -1,8 +1,3 @@
using System;
using System.Collections;
using BCG.Utilities;
namespace BCG { namespace BCG {
/// <summary> /// <summary>
/// Computes the max for a given column /// Computes the max for a given column
@@ -17,10 +12,10 @@ namespace BCG {
public override IBlueRow? evaluate(IDataPacket dp) { public override IBlueRow? evaluate(IDataPacket dp) {
myOutputTable ??= new GroupByOutputTable(dp.Schema); myOutputTable ??= new GroupByOutputTable(dp.Schema);
string hashString = BlueDataUtility.rowToString(GroupByStream.getRepRow(dp), myGroupByCols); string hashString = BlueDataUtility.rowToString(GroupByStream.getRepRow(dp), myGroupByCols);
IBlueRow currMaxRow = myOutputTable.getRow(hashString); var currMaxRow = myOutputTable.getRow(hashString);
IBlueRow maxRow = BlueDataUtility.computeMaxRow(dp.deletedRows, myOutputColumn); IBlueRow maxRow = BlueDataUtility.computeMaxRow(dp.deletedRows, myOutputColumn);
if (maxRow != null) { if (maxRow != null) {
bts.Log.assert(currMaxRow != null); bts.Log.notnull(currMaxRow);
bts.Log.assert(BlueDataUtility.compare_rows(maxRow, currMaxRow, myOutputColumn) <= 0); bts.Log.assert(BlueDataUtility.compare_rows(maxRow, currMaxRow, myOutputColumn) <= 0);
if (BlueDataUtility.compare_rows(maxRow, currMaxRow, myOutputColumn) == 0) { if (BlueDataUtility.compare_rows(maxRow, currMaxRow, myOutputColumn) == 0) {
maxRow = BlueDataUtility.computeMaxRow(dp.newRows, myOutputColumn); maxRow = BlueDataUtility.computeMaxRow(dp.newRows, myOutputColumn);
@@ -1,10 +1,9 @@
using System;
using System.Collections;
using BCG.Utilities;
using System.IO;
namespace BCG namespace BCG
{ {
using Spark.Core.Diagnostics;
using System.Diagnostics.CodeAnalysis;
/// <summary> /// <summary>
/// Some basic event handling functionality factorred out of any Stream whose behaviour /// Some basic event handling functionality factorred out of any Stream whose behaviour
/// is driven by some interface passed on by the user /// is driven by some interface passed on by the user
@@ -12,8 +11,8 @@ namespace BCG
public abstract class MethodDrivenStream : ViewStream public abstract class MethodDrivenStream : ViewStream
{ {
protected ViewStream myStream; protected ViewStream myStream;
protected DataPacket myLastArrivedPacket; protected DataPacket? myLastArrivedPacket;
protected DataPacket myLastFiredPacket; protected DataPacket? myLastFiredPacket;
protected bool myFireSwitch = true; protected bool myFireSwitch = true;
public MethodDrivenStream(ViewStream vs) {myStream = vs;} public MethodDrivenStream(ViewStream vs) {myStream = vs;}
@@ -59,6 +58,7 @@ namespace BCG
/// To handle the first call, the LastFiredPacket is supposed to be initialized and filled /// To handle the first call, the LastFiredPacket is supposed to be initialized and filled
/// with data. /// with data.
/// </summary> /// </summary>
[MemberNotNull(nameof(myLastFiredPacket))]
protected abstract void myHandleFirstCall (); protected abstract void myHandleFirstCall ();
/// <summary> /// <summary>
@@ -89,5 +89,13 @@ namespace BCG
{ {
myStream.onChange -= new DataStream.ChangeDelegate (myChangeHandler); myStream.onChange -= new DataStream.ChangeDelegate (myChangeHandler);
} }
[MemberNotNull(nameof(myLastArrivedPacket))]
protected void requireLastArrivedPacket() =>
ExceptionAssert.NotNull(myLastArrivedPacket);
[MemberNotNull(nameof(myLastFiredPacket))]
protected void requireLastFiredPacket() =>
ExceptionAssert.NotNull(myLastFiredPacket);
} }
} }
@@ -9,7 +9,7 @@ namespace BCG {
public override IBlueRow? evaluate(IDataPacket dp) { public override IBlueRow? evaluate(IDataPacket dp) {
myOutputTable ??= new GroupByOutputTable(dp.Schema); myOutputTable ??= new GroupByOutputTable(dp.Schema);
string stringRep = BlueDataUtility.rowToString(GroupByStream.getRepRow(dp), myGroupByCols); string stringRep = BlueDataUtility.rowToString(GroupByStream.getRepRow(dp), myGroupByCols);
IBlueRow currMinRow = myOutputTable.getRow(stringRep); var currMinRow = myOutputTable.getRow(stringRep);
IBlueRow minRow = BlueDataUtility.computeMinRow(dp.deletedRows, myOutputColumn); IBlueRow minRow = BlueDataUtility.computeMinRow(dp.deletedRows, myOutputColumn);
if (minRow != null) { if (minRow != null) {
// BlueEventLog.logAssert (currMinRow != null); // BlueEventLog.logAssert (currMinRow != null);
@@ -45,12 +45,13 @@ namespace BCG {
} }
public SumStream(ViewStream stream, string[] output_columns, string[] groupby_columns, bool force_null_into_other_columns) { public SumStream(ViewStream stream, string[] output_columns, string[] groupby_columns, bool force_null_into_other_columns) {
ViewStream rs = null; ViewStream? rs = null;
groupby_columns = groupby_columns ?? new string[] { }; groupby_columns = groupby_columns ?? new string[] { };
if (groupby_columns.Length == 0) { if (groupby_columns.Length == 0) {
// rs = new AddColumnsStream(stream, "fake_col=0"); // rs = new AddColumnsStream(stream, "fake_col=0");
// rs = new GroupByStream(rs, new MultiColumnSumGroupByEvaluator(output_columns, new string[] { "fake_col" })); // rs = new GroupByStream(rs, new MultiColumnSumGroupByEvaluator(output_columns, new string[] { "fake_col" }));
// rs = new ProjectStream(rs, "!fake_col"); // rs = new ProjectStream(rs, "!fake_col");
throw new ArgumentException("No groupby columns specified", nameof(groupby_columns));
} else { } else {
rs = new GroupByStream(stream, new MultiColumnSumGroupByEvaluator(output_columns, groupby_columns)); rs = new GroupByStream(stream, new MultiColumnSumGroupByEvaluator(output_columns, groupby_columns));
} }
@@ -1,5 +1,7 @@
using mbts; using mbts;
using Spark.Core.Diagnostics;
using System; using System;
using System.Collections.Generic; using System.Collections.Generic;
@@ -35,7 +37,7 @@ namespace BCG {
List<NonIncJoinSpec> rv = new List<NonIncJoinSpec>(); List<NonIncJoinSpec> rv = new List<NonIncJoinSpec>();
foreach (Triplet<ViewStream, string, string> r in join_specs) { foreach (Triplet<ViewStream, string, string> r in join_specs) {
string[] keys = String.IsNullOrEmpty(r.second) ? new string[] { } : r.second.Split(' '); string[] keys = String.IsNullOrEmpty(r.second) ? new string[] { } : r.second.Split(' ');
string[] visible_cols = r.third.Split(' '); string[] visible_cols = ExceptionAssert.NotNull(r.third).Split(' ');
List<string> data_col_names = new List<string>(), mapped_col_names = new List<string>(); List<string> data_col_names = new List<string>(), mapped_col_names = new List<string>();
foreach (string s in visible_cols) { foreach (string s in visible_cols) {
string[] toks = s.Split(':'); string[] toks = s.Split(':');
@@ -47,7 +49,7 @@ namespace BCG {
mapped_col_names.Add(toks[1]); mapped_col_names.Add(toks[1]);
} }
} }
rv.Add(new NonIncJoinSpec(r.first, keys, data_col_names.ToArray(), mapped_col_names.ToArray())); rv.Add(new(ExceptionAssert.NotNull(r.first), keys, data_col_names.ToArray(), mapped_col_names.ToArray()));
} }
return rv.ToArray(); return rv.ToArray();
} }
@@ -290,6 +290,7 @@ namespace BCG
} }
} }
} }
ExceptionAssert.NotNull(myState);
} }
private void myAddRows(IBlueRowCollection addedRows, bool isLeft, BlueTable state, BlueTable addedRowsOut) private void myAddRows(IBlueRowCollection addedRows, bool isLeft, BlueTable state, BlueTable addedRowsOut)
@@ -1,3 +1,5 @@
using Spark.Core.Diagnostics;
using System; using System;
namespace BCG namespace BCG
@@ -18,16 +20,21 @@ namespace BCG
public override void onTableClear (bool isLeft) public override void onTableClear (bool isLeft)
{ {
requireSchema();
requireLastFiredPacket();
myLastFiredPacket.deletedRows = myLastFiredPacket.newRows; myLastFiredPacket.deletedRows = myLastFiredPacket.newRows;
myLastFiredPacket.newRows = new JoinForeignTable (mySchema); myLastFiredPacket.newRows = new JoinForeignTable (mySchema);
myLastFiredPacket.addedRows = new BlueTable (mySchema); myLastFiredPacket.addedRows = new BlueTable (mySchema);
if (!isLeft) if (!isLeft) {
ExceptionAssert.NotNull(myLeftDataPacket);
foreach (IBlueRow row in myLeftDataPacket.newRows) foreach (IBlueRow row in myLeftDataPacket.newRows)
((JoinTable)myLastFiredPacket.newRows).addRow (row, null, myLastFiredPacket); ((JoinTable)myLastFiredPacket.newRows).addRow (row, null, myLastFiredPacket);
}
} }
protected override void myInitFiredPacket () protected override void myInitFiredPacket ()
{ {
requireSchema();
myLastFiredPacket = new DataPacket (); myLastFiredPacket = new DataPacket ();
myLastFiredPacket.newRows = new JoinForeignTable (mySchema); myLastFiredPacket.newRows = new JoinForeignTable (mySchema);
myLastFiredPacket.addedRows = new BlueTable (mySchema); myLastFiredPacket.addedRows = new BlueTable (mySchema);
@@ -15,7 +15,7 @@ namespace BCG
: this (r, null, null) : this (r, null, null)
{} {}
public SimpleNode(IBlueRow r, SimpleNode? nx, SimpleNode? pv) { public SimpleNode(IBlueRow? r, SimpleNode? nx, SimpleNode? pv) {
row = r; row = r;
next = nx; next = nx;
prev = pv; prev = pv;
@@ -1,5 +1,7 @@
using bts; using bts;
using Spark.Core.Diagnostics;
using System; using System;
using System.Collections.Generic; using System.Collections.Generic;
@@ -60,7 +62,7 @@ namespace BCG {
if (columnAndType.Length == 1) if (columnAndType.Length == 1)
ci[n] = new ColumnInfo(c, n, typeof(double)); ci[n] = new ColumnInfo(c, n, typeof(double));
else else
ci[n] = new ColumnInfo(columnAndType[0], n, Type.GetType("System." + columnAndType[1], true)); ci[n] = new(columnAndType[0], n, ExceptionAssert.NotNull(Type.GetType("System." + columnAndType[1], true)));
n++; n++;
} }
@@ -44,7 +44,7 @@ namespace bts {
BlueDataUtility.group_together(dp.newRows, new string[] { _group_by_col }) BlueDataUtility.group_together(dp.newRows, new string[] { _group_by_col })
: :
new Dictionary<int, List<IBlueRow>>() { { 0, dp.newRows.to_list() } }; new Dictionary<int, List<IBlueRow>>() { { 0, dp.newRows.to_list() } };
BlueTable out_table = null; BlueTable? out_table = null;
foreach (var kv in groups) { foreach (var kv in groups) {
List<IBlueRow> group_rows = kv.Value; List<IBlueRow> group_rows = kv.Value;
group_rows.Sort((r1, r2) => BlueDataUtility.compare_rows(r2, r1, _sort_col_name)); group_rows.Sort((r1, r2) => BlueDataUtility.compare_rows(r2, r1, _sort_col_name));
@@ -73,4 +73,4 @@ namespace bts {
} }
} }
@@ -23,6 +23,6 @@ namespace BCG
/// fires an event when the mapper /// fires an event when the mapper
/// changes itself /// changes itself
/// </summary> /// </summary>
event MapperChangeDelegate onMapperChange; event MapperChangeDelegate? onMapperChange;
} }
} }
@@ -75,6 +75,7 @@ namespace BCG {
protected override void myHandleFirstCall() { protected override void myHandleFirstCall() {
ArgumentNullException.ThrowIfNull(myMapper); ArgumentNullException.ThrowIfNull(myMapper);
requireLastArrivedPacket();
IBlueTableSchema outputSchema = myMapper.getOutputSchema(myLastArrivedPacket.newRows.schema); IBlueTableSchema outputSchema = myMapper.getOutputSchema(myLastArrivedPacket.newRows.schema);
myLastFiredPacket = new DataPacket(); myLastFiredPacket = new DataPacket();
myLastFiredPacket.newRows = new BlueTable(outputSchema); myLastFiredPacket.newRows = new BlueTable(outputSchema);
@@ -85,10 +86,12 @@ namespace BCG {
} }
protected override void myHandleIncrementalAdd() { protected override void myHandleIncrementalAdd() {
requireLastArrivedPacket();
if (myLastArrivedPacket.addedRows.Count == 0) { if (myLastArrivedPacket.addedRows.Count == 0) {
Log.Gen.warn("MapStream.myHandleIncrementalAdd called with 0 addedRows"); Log.Gen.warn("MapStream.myHandleIncrementalAdd called with 0 addedRows");
return; return;
} }
requireLastFiredPacket();
ArgumentNullException.ThrowIfNull(myMapper); ArgumentNullException.ThrowIfNull(myMapper);
var addedRows = new BlueTable(myLastFiredPacket.newRows.schema); var addedRows = new BlueTable(myLastFiredPacket.newRows.schema);
var newRows = ExceptionAssert.Cast(myLastFiredPacket.newRows).To<BlueTable>(); var newRows = ExceptionAssert.Cast(myLastFiredPacket.newRows).To<BlueTable>();
@@ -101,10 +104,12 @@ namespace BCG {
} }
protected override void myHandleIncrementalDelete() { protected override void myHandleIncrementalDelete() {
requireLastArrivedPacket();
if (myLastArrivedPacket.deletedRows.Count == 0) { if (myLastArrivedPacket.deletedRows.Count == 0) {
Log.Gen.warn("MapStream.myHandleIncrementalDelete called with 0 deletedRows"); Log.Gen.warn("MapStream.myHandleIncrementalDelete called with 0 deletedRows");
return; return;
} }
requireLastFiredPacket();
var deletedRows = new BlueTable(myLastFiredPacket.newRows.schema); var deletedRows = new BlueTable(myLastFiredPacket.newRows.schema);
var newRows = ExceptionAssert.Cast(myLastFiredPacket.newRows).To<BlueTable>(); var newRows = ExceptionAssert.Cast(myLastFiredPacket.newRows).To<BlueTable>();
foreach (IBlueRow row in myLastArrivedPacket.deletedRows) { foreach (IBlueRow row in myLastArrivedPacket.deletedRows) {
@@ -127,6 +132,7 @@ namespace BCG {
} }
protected override void myOnTableClear() { protected override void myOnTableClear() {
requireLastFiredPacket();
myLastFiredPacket.deletedRows = myLastFiredPacket.newRows; myLastFiredPacket.deletedRows = myLastFiredPacket.newRows;
myLastFiredPacket.newRows = new BlueTable(myLastFiredPacket.deletedRows.schema); myLastFiredPacket.newRows = new BlueTable(myLastFiredPacket.deletedRows.schema);
myLastFiredPacket.addedRows = new BlueTable(myLastFiredPacket.deletedRows.schema); myLastFiredPacket.addedRows = new BlueTable(myLastFiredPacket.deletedRows.schema);
@@ -47,14 +47,14 @@ namespace BCG {
_map_rows = map_rows; _map_rows = map_rows;
} }
event MapperChangeDelegate? ISuperStreamMapper.onMapperChange { add { } remove { } }
public IBlueTableSchema getOutputSchema(IBlueTableSchema schema) { return _get_schema(schema); } public IBlueTableSchema getOutputSchema(IBlueTableSchema schema) { return _get_schema(schema); }
public IBlueRow[] map(IBlueRow r) { public IBlueRow[] map(IBlueRow r) {
IEnumerable<IBlueRow> rows = _map_rows(r); IEnumerable<IBlueRow> rows = _map_rows(r);
return rows == null ? new IBlueRow[] { } : new List<IBlueRow>(rows).ToArray(); return rows == null ? new IBlueRow[] { } : new List<IBlueRow>(rows).ToArray();
} }
public event MapperChangeDelegate? onMapperChange;
} }
//maps the id of the underlying stream to the //maps the id of the underlying stream to the
@@ -608,8 +608,8 @@ namespace BCG {
// 5. deserialize the state into a stream op // 5. deserialize the state into a stream op
str_io_msg_p = deser.deserialize(content_p); str_io_msg_p = deser.deserialize(content_p);
// 6. check to make sure the state's rows are good // 6. check to make sure the state's rows are good
passes = passes && str_io_msg_p.op == bts.StreamIOMessage.StreamOp.CREATE && passes &= str_io_msg_p.op == StreamIOMessage.StreamOp.CREATE
BlueDataUtility.compute_equality(cr_rows, str_io_msg_p.create_message.rows); && BlueDataUtility.compute_equality(cr_rows, ExceptionAssert.NotNull(str_io_msg_p.create_message).rows);
return new TestResult { return new TestResult {
test_result = passes, test_result = passes,
test_name = "run_pubsub_server_stream_tag_handler_test" test_name = "run_pubsub_server_stream_tag_handler_test"
@@ -692,7 +692,7 @@ namespace BCG {
var content = st_data.get_current_content(); var content = st_data.get_current_content();
var msg_p = deser.deserialize(content); var msg_p = deser.deserialize(content);
Log.assert(msg_p.op == bts.StreamIOMessage.StreamOp.CREATE); Log.assert(msg_p.op == bts.StreamIOMessage.StreamOp.CREATE);
return msg_p.create_message.rows; return ExceptionAssert.NotNull(msg_p.create_message).rows;
} }
@@ -1,5 +1,7 @@
using Bts.ColumnSortExpressions; using Bts.ColumnSortExpressions;
using Spark.Core.Diagnostics;
using System; using System;
using System.Collections; using System.Collections;
using System.Collections.Generic; using System.Collections.Generic;
@@ -39,7 +41,7 @@ public class SortRow : IComparable, IBlueRow {
var otherVal = otherRow[col.Name]; var otherVal = otherRow[col.Name];
int rv = myIsNull(this[col.Name]) ? int rv = myIsNull(this[col.Name]) ?
(myIsNull(otherVal) ? 0 : -1) : (myIsNull(otherVal) ? 0 : -1) :
(myIsNull(otherVal) ? 1 : ((IComparable)this[col.Name]).CompareTo(otherVal)); (myIsNull(otherVal) ? 1 : ExceptionAssert.Argument.Is<IComparable>(this[col.Name]).CompareTo(otherVal));
rv *= col.IsAscending ? 1 : -1; rv *= col.IsAscending ? 1 : -1;
return rv; return rv;
} }
@@ -1,4 +1,6 @@
using System; using Spark.Core.Diagnostics;
using System;
using System.Collections.Generic; using System.Collections.Generic;
using System.IO; using System.IO;
@@ -60,13 +62,13 @@ namespace BCG
List<string> columnInfoStrParts = new List<string>(columnInfoStr.Split(new char[] { ':' }, StringSplitOptions.None)); List<string> columnInfoStrParts = new List<string>(columnInfoStr.Split(new char[] { ':' }, StringSplitOptions.None));
if (columnInfoStrParts.Count != 2) if (columnInfoStrParts.Count != 2)
{ {
throw (new Exception(String.Format("Unable to decode schema string '{0}'", schemaStr))); throw new Exception($"Unable to decode schema string '{schemaStr}'");
} }
columnInfos.Add(new ColumnInfo(columnInfoStrParts[0], i, Type.GetType(columnInfoStrParts[1]))); columnInfos.Add(new ColumnInfo(columnInfoStrParts[0], i, ExceptionAssert.NotNull(Type.GetType(columnInfoStrParts[1]))));
} }
return (new BlueTableSchema(columnInfos.ToArray(), "")); return new BlueTableSchema(columnInfos.ToArray(), string.Empty);
} }
private IBlueRow myGetRow(IBlueTableSchema schema, string rowStr) private IBlueRow myGetRow(IBlueTableSchema schema, string rowStr)
@@ -75,7 +77,7 @@ namespace BCG
if (schema.columnInfo.Length != valueStrs.Count) if (schema.columnInfo.Length != valueStrs.Count)
{ {
throw (new Exception(String.Format("Unable to decode row string '{0}'", rowStr))); throw new Exception($"Unable to decode row string '{rowStr}'");
} }
List<object> values = new List<object>(); List<object> values = new List<object>();
@@ -84,7 +86,7 @@ namespace BCG
values.Add(Convert.ChangeType(valueStrs[i], schema.columnInfo[i].colType)); values.Add(Convert.ChangeType(valueStrs[i], schema.columnInfo[i].colType));
} }
return (new BlueRow(schema, values.ToArray())); return new BlueRow(schema, values.ToArray());
} }
#endregion #endregion
@@ -1,6 +1,7 @@
using BCG.Utilities.General; using BCG.Utilities.General;
using System; using System;
using System.Collections.Generic;
using System.Threading; using System.Threading;
namespace BCG namespace BCG
@@ -97,7 +98,7 @@ namespace BCG
// BlueEventLog2.instance().assert(Thread.CurrentThread.ManagedThreadId == _target_thread, // BlueEventLog2.instance().assert(Thread.CurrentThread.ManagedThreadId == _target_thread,
// "onTargetDoWork should only be called from target thread"); // "onTargetDoWork should only be called from target thread");
var updates = _queued_updates.dequeueAll(); var updates = _queued_updates.dequeueAll();
foreach (var u in updates) { foreach (var u in updates.ExceptNull()) {
if (u.subscription == _current_target_subscription) if (u.subscription == _current_target_subscription)
myFireEvent(u.dp); myFireEvent(u.dp);
} }
@@ -1,7 +1,9 @@
using System;
using BCG.Utilities.General; using BCG.Utilities.General;
using Spark.Core.Diagnostics;
using System;
namespace BCG { namespace BCG {
public abstract class TimeUnitStream : ViewStream, IWorker { public abstract class TimeUnitStream : ViewStream, IWorker {
protected TimeSpan myOffset; protected TimeSpan myOffset;
@@ -46,8 +48,8 @@ namespace BCG {
myRoundTime(myZeroTime + new TimeSpan(myOffset.Ticks * periods)); myRoundTime(myZeroTime + new TimeSpan(myOffset.Ticks * periods));
if (myGetLatestRow() == null || var latestRow = myGetLatestRow();
!myGetLatestRow()["time"].Equals(dt)) { if (latestRow is null || !ExceptionAssert.Argument.Is<DateTime>(latestRow["time"]).Equals(dt)) {
IBlueRow ar = new BlueRow(myCollection.schema, IBlueRow ar = new BlueRow(myCollection.schema,
myId++, dt); myId++, dt);
IBlueRow dr = myCollection.addRowWithReturn(ar); IBlueRow dr = myCollection.addRowWithReturn(ar);
@@ -71,11 +73,10 @@ namespace BCG {
#region Private Helper methods #region Private Helper methods
private IBlueRow myGetLatestRow() { IBlueRow? myGetLatestRow() =>
return myCollection.Count == 0 ? myCollection.Count == 0
null : ? null
BlueDataUtility.extractRow(myCollection, myCollection.Count - 1); : BlueDataUtility.extractRow(myCollection, myCollection.Count - 1);
}
private IBlueRowCollection myConvertToCollection(IBlueRow r) { private IBlueRowCollection myConvertToCollection(IBlueRow r) {
BlueTable retVal = new BlueTable(myCollection.schema); BlueTable retVal = new BlueTable(myCollection.schema);
@@ -1,43 +1,42 @@
using System; namespace BCG;
namespace BCG { public class TopStream : MethodDrivenStream {
public class TopStream : MethodDrivenStream { int myTopCount;
private int myTopCount;
public TopStream(ViewStream stream, int topCount) : base(stream) { myTopCount = topCount; } public TopStream(ViewStream stream, int topCount)
: base(stream) =>
myTopCount = topCount;
public TopStream(ViewStream stream, public TopStream(ViewStream stream, int topCount, string sortExpression)
int topCount, : this(sortExpression is null ? stream : new SortStream(stream, sortExpression), topCount) { }
string sortExpression)
:
this(sortExpression == null ?
stream :
new SortStream(stream, sortExpression), topCount) { }
protected override void myOnTableClear() { protected override void myOnTableClear() {
myLastFiredPacket.deletedRows = myLastFiredPacket.newRows; requireLastFiredPacket();
myLastFiredPacket.newRows = new OrderedBlueTable(myLastFiredPacket.deletedRows.schema); myLastFiredPacket.deletedRows = myLastFiredPacket.newRows;
myLastFiredPacket.addedRows = new OrderedBlueTable(myLastFiredPacket.deletedRows.schema); myLastFiredPacket.newRows = new OrderedBlueTable(myLastFiredPacket.deletedRows.schema);
} myLastFiredPacket.addedRows = new OrderedBlueTable(myLastFiredPacket.deletedRows.schema);
}
protected override void myHandleFirstCall() { protected override void myHandleFirstCall() {
myLastFiredPacket = new DataPacket(); requireLastArrivedPacket();
myLastFiredPacket.newRows = new OrderedBlueTable(myLastArrivedPacket.newRows, myTopCount); myLastFiredPacket = new DataPacket();
myLastFiredPacket.addedRows = new OrderedBlueTable(myLastFiredPacket.newRows); myLastFiredPacket.newRows = new OrderedBlueTable(myLastArrivedPacket.newRows, myTopCount);
myLastFiredPacket.deletedRows = new OrderedBlueTable(myLastFiredPacket.newRows.schema); myLastFiredPacket.addedRows = new OrderedBlueTable(myLastFiredPacket.newRows);
} myLastFiredPacket.deletedRows = new OrderedBlueTable(myLastFiredPacket.newRows.schema);
}
protected override void myHandleIncrementalAdd() { } protected override void myHandleIncrementalAdd() { }
protected override void myHandleIncrementalDelete() { protected override void myHandleIncrementalDelete() {
IBlueRowCollection oldRows = myLastFiredPacket.newRows; requireLastArrivedPacket();
myLastFiredPacket.newRows = new OrderedBlueTable(myLastArrivedPacket.newRows, myTopCount); requireLastFiredPacket();
BlueDataUtility.computeDifferences(myLastFiredPacket.newRows, IBlueRowCollection oldRows = myLastFiredPacket.newRows;
oldRows, myLastFiredPacket.newRows = new OrderedBlueTable(myLastArrivedPacket.newRows, myTopCount);
myLastFiredPacket.schema.columnNames, BlueDataUtility.computeDifferences(myLastFiredPacket.newRows,
out myLastFiredPacket.deletedRows, oldRows,
out myLastFiredPacket.addedRows); myLastFiredPacket.schema.columnNames,
myFireSwitch = myLastFiredPacket.addedRows.Count + myLastFiredPacket.deletedRows.Count == 0 ? false : true; out myLastFiredPacket.deletedRows,
} out myLastFiredPacket.addedRows);
} myFireSwitch = myLastFiredPacket.addedRows.Count + myLastFiredPacket.deletedRows.Count == 0 ? false : true;
}
} }
@@ -1,12 +1,10 @@
using System;
using System.Collections; using System.Collections;
namespace BCG { namespace BCG {
public class AlphabetizeColumnsStream : ViewStream { public class AlphabetizeColumnsStream : ViewStream {
private ViewStream mySrc; private ViewStream mySrc;
private IBlueTableSchema mySchema = null; IBlueTableSchema? mySchema = null;
public AlphabetizeColumnsStream(ViewStream src) { public AlphabetizeColumnsStream(ViewStream src) {
mySrc = src; mySrc = src;
@@ -1,12 +1,12 @@
using System; using System.Linq;
namespace BCG namespace BCG
{ {
public class AppendStreamModified : ViewStream public class AppendStreamModified : ViewStream
{ {
private AppendCollection myColl = new AppendCollection (); private AppendCollection myColl = new AppendCollection ();
private ViewStream myLeftStream, myRightstream; ViewStream? myLeftStream, myRightstream;
public AppendStreamModified (ViewStream leftStream, ViewStream rightStream) public AppendStreamModified (ViewStream leftStream, ViewStream rightStream)
{ {
myLeftStream = leftStream; myLeftStream = leftStream;
@@ -15,11 +15,24 @@ namespace BCG
public AppendStreamModified (ViewStream[] vss) public AppendStreamModified (ViewStream[] vss)
{ {
switch (vss.Length) {
case 0:
break;
case 1:
mySetRealSource (vss[0]);
break;
default:
mySetRealSource(
vss[1..].Aggregate(
new AppendStreamModified(new[] { vss[0] }),
(asm, vs) => new AppendStreamModified(asm, vs)));
break;
}
if (vss.Length == 1) if (vss.Length == 1)
mySetRealSource (vss[0]); mySetRealSource (vss[0]);
else else
{ {
AppendStreamModified rv = null; AppendStreamModified? rv = null;
foreach (ViewStream s in vss) foreach (ViewStream s in vss)
rv = rv == null ? new AppendStreamModified (new ViewStream[] {s}) : new AppendStreamModified (rv, s); rv = rv == null ? new AppendStreamModified (new ViewStream[] {s}) : new AppendStreamModified (rv, s);
mySetRealSource (rv); mySetRealSource (rv);
@@ -8,11 +8,11 @@ using Bts.DataStreamCore;
using Bts.DataStreamExpressions; using Bts.DataStreamExpressions;
using Bts.PubSub; using Bts.PubSub;
using global::Spark.Core.Disposables;
using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options; using Microsoft.Extensions.Options;
using Spark.Core.Disposables;
using System; using System;
using System.Collections.Generic; using System.Collections.Generic;
using System.Linq; using System.Linq;
@@ -8,7 +8,6 @@ using Bts.PubSub.Services;
using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Hosting;
using Spark.Api.MarketData;
using Spark.Api.Services; using Spark.Api.Services;
using Spark.Core.Services; using Spark.Core.Services;
+56
View File
@@ -43,6 +43,14 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Net", "..\cs\Bts.Net\Bt
EndProject EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.IO.Compression", "..\cs\Bts.IO.Compression\Bts.IO.Compression.csproj", "{3BF059B0-51C0-44A8-8297-9B7464DC5BA3}" Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.IO.Compression", "..\cs\Bts.IO.Compression\Bts.IO.Compression.csproj", "{3BF059B0-51C0-44A8-8297-9B7464DC5BA3}"
EndProject EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Collections", "..\cs\Bts.Collections\Bts.Collections.csproj", "{ADBEBCB3-5367-431B-AFC8-F57CAF9F029D}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Core", "..\cs\Bts.Core\Bts.Core.csproj", "{4E3D817D-9A0A-40F4-A1A1-4865375F40DD}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Disposables", "..\cs\Bts.Disposables\Bts.Disposables.csproj", "{ED034E82-07F4-473D-9C3A-F5EF9C961CFD}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bts.Numerics", "..\cs\Bts.Numerics\Bts.Numerics.csproj", "{9A3E289A-3A88-4A76-AD0B-32C12EBA1B8B}"
EndProject
Global Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU Debug|Any CPU = Debug|Any CPU
@@ -293,6 +301,54 @@ Global
{3BF059B0-51C0-44A8-8297-9B7464DC5BA3}.Release|x64.Build.0 = Release|Any CPU {3BF059B0-51C0-44A8-8297-9B7464DC5BA3}.Release|x64.Build.0 = Release|Any CPU
{3BF059B0-51C0-44A8-8297-9B7464DC5BA3}.Release|x86.ActiveCfg = Release|Any CPU {3BF059B0-51C0-44A8-8297-9B7464DC5BA3}.Release|x86.ActiveCfg = Release|Any CPU
{3BF059B0-51C0-44A8-8297-9B7464DC5BA3}.Release|x86.Build.0 = Release|Any CPU {3BF059B0-51C0-44A8-8297-9B7464DC5BA3}.Release|x86.Build.0 = Release|Any CPU
{ADBEBCB3-5367-431B-AFC8-F57CAF9F029D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{ADBEBCB3-5367-431B-AFC8-F57CAF9F029D}.Debug|Any CPU.Build.0 = Debug|Any CPU
{ADBEBCB3-5367-431B-AFC8-F57CAF9F029D}.Debug|x64.ActiveCfg = Debug|Any CPU
{ADBEBCB3-5367-431B-AFC8-F57CAF9F029D}.Debug|x64.Build.0 = Debug|Any CPU
{ADBEBCB3-5367-431B-AFC8-F57CAF9F029D}.Debug|x86.ActiveCfg = Debug|Any CPU
{ADBEBCB3-5367-431B-AFC8-F57CAF9F029D}.Debug|x86.Build.0 = Debug|Any CPU
{ADBEBCB3-5367-431B-AFC8-F57CAF9F029D}.Release|Any CPU.ActiveCfg = Release|Any CPU
{ADBEBCB3-5367-431B-AFC8-F57CAF9F029D}.Release|Any CPU.Build.0 = Release|Any CPU
{ADBEBCB3-5367-431B-AFC8-F57CAF9F029D}.Release|x64.ActiveCfg = Release|Any CPU
{ADBEBCB3-5367-431B-AFC8-F57CAF9F029D}.Release|x64.Build.0 = Release|Any CPU
{ADBEBCB3-5367-431B-AFC8-F57CAF9F029D}.Release|x86.ActiveCfg = Release|Any CPU
{ADBEBCB3-5367-431B-AFC8-F57CAF9F029D}.Release|x86.Build.0 = Release|Any CPU
{4E3D817D-9A0A-40F4-A1A1-4865375F40DD}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{4E3D817D-9A0A-40F4-A1A1-4865375F40DD}.Debug|Any CPU.Build.0 = Debug|Any CPU
{4E3D817D-9A0A-40F4-A1A1-4865375F40DD}.Debug|x64.ActiveCfg = Debug|Any CPU
{4E3D817D-9A0A-40F4-A1A1-4865375F40DD}.Debug|x64.Build.0 = Debug|Any CPU
{4E3D817D-9A0A-40F4-A1A1-4865375F40DD}.Debug|x86.ActiveCfg = Debug|Any CPU
{4E3D817D-9A0A-40F4-A1A1-4865375F40DD}.Debug|x86.Build.0 = Debug|Any CPU
{4E3D817D-9A0A-40F4-A1A1-4865375F40DD}.Release|Any CPU.ActiveCfg = Release|Any CPU
{4E3D817D-9A0A-40F4-A1A1-4865375F40DD}.Release|Any CPU.Build.0 = Release|Any CPU
{4E3D817D-9A0A-40F4-A1A1-4865375F40DD}.Release|x64.ActiveCfg = Release|Any CPU
{4E3D817D-9A0A-40F4-A1A1-4865375F40DD}.Release|x64.Build.0 = Release|Any CPU
{4E3D817D-9A0A-40F4-A1A1-4865375F40DD}.Release|x86.ActiveCfg = Release|Any CPU
{4E3D817D-9A0A-40F4-A1A1-4865375F40DD}.Release|x86.Build.0 = Release|Any CPU
{ED034E82-07F4-473D-9C3A-F5EF9C961CFD}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{ED034E82-07F4-473D-9C3A-F5EF9C961CFD}.Debug|Any CPU.Build.0 = Debug|Any CPU
{ED034E82-07F4-473D-9C3A-F5EF9C961CFD}.Debug|x64.ActiveCfg = Debug|Any CPU
{ED034E82-07F4-473D-9C3A-F5EF9C961CFD}.Debug|x64.Build.0 = Debug|Any CPU
{ED034E82-07F4-473D-9C3A-F5EF9C961CFD}.Debug|x86.ActiveCfg = Debug|Any CPU
{ED034E82-07F4-473D-9C3A-F5EF9C961CFD}.Debug|x86.Build.0 = Debug|Any CPU
{ED034E82-07F4-473D-9C3A-F5EF9C961CFD}.Release|Any CPU.ActiveCfg = Release|Any CPU
{ED034E82-07F4-473D-9C3A-F5EF9C961CFD}.Release|Any CPU.Build.0 = Release|Any CPU
{ED034E82-07F4-473D-9C3A-F5EF9C961CFD}.Release|x64.ActiveCfg = Release|Any CPU
{ED034E82-07F4-473D-9C3A-F5EF9C961CFD}.Release|x64.Build.0 = Release|Any CPU
{ED034E82-07F4-473D-9C3A-F5EF9C961CFD}.Release|x86.ActiveCfg = Release|Any CPU
{ED034E82-07F4-473D-9C3A-F5EF9C961CFD}.Release|x86.Build.0 = Release|Any CPU
{9A3E289A-3A88-4A76-AD0B-32C12EBA1B8B}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{9A3E289A-3A88-4A76-AD0B-32C12EBA1B8B}.Debug|Any CPU.Build.0 = Debug|Any CPU
{9A3E289A-3A88-4A76-AD0B-32C12EBA1B8B}.Debug|x64.ActiveCfg = Debug|Any CPU
{9A3E289A-3A88-4A76-AD0B-32C12EBA1B8B}.Debug|x64.Build.0 = Debug|Any CPU
{9A3E289A-3A88-4A76-AD0B-32C12EBA1B8B}.Debug|x86.ActiveCfg = Debug|Any CPU
{9A3E289A-3A88-4A76-AD0B-32C12EBA1B8B}.Debug|x86.Build.0 = Debug|Any CPU
{9A3E289A-3A88-4A76-AD0B-32C12EBA1B8B}.Release|Any CPU.ActiveCfg = Release|Any CPU
{9A3E289A-3A88-4A76-AD0B-32C12EBA1B8B}.Release|Any CPU.Build.0 = Release|Any CPU
{9A3E289A-3A88-4A76-AD0B-32C12EBA1B8B}.Release|x64.ActiveCfg = Release|Any CPU
{9A3E289A-3A88-4A76-AD0B-32C12EBA1B8B}.Release|x64.Build.0 = Release|Any CPU
{9A3E289A-3A88-4A76-AD0B-32C12EBA1B8B}.Release|x86.ActiveCfg = Release|Any CPU
{9A3E289A-3A88-4A76-AD0B-32C12EBA1B8B}.Release|x86.Build.0 = Release|Any CPU
EndGlobalSection EndGlobalSection
GlobalSection(SolutionProperties) = preSolution GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE HideSolutionNode = FALSE
@@ -5,7 +5,7 @@ using BCG.Utilities.General;
using Bts.DataStreamCore; using Bts.DataStreamCore;
using global::Spark.Core.Collections; using Spark.Core.Collections;
public class TestDriverStream : ViewStream, IWorker { public class TestDriverStream : ViewStream, IWorker {
readonly IBlueTableSchema schema; readonly IBlueTableSchema schema;
@@ -22,7 +22,6 @@ namespace BCG {
/// with PubSub, as some clients are still using ManagedSubcribers) /// with PubSub, as some clients are still using ManagedSubcribers)
/// </summary> /// </summary>
public class ManagedSubscriber : IManagedSubscriber, IWorker, IDisposable { public class ManagedSubscriber : IManagedSubscriber, IWorker, IDisposable {
private static ManagedSubscriber? _self;
private ISubscriberCore _subcore; private ISubscriberCore _subcore;
// private IDictionary myTypedDispatchTable = new Hashtable(), // private IDictionary myTypedDispatchTable = new Hashtable(),
// myRawDispatchTable = new Hashtable(); // myRawDispatchTable = new Hashtable();
@@ -34,7 +33,6 @@ namespace BCG {
ManagedSubscriber(ISubscriberCore subsCore) { _subcore = subsCore; } ManagedSubscriber(ISubscriberCore subsCore) { _subcore = subsCore; }
public void doWork() { public void doWork() {
var msg_map = new Dictionary<string, List<bts.PubSubServerMessage>>();
var updates = _subcore.get_updates(TimeSpan.Zero); var updates = _subcore.get_updates(TimeSpan.Zero);
if (updates.Count() == 0) if (updates.Count() == 0)
return; return;
@@ -45,11 +45,7 @@ namespace BCG
get { return myConnInfo; } get { return myConnInfo; }
} }
override public string ToString() public override string ToString() =>
{ $"type: {myMsgType} conn: {myConnInfo} msg: {myMsg?.Length ?? 0}";
Func<byte[], int> len = (byte[] m) => { return m == null ? 0 : m.Length; };
return $"type: {myMsgType} conn: {myConnInfo} msg: {len(myMsg)}";
}
} }
} }
@@ -46,7 +46,7 @@ namespace bts {
System.Threading.Thread.Sleep(TimeSpan.FromSeconds(5)); System.Threading.Thread.Sleep(TimeSpan.FromSeconds(5));
_accept_socket.BeginAccept(new AsyncCallback(_on_accept), new object[] { }); _accept_socket.BeginAccept(new AsyncCallback(_on_accept), new object[] { });
if (ex is OutOfMemoryException) if (ex is OutOfMemoryException)
throw ex; throw;
return; return;
} }
new_sock.LingerState = new LingerOption(false, 0); new_sock.LingerState = new LingerOption(false, 0);
@@ -1,9 +1,8 @@
using bts;
using System; using System;
using System.IO; using System.IO;
using System.Collections; using System.Text;
using bts;
using System.Threading;
using System.Reflection;
namespace BCG { namespace BCG {
/// <summary> /// <summary>
@@ -69,7 +68,8 @@ namespace BCG {
public static byte[] getBytesFromStream(params Stream[] streams) { public static byte[] getBytesFromStream(params Stream[] streams) {
long totalSize = 0; long totalSize = 0;
BinaryWriter bw = new BinaryWriter(new MemoryStream()); using var ms = new MemoryStream();
using var bw = new BinaryWriter(ms, Encoding.UTF8, true);
foreach (Stream s in streams) foreach (Stream s in streams)
totalSize += s.Length - s.Position; totalSize += s.Length - s.Position;
bw.Write(Convert.ToInt32(totalSize)); bw.Write(Convert.ToInt32(totalSize));
@@ -78,7 +78,8 @@ namespace BCG {
int lengthToReadForThisStream = Convert.ToInt32(s.Length - s.Position); int lengthToReadForThisStream = Convert.ToInt32(s.Length - s.Position);
bw.Write(br.ReadBytes(lengthToReadForThisStream)); bw.Write(br.ReadBytes(lengthToReadForThisStream));
} }
return (bw.BaseStream as MemoryStream).ToArray(); bw.Flush();
return ms.ToArray();
} }
public static byte[] read_bytes(Stream str) { public static byte[] read_bytes(Stream str) {
+1 -1
View File
@@ -92,7 +92,7 @@ namespace BCG {
public void publish(PubSubClientMessage msg) { public void publish(PubSubClientMessage msg) {
Log.assert(msg.msg_type == PubSubMessageType.PUBLISH); Log.assert(msg.msg_type == PubSubMessageType.PUBLISH);
Log.assert(!String.IsNullOrEmpty(msg.pub_msg.tag_type)); Log.assert(!string.IsNullOrEmpty(msg.pub_msg?.tag_type));
bts.Log.Gen.info($"PublisherCore.publish msg: {msg}"); bts.Log.Gen.info($"PublisherCore.publish msg: {msg}");
_main_queue.enqueue(msg); _main_queue.enqueue(msg);
} }
@@ -143,7 +143,7 @@ namespace BCG {
public void doWork() { public void doWork() {
if (_connection_cb != null) { if (_connection_cb != null) {
var conns = _conn_callback_queue.dequeueAll(); var conns = _conn_callback_queue.dequeueAll();
foreach (var ci in conns) { foreach (var ci in conns.ExceptNull()) {
_connection_cb(ci.client_url, ci.status); _connection_cb(ci.client_url, ci.status);
} }
} }
@@ -175,7 +175,7 @@ namespace BCG {
case RemoteMessageType.Data: case RemoteMessageType.Data:
// if callback is registered fire it right away, else enqueue it // if callback is registered fire it right away, else enqueue it
if (_data_cb != null) if (_data_cb != null)
_data_cb(sender_rci.ToString(), message.message); _data_cb(sender_rci.ToString(), message.message ?? Array.Empty<byte>());
else else
_received_data_queue.enqueue(new Pair<string, byte[]>(sender_rci.ToString(), message.message)); _received_data_queue.enqueue(new Pair<string, byte[]>(sender_rci.ToString(), message.message));
break; break;
@@ -258,7 +258,7 @@ namespace BCG {
// (which gets called in another thread) // (which gets called in another thread)
private void _handle_new_connections() { private void _handle_new_connections() {
var new_conns = _new_connections.dequeueAll(); var new_conns = _new_connections.dequeueAll();
foreach (ITCPConnection conn in new_conns) { foreach (var conn in new_conns.ExceptNull()) {
var peer_rci = conn.get_peer_address(); var peer_rci = conn.get_peer_address();
bts.Log.Gen.info($"ManagedTCPMessageHub._handle_new_connections conn: {conn}"); bts.Log.Gen.info($"ManagedTCPMessageHub._handle_new_connections conn: {conn}");
///to stop it from being garbabge collected immediatly ///to stop it from being garbabge collected immediatly
@@ -16,8 +16,8 @@ public interface IPubSubState {
ICollection<string> all_tags { get; } ICollection<string> all_tags { get; }
byte[] get_content(string tag); byte[]? get_content(string tag);
string getTagType(string tag); string? getTagType(string tag);
void get_all_connections( void get_all_connections(
string tag, string tag,
out Dictionary<string, string> subscribers, out Dictionary<string, string> subscribers,
@@ -188,11 +188,11 @@ namespace BCG {
_state.handle_sub(sub_msg); // state keeps the list of all subscribers _state.handle_sub(sub_msg); // state keeps the list of all subscribers
///to all these new subscriber s, send the existing data ///to all these new subscriber s, send the existing data
foreach (string tag in sub_msg.tags.empty_if_null()) { foreach (string tag in sub_msg.tags.empty_if_null()) {
byte[] data = _state.get_content(tag); byte[]? data = _state.get_content(tag);
//ie data has been published once //ie data has been published once
if (data != null) { if (data != null) {
bts.Log.Gen.info($"ManagedTCPMessgeHub._handle_sub get_content tag: {tag} data len: {data.Length}"); bts.Log.Gen.info($"ManagedTCPMessgeHub._handle_sub get_content tag: {tag} data len: {data.Length}");
string tag_type_ = _state.getTagType(tag); string? tag_type_ = _state.getTagType(tag);
var msg_to_send = new PubSubServerMessage() { var msg_to_send = new PubSubServerMessage() {
tag = tag, tag = tag,
tag_type = tag_type_, tag_type = tag_type_,
@@ -148,7 +148,7 @@ namespace BCG {
return new string[] { }; return new string[] { };
} }
public byte[] get_content(string tag) { public byte[]? get_content(string tag) {
if (!_tag_to_data_map.ContainsKey(tag)) if (!_tag_to_data_map.ContainsKey(tag))
return null; return null;
var rv = _tag_to_data_map[tag].get_current_content(); var rv = _tag_to_data_map[tag].get_current_content();
@@ -157,7 +157,7 @@ namespace BCG {
return rv; return rv;
} }
public string getTagType(string tag) { public string? getTagType(string tag) {
if (!_tag_to_data_map.ContainsKey(tag)) if (!_tag_to_data_map.ContainsKey(tag))
return null; return null;
else else
@@ -20,7 +20,7 @@ namespace bts {
public class SingleTagData { public class SingleTagData {
private readonly IPubSubServerTagHandler? _tag_handler; private readonly IPubSubServerTagHandler? _tag_handler;
public SingleTagData(string tag, string tag_type, IPubSubServerTagHandler tag_handler) { public SingleTagData(string tag, string tag_type, IPubSubServerTagHandler? tag_handler) {
bts.Log.Gen.info($"SingleTagData constructed for tag: {tag} tag_type: {tag_type}"); bts.Log.Gen.info($"SingleTagData constructed for tag: {tag} tag_type: {tag_type}");
this.tag = tag; this.tag = tag;
this.tag_type = tag_type; this.tag_type = tag_type;
@@ -46,7 +46,6 @@ namespace BCG {
} }
public void onConnection(ITCPConnection connection) { public void onConnection(ITCPConnection connection) {
string ip; int port;
var peer_rci = connection.get_peer_address(); var peer_rci = connection.get_peer_address();
bts.Log.Gen.info($"SimpleTCPClient.onConnection connected to server: {peer_rci}, starting task to send data in a loop"); bts.Log.Gen.info($"SimpleTCPClient.onConnection connected to server: {peer_rci}, starting task to send data in a loop");
if (_listener != null) { if (_listener != null) {
@@ -5,9 +5,10 @@ using BCG;
using bts.api; using bts.api;
using bts.md; using bts.md;
using Bts.DataStream;
using Bts.DataStreamCore; using Bts.DataStreamCore;
using global::Spark.Api.MarketData; using Spark.Api.MarketData;
public class BookSchemaBuilder { public class BookSchemaBuilder {
public int NumLevels { get; set; } = 5; public int NumLevels { get; set; } = 5;
@@ -2,7 +2,7 @@ namespace Bts.SharedStreams;
using bts.api; using bts.api;
using global::Spark.Api.Orders; using Spark.Api.Orders;
public static class WellKnownColumns { public static class WellKnownColumns {
public static (string Name, Type Type) AskNotional => (WellKnownColumnNames.AskNotional, typeof(double)); public static (string Name, Type Type) AskNotional => (WellKnownColumnNames.AskNotional, typeof(double));
@@ -3,6 +3,7 @@ using bts.config;
using System; using System;
using System.Collections; using System.Collections;
using System.Collections.Generic; using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
namespace BCG { namespace BCG {
@@ -21,7 +22,9 @@ namespace BCG {
} }
public class RenkoCalculator { public class RenkoCalculator {
public readonly double brick_size; public readonly double brick_size;
public RenkoBrick current_brick = null; public RenkoBrick? current_brick;
[MemberNotNull(nameof(current_brick))]
private bool _add_price(double price) { private bool _add_price(double price) {
if (current_brick == null) { if (current_brick == null) {
current_brick = new RenkoBrick { current_brick = new RenkoBrick {
@@ -309,4 +312,4 @@ namespace BCG {
return wma_values; return wma_values;
} }
} }
} }
@@ -10,7 +10,7 @@
[DataStreamUserRespresentation("nl")] [DataStreamUserRespresentation("nl")]
public class BlueOrbNLStream : ViewStream { public class BlueOrbNLStream : ViewStream {
private readonly string _nl_prompt; private readonly string _nl_prompt;
private ViewStream _nl_to_dsl_stream, _dsl_stream; private ViewStream? _nl_to_dsl_stream, _dsl_stream;
private IDictionary<string, object> _dp_metadata = new Dictionary<string, object>(); private IDictionary<string, object> _dp_metadata = new Dictionary<string, object>();
public BlueOrbNLStream(string nl_prompt) { public BlueOrbNLStream(string nl_prompt) {
_nl_prompt = nl_prompt; _nl_prompt = nl_prompt;
@@ -85,7 +85,8 @@ namespace bts.BlueOrb {
.GetProperty("choices")[0] .GetProperty("choices")[0]
.GetProperty("message") .GetProperty("message")
.GetProperty("content") .GetProperty("content")
.GetString(); .GetString()
?? string.Empty;
//return gpt_response; //return gpt_response;
DateTime t1 = DateTime.Now; DateTime t1 = DateTime.Now;
@@ -4,6 +4,8 @@ using BCG.Utilities.General;
using Bts.PubSub; using Bts.PubSub;
using Bts.SharedStreams.Tables; using Bts.SharedStreams.Tables;
using Spark.Core.Diagnostics;
namespace bts { namespace bts {
// TODO: Use dependency injection to instantiate this stream, and others like it that depend on // TODO: Use dependency injection to instantiate this stream, and others like it that depend on
// ISubscriberCore(Factory). Until then, hide it from DSQA. // ISubscriberCore(Factory). Until then, hide it from DSQA.
@@ -40,7 +42,7 @@ namespace bts {
foreach (var u in updates) { foreach (var u in updates) {
if (u.tag != "System_AllStatistics") if (u.tag != "System_AllStatistics")
continue; continue;
var tag_infos = TagStatsInfo.deserialize(u.data); var tag_infos = TagStatsInfo.deserialize(ExceptionAssert.NotNull(u.data));
foreach (var ti in tag_infos) { foreach (var ti in tag_infos) {
_tag_map[ti.tag] = ti; _tag_map[ti.tag] = ti;
} }
+6 -5
View File
@@ -2,14 +2,14 @@ ARG IMAGE="sparkd"
ARG CONFIG_DIR="src/app_sparkd/configurations/dev" ARG CONFIG_DIR="src/app_sparkd/configurations/dev"
ARG SOURCE_DIR="install/sparkd-release-gcc" ARG SOURCE_DIR="install/sparkd-release-gcc"
ARG PYTHON_REQUIREMENTS="requirements.txt" ARG PYTHON_REQUIREMENTS="requirements.txt"
ARG SPARKD_ENTRYPOINT="src/app_sparkd/sparkd-entrypoint.sh" ARG ENTRYPOINT_SCRIPT="src/app_sparkd/sparkd-entrypoint.sh"
# sparkd_base # sparkd_base
FROM bluetradingsystems/linux_dotnet8 AS sparkd_base FROM bluetradingsystems/linux_dotnet8 AS sparkd_base
ARG CONFIG_DIR ARG CONFIG_DIR
ARG SOURCE_DIR ARG SOURCE_DIR
ARG PYTHON_REQUIREMENTS ARG PYTHON_REQUIREMENTS
ARG SPARKD_ENTRYPOINT ARG ENTRYPOINT_SCRIPT
USER "root" USER "root"
RUN apt-get update && apt-get install -y liblttng-ust-dev RUN apt-get update && apt-get install -y liblttng-ust-dev
@@ -23,7 +23,7 @@ RUN pip3 install -r "/home/bts/bin/requirements.txt"
USER "bts" USER "bts"
COPY --chown=bts:bts "${SOURCE_DIR}" "." COPY --chown=bts:bts "${SOURCE_DIR}" "."
COPY --chown=bts:bts "${CONFIG_DIR}" "/home/bts/.local/share/bts/sparkd/conf" COPY --chown=bts:bts "${CONFIG_DIR}" "/home/bts/.local/share/bts/sparkd/conf"
COPY --chown=bts:bts "${SPARKD_ENTRYPOINT}" "/home/bts/bin/sparkd-entrypoint.sh" COPY --chown=bts:bts "${ENTRYPOINT_SCRIPT}" "/home/bts/bin/sparkd-entrypoint.sh"
WORKDIR "/home/bts/bin" WORKDIR "/home/bts/bin"
# sparkd # sparkd
@@ -38,7 +38,7 @@ COPY --chown=bts:bts "${MD_RECORDING_DIR}" "/home/bts/.local/share/bts/sparkd/da
FROM bluetradingsystems/linux_dotnet8 AS sparkd_symbols FROM bluetradingsystems/linux_dotnet8 AS sparkd_symbols
ARG SOURCE_DIR ARG SOURCE_DIR
ARG PYTHON_REQUIREMENTS="requirements.txt" ARG PYTHON_REQUIREMENTS="requirements.txt"
ARG SPARKD_ENTRYPOINT ARG ENTRYPOINT_SCRIPT
USER "root" USER "root"
RUN apt-get update && apt-get install -y liblttng-ust-dev RUN apt-get update && apt-get install -y liblttng-ust-dev
@@ -51,8 +51,9 @@ RUN pip3 install -r "/home/bts/bin/requirements.txt"
USER "bts" USER "bts"
COPY --chown=bts:bts "${SOURCE_DIR}" "." COPY --chown=bts:bts "${SOURCE_DIR}" "."
COPY --chown=bts:bts "${SPARKD_ENTRYPOINT}" "/home/bts/bin/sparkd-entrypoint.sh" COPY --chown=bts:bts "${ENTRYPOINT_SCRIPT}" "/home/bts/bin/sparkd-entrypoint.sh"
WORKDIR "/home/bts/bin" WORKDIR "/home/bts/bin"
# selected image # selected image
FROM ${IMAGE} FROM ${IMAGE}
ENTRYPOINT "./sparkd-entrypoint.sh"
@@ -0,0 +1,4 @@
SPARKD__CONFIG_DIR=src/app_sparkd/configurations/cme_replay
SPARKD__PYTHON_REQUIREMENTS=src/app_sparkd/configurations/cme_replay/requirements.txt
SPARKD__ENTRYPOINT=src/app_sparkd/sparkd-entrypoint.sh
TZ=America/Chicago
@@ -0,0 +1,38 @@
PublisherOptions {
value {
network_interface: "0.0.0.0"
heartbeat_ms: 5000
heartbeat_boot_seconds: 60
skip_options: true
hot_initial: 1024
hot_growth: 1024
cold_initial: 8192
cold_growth: 2048
channel_prefix: ""
disable_heartbeat: true
}
}
MarketDataConfig {
value {
# glbx{
# use_databento: true
# databento_settings {
# read_from_file: true
# key: "db-DgxpuTXneNgVvd7sJCAEKYDhyCApW"
# bbo_file_name: "glbx-mdp3-20251103.mbp-1.dbn.zst"
# contracts_file_name: "glbx-mdp3-20251103.definition.dbn.zst"
# }
# }
# recording{
# venue: VENUE_GLBX
# num_of_seconds_to_run_file_writer: 240
# num_of_seconds_to_run_db_writer : 0
# n_book_levels: 10
# batch_size: 100 #for the db writer
# }
replay {
venue: VENUE_GLBX
}
}
}
@@ -0,0 +1,48 @@
Symbols {
value {
venues {
name: "VENUE_GLBX"
future: "ES"
# option: "E1A"
# option: "E1B"
# option: "E1C"
# option: "E1D"
# option: "E2A"
# option: "E2B"
# option: "E2C"
# option: "E2D"
# option: "E3A"
# option: "E3B"
# option: "E3C"
# option: "E3D"
# option: "E4A"
# option: "E4B"
# option: "E4C"
# option: "E4D"
# option: "E5A"
# option: "E5B"
# option: "E5C"
# option: "E5D"
# option: "EW1"
# option: "EW2"
# option: "EW3"
# option: "EW4"
# option: "ESH9"
# option: "ESZ6"
# option: "ESH0"
# option: "ESM6"
# option: "ESZ7"
# option: "ESH6"
# option: "ESM7"
# option: "ESZ8"
# option: "ESH7"
# option: "ESH8"
# option: "ESU6"
# option: "ESZ5"
# option: "ESZ9"
# option: "E1D"
# option: "ESZ5"
# option: "ES"
}
}
}
@@ -1,5 +0,0 @@
SPARKD__CONFIG_DIR=src/app_sparkd/configurations/ophir_sim_live_wb
SPARKD__DATA_DIR=/home/wedbushprod/8000/sparkdata/data
SPARKD__IMAGE=sparkd_symbols
SPARKD__PYTHON_REQUIREMENTS=src/app_sparkd/configurations/ophir_sim_live_wb/requirements.txt
SPARKD__RUNTIME_CONF_DIR=/home/wedbushprod/8000/sparkdata/conf
@@ -1,5 +0,0 @@
numpy==2.2.6
pybit==5.11.0
requests==2.31.0
websocket-client==1.8.0
websockets==15.0.1
@@ -1,116 +0,0 @@
JetFIX {
value {
connection {
name: "glbx_1"
venue: VENUE_GLBX
max_order_msgs_per_window: 300
window_duration_msecs: 1000
}
}
}
PublisherOptions {
value {
network_interface_a: 'enp193s0f0np0'
network_interface_b: 'enp193s0f0np0'
#these get overriden if not entered here
heartbeat_ms: 5000
heartbeat_boot_seconds: 60
skip_options: false
hot_initial: 1024
hot_growth: 1024
cold_initial: 8192
cold_growth: 2048
channel_prefix: ""
disable_heartbeat: false
}
}
CMEOptions {
value {
feed_engine {
name: "mdp3feed"
spin_time: 0
channels_per_thread: 3
data_wait: 10
udp_socket_buffer_size: 268435456
}
channel {
id: "*"
feed_engine: "mdp3feed"
use_tcp_replay: false
recovery_queue_size: 640000
lost_packet_wait_timeout_ms: 10
onixs_log_mode: CME_ONIXS_LOG_MODE_UNSPECIFIED:q
#max_gap_recovery_cache_size: 4
use_natural_refresh: false
join_snapshot_on_gap: false
}
}
}
Logging {
value {
initial_queue_size: 120001
max_queue_growth: 4
console {
default_level: "Info"
subsystem {
name: "md"
level: "Info"
}
subsystem {
name: "OrderMan"
level: "Info"
}
}
}
}
MarketDataConfig {
value {
glbx {
unique_check_value: 15
use_databento: false
}
recording{
venue: VENUE_GLBX
num_of_seconds_to_run_file_writer: 300
num_of_seconds_to_run_db_writer : 0
n_book_levels: 10
batch_size: 100 #for the db writer
enable_book_recording: true
enable_trade_recording: true
enable_bbo_recording: false
enable_tbbo_recording: false
enable_misc_recording: true
}
}
}
# JetFIX {
# value {
# #connection: @JetFIXSimConnection.sim
# connection {
# name: "mlx_004_54"
# venue: VENUE_GLBX
# socket_connect_host_ip: "69.50.112.191"
# socket_connect_port: 42876
# sender_compid: "MLX004"
# target_compid: "CME"
# target_subid: "54"
# cme_market_segment_id: 54
# start_time: "00:10:00"
# end_time: "23:55:00"
# start_day: "sunday"
# end_day: "friday"
# seconds_between_hbs: 30
# sender_subid: "BTS"
# use_ilink3: true
# # raw_data: "S4PMA"
# # auto_fix_seq_num: true
# }
# }
# }
@@ -1,61 +0,0 @@
SymbolLists {
value {
list {
key: "GlbxFutures"
symbol: "ES"
}
list {
key: "GlbxOptions"
symbol: "E1A"
symbol: "E1B"
symbol: "E1C"
symbol: "E1D"
symbol: "E2A"
symbol: "E2B"
symbol: "E2C"
symbol: "E2D"
symbol: "E3A"
symbol: "E3B"
symbol: "E3C"
symbol: "E3D"
symbol: "E4A"
symbol: "E4B"
symbol: "E4C"
symbol: "E4D"
symbol: "E5A"
symbol: "E5B"
symbol: "E5C"
symbol: "E5D"
symbol: "EW1"
symbol: "EW2"
symbol: "EW3"
symbol: "EW4"
symbol: "ESH9"
symbol: "ESZ6"
symbol: "ESH0"
symbol: "ESM6"
symbol: "ESZ7"
symbol: "ESH6"
symbol: "ESM7"
symbol: "ESZ8"
symbol: "ESH7"
symbol: "ESH8"
symbol: "ESU6"
symbol: "ESZ5"
symbol: "ESZ9"
symbol: "E1D"
symbol: "ESZ5"
symbol: "ES"
}
}
}
Symbols {
value {
venues {
name: "VENUE_GLBX"
future: @SymbolLists.list[GlbxFutures].symbol
option: @SymbolLists.list[GlbxOptions].symbol
}
}
}
@@ -1,4 +0,0 @@
SPARKD__CONFIG_DIR=src/app_sparkd/configurations/ophir_sim_replay
SPARKD__PYTHON_REQUIREMENTS=src/app_sparkd/configurations/ophir_sim_replay/requirements.txt
TZ=America/Chicago
SPARKD__MD_RECORDING_DIR=/home/wedbushprod/8000/md_recording
@@ -1,33 +0,0 @@
JetFIX {
value {
connection {
name: "glbx_1"
venue: VENUE_GLBX
max_order_msgs_per_window: 300
window_duration_msecs: 1000
}
}
}
PublisherOptions {
value {
network_interface: "0.0.0.0"
heartbeat_ms: 5000
heartbeat_boot_seconds: 60
skip_options: true
hot_initial: 1024
hot_growth: 1024
cold_initial: 8192
cold_growth: 2048
channel_prefix: ""
disable_heartbeat: true
}
}
MarketDataConfig {
value {
replay {
venue: VENUE_GLBX
}
}
}
@@ -1,46 +0,0 @@
SymbolLists {
value {
list {
key: "GlbxFutures"
symbol: "ES"
}
list {
key: "GlbxOptions"
symbol: "E1A"
symbol: "E1B"
symbol: "E1C"
symbol: "E1D"
symbol: "E2A"
symbol: "E2B"
symbol: "E2C"
symbol: "E2D"
symbol: "E3A"
symbol: "E3B"
symbol: "E3C"
symbol: "E3D"
symbol: "E4A"
symbol: "E4B"
symbol: "E4C"
symbol: "E4D"
symbol: "E5A"
symbol: "E5B"
symbol: "E5C"
symbol: "E5D"
symbol: "ES"
symbol: "EW1"
symbol: "EW2"
symbol: "EW3"
symbol: "EW4"
}
}
}
Symbols {
value {
venues {
name: "VENUE_GLBX"
future: @SymbolLists.list[GlbxFutures].symbol
option: @SymbolLists.list[GlbxOptions].symbol
}
}
}
@@ -1,9 +0,0 @@
SPARKD__CONFIG_DIR=src/app_sparkd/configurations/ophir_sim_replay_wb
SPARKD__DATA_DIR=/home/wedbushprod/9000/sparkdata/data
SPARKD__IMAGE=sparkd_symbols
SPARKD__PYTHON_REQUIREMENTS=src/app_sparkd/configurations/ophir_sim_replay_wb/requirements.txt
SPARKD__RUNTIME_CONF_DIR=/home/wedbushprod/9000/sparkdata/conf
SPARKD__MD_RECORDING_DIR=/home/wedbushprod/md_recording
SPARKD__CONFIG_PORT=9000
SPARKD__QUICKBUS_PORT=9001
SPARKD__SIM_FIX_PORT=9002
@@ -1,5 +0,0 @@
numpy==2.2.6
pybit==5.11.0
requests==2.31.0
websocket-client==1.8.0
websockets==15.0.1
@@ -1,46 +0,0 @@
JetFIX {
value {
connection {
name: "glbx_1"
venue: VENUE_GLBX
max_order_msgs_per_window: 300
window_duration_msecs: 1000
}
}
}
Ports {
value {
config_port: 9000
qbus_port: 9001
glbx_sim_fix_port: 9002
ice_sim_fix_port: 9003
}
}
Logging {
value {
initial_queue_size: 120001
max_queue_growth: 4
console {
default_level: "Info"
subsystem {
name: "md"
level: "Info"
}
subsystem {
name: "OrderMan"
level: "Info"
}
}
}
}
MarketDataConfig {
value {
replay {
venue: VENUE_GLBX
}
}
}
@@ -1,61 +0,0 @@
SymbolLists {
value {
list {
key: "GlbxFutures"
symbol: "ES"
}
list {
key: "GlbxOptions"
symbol: "E1A"
symbol: "E1B"
symbol: "E1C"
symbol: "E1D"
symbol: "E2A"
symbol: "E2B"
symbol: "E2C"
symbol: "E2D"
symbol: "E3A"
symbol: "E3B"
symbol: "E3C"
symbol: "E3D"
symbol: "E4A"
symbol: "E4B"
symbol: "E4C"
symbol: "E4D"
symbol: "E5A"
symbol: "E5B"
symbol: "E5C"
symbol: "E5D"
symbol: "EW1"
symbol: "EW2"
symbol: "EW3"
symbol: "EW4"
symbol: "ESH9"
symbol: "ESZ6"
symbol: "ESH0"
symbol: "ESM6"
symbol: "ESZ7"
symbol: "ESH6"
symbol: "ESM7"
symbol: "ESZ8"
symbol: "ESH7"
symbol: "ESH8"
symbol: "ESU6"
symbol: "ESZ5"
symbol: "ESZ9"
symbol: "E1D"
symbol: "ESZ5"
symbol: "ES"
}
}
}
Symbols {
value {
venues {
name: "VENUE_GLBX"
future: @SymbolLists.list[GlbxFutures].symbol
option: @SymbolLists.list[GlbxOptions].symbol
}
}
}
+4 -14
View File
@@ -4,12 +4,9 @@ init_timezone() {
tz="${1:-$TZ}" tz="${1:-$TZ}"
zoneinfo_dir=/usr/share/zoneinfo zoneinfo_dir=/usr/share/zoneinfo
if [ -f "$zoneinfo_dir/$tz" -a "$(readlink -f /etc/localtime)" != "$zoneinfo_dir/$tz" ]; then if [ -f "$zoneinfo_dir/$tz" -a "$(readlink -f /etc/localtime)" != "$zoneinfo_dir/$tz" ]; then
echo "Setting locales and timezone to $tz" echo "Initializing locales and timezone"
if [ -e /etc/localtime ]; then ln -snf "$zoneinfo_dir/$tz" /etc/localtime
sudo -n rm -f /etc/localtime sudo echo $tz > /etc/timezone
fi
sudo -n ln -snf "$zoneinfo_dir/$tz" /etc/localtime
echo $tz | sudo -n tee /etc/timezone > /dev/null
fi fi
} }
@@ -39,17 +36,10 @@ roll_log() {
fi fi
} }
# Acquire sudo permission once for the script (needed for a few calls)
echo bts | sudo -S -v
data_root="${HOME:-/home/bts}/.local/share/bts/sparkd" data_root="${HOME:-/home/bts}/.local/share/bts/sparkd"
sudo -n chown -R bts:bts "$data_root" echo bts | sudo -S chown -R bts:bts "$data_root"
load_env "$data_root/conf/.env.local" load_env "$data_root/conf/.env.local"
init_timezone "$TZ" init_timezone "$TZ"
roll_log "$data_root/data/logging.sparkd/log" roll_log "$data_root/data/logging.sparkd/log"
symlink "$data_root/conf/plugins" /home/bts/bin/plugins symlink "$data_root/conf/plugins" /home/bts/bin/plugins
if [ -e "/md_recording" -a ! -e "/home/bts/.local/share/bts/sparkd/data/md_recording" ]; then
symlink "/md_recording" /home/bts/.local/share/bts/sparkd/data/md_recording
fi
bash -c "${@:-./sparkd run}" bash -c "${@:-./sparkd run}"
+1 -1
View File
@@ -10,7 +10,7 @@
#include "spark/md/MBOServiceServer.h" #include "spark/md/MBOServiceServer.h"
#include "spark/md/MDClient.h" #include "spark/md/MDClient.h"
#include "spark/md/SyntheticSecurityService.h" #include "spark/md/SyntheticSecurityService.h"
#include "spark/md/relay/QueryServer.h" #include "spark/md/relay/query_server.h"
#include "spark/md/service/MarketDataServiceServer.h" #include "spark/md/service/MarketDataServiceServer.h"
#include "spark/metrics/metrics.h" #include "spark/metrics/metrics.h"
#include "spark/metrics/system_metrics.h" #include "spark/metrics/system_metrics.h"
+1 -1
View File
@@ -6,7 +6,7 @@
#include "spark/config/protos/quickbus_cfg.pb.h" #include "spark/config/protos/quickbus_cfg.pb.h"
#include "spark/md/MDClient.h" #include "spark/md/MDClient.h"
#include "spark/md/SyntheticSecurityService.h" #include "spark/md/SyntheticSecurityService.h"
#include "spark/md/relay/QueryServer.h" #include "spark/md/relay/query_server.h"
#include "spark/metrics/metrics.h" #include "spark/metrics/metrics.h"
#include "spark/metrics/system_metrics.h" #include "spark/metrics/system_metrics.h"
#include "spark/quickbus/core/Quickbus.h" #include "spark/quickbus/core/Quickbus.h"
@@ -21,10 +21,11 @@ using bts::domain::SecurityMarketDataStatistics;
using bts::io::JsonReader; using bts::io::JsonReader;
using std::map; using std::map;
using std::string; using std::string;
using std::string_view;
using std::vector; using std::vector;
static bool parse( static bool parse(
const map<string, string> &name_values, const map<string_view, string> &name_values,
vector<BinanceMiniTickerUpdate> &ticker_upds) { vector<BinanceMiniTickerUpdate> &ticker_upds) {
auto it = name_values.find(CCAPI_HTTP_BODY); auto it = name_values.find(CCAPI_HTTP_BODY);
if (it == name_values.end()) { if (it == name_values.end()) {
@@ -21,10 +21,11 @@ using bts::domain::SecurityMarketDataStatistics;
using bts::io::JsonReader; using bts::io::JsonReader;
using std::map; using std::map;
using std::string; using std::string;
using std::string_view;
using std::vector; using std::vector;
static bool parse( static bool parse(
const map<string, string> &name_values, const map<string_view, string> &name_values,
vector<BinancePriceStatistics> &result) { vector<BinancePriceStatistics> &result) {
CCApiReader ccapi_reader{&name_values}; CCApiReader ccapi_reader{&name_values};
nlohmann::json js; nlohmann::json js;
@@ -180,7 +180,7 @@ struct BinanceSecurityRequestor::PImpl {
for (const auto *msg : msgs) { for (const auto *msg : msgs) {
bts_assert_msgf( bts_assert_msgf(
msg->getType() == ccapi::Message::Type::GENERIC_PUBLIC_REQUEST, msg->getType() == ccapi::Message::Type::GENERIC_PUBLIC_REQUEST,
"Received msg w/ unexpected type:\n{}", msg->toStringPretty()); "Received msg w/ unexpected type:\n{}", msg->toPrettyString());
parse_secs(*msg, types, added_secs); parse_secs(*msg, types, added_secs);
} }
auto added_sec_descs = get_sec_descs(added_secs); auto added_sec_descs = get_sec_descs(added_secs);
@@ -214,7 +214,7 @@ struct BinanceSecurityRequestor::PImpl {
_FATAL_MSGF( _FATAL_MSGF(
"on_exchange_info. {} event was unexpected:\n{}", "on_exchange_info. {} event was unexpected:\n{}",
ccapi::Event::typeToString(event.getType()), ccapi::Event::typeToString(event.getType()),
event.toStringPretty()); event.toPrettyString());
break; break;
} }
return true; return true;
@@ -21,10 +21,11 @@ using std::map;
using std::pair; using std::pair;
using std::size_t; using std::size_t;
using std::string; using std::string;
using std::string_view;
using std::vector; using std::vector;
static bool parse( static bool parse(
const map<string, string> &name_values, const map<string_view, string> &name_values,
vector<BinanceCoinFuturesPriceStatistics> &result) { vector<BinanceCoinFuturesPriceStatistics> &result) {
CCApiReader ccapi_reader{&name_values}; CCApiReader ccapi_reader{&name_values};
nlohmann::json js; nlohmann::json js;
@@ -60,7 +60,7 @@ void BinanceCoinFuturesPriceStatisticsAdaptor::on_messages(
const auto n_total = n_hits + n_misses; const auto n_total = n_hits + n_misses;
_ERROR( _ERROR(
"on_messages. Parse failed on {}/{} elements:\n{}", n_misses, "on_messages. Parse failed on {}/{} elements:\n{}", n_misses,
n_total, event.toStringPretty()); n_total, event.toPrettyString());
continue; continue;
} }
for (const auto &bsr : binance_stats_recs) { for (const auto &bsr : binance_stats_recs) {
@@ -177,7 +177,7 @@ struct BinanceCoinFuturesSecurityRequestor::PImpl {
for (const auto *msg : msgs) { for (const auto *msg : msgs) {
bts_assert_msgf( bts_assert_msgf(
msg->getType() == ccapi::Message::Type::GENERIC_PUBLIC_REQUEST, msg->getType() == ccapi::Message::Type::GENERIC_PUBLIC_REQUEST,
"Received msg w/ unexpected type:\n{}", msg->toStringPretty()); "Received msg w/ unexpected type:\n{}", msg->toPrettyString());
parse_secs(*msg, added_secs); parse_secs(*msg, added_secs);
} }
auto added_sec_descs = get_sec_descs(added_secs); auto added_sec_descs = get_sec_descs(added_secs);
@@ -211,7 +211,7 @@ struct BinanceCoinFuturesSecurityRequestor::PImpl {
_FATAL_MSGF( _FATAL_MSGF(
"on_exchange_info. {} event was unexpected:\n{}", "on_exchange_info. {} event was unexpected:\n{}",
ccapi::Event::typeToString(event.getType()), ccapi::Event::typeToString(event.getType()),
event.toStringPretty()); event.toPrettyString());
break; break;
} }
return true; return true;
@@ -23,10 +23,11 @@ using std::int64_t;
using std::map; using std::map;
using std::ostream; using std::ostream;
using std::string; using std::string;
using std::string_view;
using std::vector; using std::vector;
static bool parse( static bool parse(
const map<string, string> &name_values, const map<string_view, string> &name_values,
vector<BinanceUsPriceStatistics> &result) { vector<BinanceUsPriceStatistics> &result) {
CCApiReader ccapi_reader{&name_values}; CCApiReader ccapi_reader{&name_values};
nlohmann::json js; nlohmann::json js;
@@ -69,10 +69,10 @@ static vector<string> get_sec_descs(const vector<Security> &sec_ptrs) {
static void validate_response(const ccapi::Event &event, bool is_expected) { static void validate_response(const ccapi::Event &event, bool is_expected) {
bts_assert_msgf( bts_assert_msgf(
event.getType() == ccapi::Event::Type::RESPONSE, event.getType() == ccapi::Event::Type::RESPONSE,
"Received event w/ unexpected type:\n{}", event.toStringPretty()); "Received event w/ unexpected type:\n{}", event.toPrettyString());
bts_assert_msgf( bts_assert_msgf(
is_expected, "Received unexpected response from Binance US:\n{}", is_expected, "Received unexpected response from Binance US:\n{}",
event.toStringPretty()); event.toPrettyString());
} }
struct BinanceUsSecurityRequestor::PImpl { struct BinanceUsSecurityRequestor::PImpl {
@@ -181,7 +181,7 @@ struct BinanceUsSecurityRequestor::PImpl {
for (const auto *msg : msgs) { for (const auto *msg : msgs) {
bts_assert_msgf( bts_assert_msgf(
msg->getType() == ccapi::Message::Type::GENERIC_PUBLIC_REQUEST, msg->getType() == ccapi::Message::Type::GENERIC_PUBLIC_REQUEST,
"Received msg w/ unexpected type:\n{}", msg->toStringPretty()); "Received msg w/ unexpected type:\n{}", msg->toPrettyString());
parse_secs<T>(*msg, types, added_secs); parse_secs<T>(*msg, types, added_secs);
} }
auto added_sec_descs = get_sec_descs(added_secs); auto added_sec_descs = get_sec_descs(added_secs);
@@ -200,7 +200,7 @@ struct BinanceUsSecurityRequestor::PImpl {
const CCApiCorrelationID & /*corr_id*/) { const CCApiCorrelationID & /*corr_id*/) {
validate_response(event, exch_info_req_pending); validate_response(event, exch_info_req_pending);
LOG(md, Diag, "Received Binance US exchange info (spots):\n{}", LOG(md, Diag, "Received Binance US exchange info (spots):\n{}",
event.toStringPretty()); event.toPrettyString());
update_securities<BinanceUsSpot>(msgs, {SEC_TYPE(SPOT)}); update_securities<BinanceUsSpot>(msgs, {SEC_TYPE(SPOT)});
exch_info_req_pending = false; exch_info_req_pending = false;
if (external_req_pending && !is_busy()) { if (external_req_pending && !is_busy()) {
@@ -21,10 +21,11 @@ using std::map;
using std::pair; using std::pair;
using std::size_t; using std::size_t;
using std::string; using std::string;
using std::string_view;
using std::vector; using std::vector;
static bool parse( static bool parse(
const map<string, string> &name_values, const map<string_view, string> &name_values,
vector<BinanceUsdsFuturesPriceStatistics> &result) { vector<BinanceUsdsFuturesPriceStatistics> &result) {
CCApiReader ccapi_reader{&name_values}; CCApiReader ccapi_reader{&name_values};
nlohmann::json js; nlohmann::json js;
@@ -60,7 +60,7 @@ void BinanceUsdsFuturesPriceStatisticsAdaptor::on_messages(
const auto n_total = n_hits + n_misses; const auto n_total = n_hits + n_misses;
_ERROR( _ERROR(
"on_messages. Parse failed on {}/{} elements:\n{}", n_misses, "on_messages. Parse failed on {}/{} elements:\n{}", n_misses,
n_total, event.toStringPretty()); n_total, event.toPrettyString());
continue; continue;
} }
for (const auto &bsr : binance_stats_recs) { for (const auto &bsr : binance_stats_recs) {
@@ -178,7 +178,7 @@ struct BinanceUsdsFuturesSecurityRequestor::PImpl {
for (const auto *msg : msgs) { for (const auto *msg : msgs) {
bts_assert_msgf( bts_assert_msgf(
msg->getType() == ccapi::Message::Type::GENERIC_PUBLIC_REQUEST, msg->getType() == ccapi::Message::Type::GENERIC_PUBLIC_REQUEST,
"Received msg w/ unexpected type:\n{}", msg->toStringPretty()); "Received msg w/ unexpected type:\n{}", msg->toPrettyString());
parse_secs(*msg, added_secs); parse_secs(*msg, added_secs);
} }
auto added_sec_descs = get_sec_descs(added_secs); auto added_sec_descs = get_sec_descs(added_secs);
@@ -212,7 +212,7 @@ struct BinanceUsdsFuturesSecurityRequestor::PImpl {
_FATAL_MSGF( _FATAL_MSGF(
"on_exchange_info. {} event was unexpected:\n{}", "on_exchange_info. {} event was unexpected:\n{}",
ccapi::Event::typeToString(event.getType()), ccapi::Event::typeToString(event.getType()),
event.toStringPretty()); event.toPrettyString());
break; break;
} }
return true; return true;
@@ -8,7 +8,7 @@ namespace bts::md {
using bts::domain::Quantity; using bts::domain::Quantity;
bool CCApiDepthWalker::parse_field( bool CCApiDepthWalker::parse_field(
const std::string &name, const std::string &value) { const std::string_view &name, const std::string &value) {
bts_assert(depth_ptr); bts_assert(depth_ptr);
auto &depth = *depth_ptr; auto &depth = *depth_ptr;
if (name == price_key) { if (name == price_key) {
@@ -15,7 +15,7 @@ class CCApiDepthWalker {
std::size_t price_idx = 0; std::size_t price_idx = 0;
std::size_t qty_idx = 0; std::size_t qty_idx = 0;
bool parse_field(const std::string &name, const std::string &value); bool parse_field(const std::string_view &name, const std::string &value);
}; };
} // namespace bts::md } // namespace bts::md
@@ -57,7 +57,7 @@ CCApiEventDemuxer::CCApiEventDemuxer(std::size_t corr_id_len)
CCApiEventDemuxer::~CCApiEventDemuxer() = default; CCApiEventDemuxer::~CCApiEventDemuxer() = default;
bool CCApiEventDemuxer::processEvent( void CCApiEventDemuxer::processEvent(
const ccapi::Event &event, ccapi::Session *session) { const ccapi::Event &event, ccapi::Session *session) {
unordered_map<CCApiCorrelationID, vector<const ccapi::Message *>> unordered_map<CCApiCorrelationID, vector<const ccapi::Message *>>
demuxed_msgs; demuxed_msgs;
@@ -91,7 +91,6 @@ bool CCApiEventDemuxer::processEvent(
hnd.on_messages(event, session, msgs, hnd_corr_id); hnd.on_messages(event, session, msgs, hnd_corr_id);
} }
} }
return true;
} }
void CCApiEventDemuxer::on_messages( void CCApiEventDemuxer::on_messages(
@@ -19,7 +19,7 @@ class CCApiEventDemuxer : public ccapi::EventHandler,
explicit CCApiEventDemuxer(std::size_t corr_id_len = 0); explicit CCApiEventDemuxer(std::size_t corr_id_len = 0);
~CCApiEventDemuxer() override; ~CCApiEventDemuxer() override;
bool processEvent( void processEvent(
const ccapi::Event &event, ccapi::Session *session) override; const ccapi::Event &event, ccapi::Session *session) override;
void on_messages( void on_messages(
@@ -10,7 +10,7 @@ class CCApiRequestor_Handler : public ccapi::EventHandler {
public: public:
bool processEvent( bool processEvent(
const ccapi::Event &event, ccapi::Session * /*session*/) override { const ccapi::Event &event, ccapi::Session * /*session*/) override {
LOG(md, Info, "on ccapi event: {}", event.toStringPretty(2, 2)); LOG(md, Info, "on ccapi event: {}", event.toPrettyString(2, 2));
return true; return true;
} }
}; };
@@ -40,7 +40,7 @@ inline ccapi::Subscription CCApiJsonRpcRequest::rpc_sub(
const io::websocket::JsonRpcRequest &req, const io::websocket::JsonRpcRequest &req,
std::string venue_name, std::string venue_name,
const CCApiCorrelationID &corr_id) { const CCApiCorrelationID &corr_id) {
return { return ccapi::Subscription{
std::move(venue_name), "", CCAPI_GENERIC_PUBLIC_SUBSCRIPTION, std::move(venue_name), "", CCAPI_GENERIC_PUBLIC_SUBSCRIPTION,
req.to_string(), corr_id.to_string()}; req.to_string(), corr_id.to_string()};
} }
@@ -38,7 +38,7 @@ void CCApiJsonRpcRouter::on_messages(
BTS_FATAL_MSGF( BTS_FATAL_MSGF(
"Exception occurred handling rpc " "Exception occurred handling rpc "
"message.\n***ex***:{}\n***event***:\n{}", "message.\n***ex***:{}\n***event***:\n{}",
msg, event.toStringPretty()); msg, event.toPrettyString());
}); });
} }
@@ -45,7 +45,7 @@ static bool parse(const ccapi::Message &msg, domain::Book &book) {
ask_walker.parse_field(name, value); ask_walker.parse_field(name, value);
} catch (const std::exception &ex) { } catch (const std::exception &ex) {
LOG(md, Fatal, "Error parsing depth update: {}\nmsg:\n{}", LOG(md, Fatal, "Error parsing depth update: {}\nmsg:\n{}",
std::string{ex.what()}, msg.toStringPretty()); std::string{ex.what()}, msg.toPrettyString());
bts_assert_msg(false, "Fatal error parsing depth update."); bts_assert_msg(false, "Fatal error parsing depth update.");
} }
} }
@@ -22,7 +22,7 @@ void CCApiMessageLogger::on_messages(
const CCApiCorrelationID &corr_id) { const CCApiCorrelationID &corr_id) {
auto msg_strs = msgs | ranges::views::transform([](const ccapi::Message *m) { auto msg_strs = msgs | ranges::views::transform([](const ccapi::Message *m) {
return m->toStringPretty(); return m->toPrettyString();
}); });
auto msgs_str = join(msg_strs.begin(), msg_strs.end(), "\n"); auto msgs_str = join(msg_strs.begin(), msg_strs.end(), "\n");
LOG(md, Info, "Received CCApi messages. name: {}; corr_id: {}; msgs:\n{}", LOG(md, Info, "Received CCApi messages. name: {}; corr_id: {}; msgs:\n{}",
+1 -1
View File
@@ -15,7 +15,7 @@ namespace bts::md {
class CCApiReader { class CCApiReader {
public: public:
using NameValues = std::map<std::string, std::string>; using NameValues = std::map<std::string_view, std::string>;
explicit CCApiReader(const NameValues *name_values); explicit CCApiReader(const NameValues *name_values);
@@ -39,7 +39,7 @@ void CCApiTradeAdaptor::on_messages(
ccapi::Message::typeToString(msg->getType())); ccapi::Message::typeToString(msg->getType()));
return; return;
} }
LOG(md, Diag, "Received CCApi trade msg:\n{}", msg->toStringPretty()); LOG(md, Diag, "Received CCApi trade msg:\n{}", msg->toPrettyString());
auto ts = TimeStamp::from_int64( auto ts = TimeStamp::from_int64(
msg->getTimeReceived().time_since_epoch().count()); msg->getTimeReceived().time_since_epoch().count());
auto sec_desc = corr_id.peek_string(); auto sec_desc = corr_id.peek_string();
@@ -54,7 +54,7 @@ void CCApiTradeAdaptor::on_messages(
} }
if (trades.empty()) { if (trades.empty()) {
LOG(md, Warn, "Received ccapi trade msg with no trades:\n{}", LOG(md, Warn, "Received ccapi trade msg with no trades:\n{}",
event.toStringPretty()); event.toPrettyString());
return; return;
} }
std::vector<const domain::TradeRecord *> trade_ptrs; std::vector<const domain::TradeRecord *> trade_ptrs;
Loaded 100 of 354 files, more files were not shown because too many files have changed in this diff. Show more