Commit Graph
34 Commits
Author SHA1 Message Date
John Colvin a0d8dfd3de build: Resolve app data dir and shared mem prefix at runtime
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
2024-10-21 13:29:59 +00:00
John Colvin 146a52f0ab Configure datastream deployment with docker-compose
To deploy to aws, connect to our aws vpn then run:
tools/deploy-docker-compose.sh

The deploy script does the following:
* builds sparkd
* builds datastream apps (pubsub, md-publisher, and webapp)
* uploads the build to aws
* bounces all services (sparkd, pubsub, md-publisher, and webapp)

Note: the name 'md-publisher' refers to DataStreamPublisher w/ MarketData
configuration
2024-10-21 13:29:59 +00:00
John Colvin 6a881c8f2b Configure apps as a distributed system via Docker compose 2024-10-21 13:29:59 +00:00
John Colvin a770724de1 build: Copy configuration profiles to build dir
So they are readily available to publish/deploy scripts.
2024-10-21 13:29:58 +00:00
John Colvin e855f136a1 whitespace: Format PubSubServer.json (indentation issues) 2024-10-21 13:29:58 +00:00
John Colvin 894add7df6 Add scripts to deploy data stream apps 2024-10-21 13:29:58 +00:00
John Colvin 824912383f pubsub server checkpt 2024-10-21 13:29:58 +00:00
John Colvin a9b91de56a fix add pubsub services 2024-10-21 13:29:58 +00:00
John Colvin 6dc4387867 Remove unused PubSubConfig props 2024-10-21 13:29:58 +00:00
John Colvin ac80c8ed1d Add PubSub service modules designed for Dependency Injection
* Get rid of JsonPropertyName attributes in PubSubConfig
  - The Dependency Injection framework (based around dotnet ApplicationBuilder)
    does not support this attribute, and maps property names verbatim to json fields.

* Change PubSubServer json fields to match cs class property names verbatim
2024-10-21 13:29:57 +00:00
John Colvin 6fee2750a3 Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
In config json files (not actual code change)
2024-10-21 13:29:57 +00:00
Taha f06beff22f Added new benzinga streams 2024-07-17 02:02:01 -04:00
Taha 87b6a2044c PubSubServer's linux issues fixed 2024-06-23 21:13:37 -04:00
John Colvin 3a27af87d1 PubSubServer shouldn't download config from sparkd by default
As far as I know, it only needs spark/protobuf config to initialize the
c++ logging system. Eventually, we should get rid of that dependency and
use builtin Microsoft Logging facilities for c# instead. In the
meantime, PubSubServer will load config from local files by default
(i.e., spark.conf).
2024-06-06 09:21:01 -05:00
John Colvin d2e9fe8237 Change line endings to crlf in a few cs files 2024-06-06 09:21:00 -05:00
John Colvin b03fec7f79 Change line endings to crlf in a few cs files 2024-06-06 09:20:59 -05:00
Taha 07bade53c0 Fixing AsyncTCPCOnnection_APM's double partial messages for messages > 100 MB 2024-06-06 09:20:05 -05:00
John Colvin 45f02391d1 cleanup: Remove sparkd login creds from PubSubServerConfig
It only needs a Sparkd url to download logging config. TBH, we can probably
get rid of that too, and simply read the local spark.conf.
2024-06-06 09:15:17 -05:00
John Colvin e72e0c0a30 fix: Make PubSubServer actually use sparkd url in config 2024-06-06 09:15:17 -05:00
John Colvin c99d5fe24c Add support for core PubSub config via json 2024-06-06 09:15:16 -05:00
John Colvin 6240d89a08 Map PubSubServerConfig json to historic PubSub3Config 2024-06-03 13:46:51 +00:00
John Colvin 306c572b53 improvement: Polish clis in DataStreamSuite
Specifically PubSubServer, DataStreamPublisher and DSQA

* Use CommandLineParser (nuget pkg)
* Clean up csproj files a bit while we're at it
2024-06-03 13:46:51 +00:00
John Colvin e0459c54fd build: Fix 'pubsub' -> 'PubSub' typo in csproj ProjectReferences
Now all DataStream projects build on Linux (except DSQA, which is a winforms app)
2024-05-25 15:37:33 +00:00
Taha Afzal 5a7b2965be PubUBState cleaned up and modernized 2024-05-25 15:37:33 +00:00
Taha Afzal 287ee291cc Adding a stream that send a TCP packet to the echo server and monitors the n_sends and n_recvs 2024-01-25 13:37:41 -05:00
Taha Afzal 8cbbbc6da3 Merging 2024-01-13 23:45:51 -05:00
Taha Afzal d50e8cda50 Adding AsyncTCPConnectionAcceptor_APM and AsyncTCPConnector_APM to create ITCPConnection 2024-01-11 16:10:51 -05:00
Taha Afzal 694f8b8d11 AsyncTCPConnection_APM works now 2024-01-09 14:34:15 -05:00
Taha Afzal 5b4b41d4eb Adding MongoDB astream integration 2024-01-02 00:44:52 -05:00
Taha Afzal 1ceea4af5a Changing TCPUtils to use SocketAsyncEventArgs instead of TAP functions, simple client<->server works with echoing messages 2023-12-20 11:32:53 -05:00
Taha 7bb7f1a5e2 PubSub is working now 2023-09-30 12:53:51 -04:00
Taha 404698152e Merging, and adding test project 2023-09-30 12:52:37 -04:00
Taha 797816be7e Merging changes 2023-09-04 16:25:40 -04:00
John Colvin ecbae62d07 feat: Add bones for DataStreamSuite
TODO (Taha): Migrate source files from edge
2023-08-22 15:04:08 -04:00