Commit Graph
32 Commits
Author SHA1 Message Date
John Colvin b20c38bd1a build: Resolve app data dir and shared mem prefix at runtime
We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
2024-11-29 14:58:54 +00:00
John Colvin 7e8ee6a22c Configure datastream deployment with docker-compose
To deploy to aws, connect to our aws vpn then run:
tools/deploy-docker-compose.sh

The deploy script does the following:
* builds sparkd
* builds datastream apps (pubsub, md-publisher, and webapp)
* uploads the build to aws
* bounces all services (sparkd, pubsub, md-publisher, and webapp)

Note: the name 'md-publisher' refers to DataStreamPublisher w/ MarketData
configuration
2024-11-29 14:57:44 +00:00
John Colvin 00c110beb2 Configure apps as a distributed system via Docker compose 2024-11-29 14:57:44 +00:00
John Colvin 435bad013f build: Copy configuration profiles to build dir
So they are readily available to publish/deploy scripts.
2024-11-29 14:57:44 +00:00
John Colvin 397f6156dc whitespace: Format PubSubServer.json (indentation issues) 2024-11-29 14:57:43 +00:00
John Colvin 250f34e3de Add scripts to deploy data stream apps 2024-11-29 14:57:43 +00:00
John Colvin 3179e9add3 Remove unused PubSubConfig props 2024-11-29 14:57:43 +00:00
John Colvin 31b065c854 Add PubSub service modules designed for Dependency Injection
* Get rid of JsonPropertyName attributes in PubSubConfig
  - The Dependency Injection framework (based around dotnet ApplicationBuilder)
    does not support this attribute, and maps property names verbatim to json fields.

* Change PubSubServer json fields to match cs class property names verbatim
2024-11-29 14:57:43 +00:00
John Colvin 0d1966e752 Change pub sub schema prefix 'bluto://' -> 'bts-cloud://'
In config json files (not actual code change)
2024-11-29 14:57:43 +00:00
Taha f06beff22f Added new benzinga streams 2024-07-17 02:02:01 -04:00
Taha 87b6a2044c PubSubServer's linux issues fixed 2024-06-23 21:13:37 -04:00
John Colvin 3a27af87d1 PubSubServer shouldn't download config from sparkd by default
As far as I know, it only needs spark/protobuf config to initialize the
c++ logging system. Eventually, we should get rid of that dependency and
use builtin Microsoft Logging facilities for c# instead. In the
meantime, PubSubServer will load config from local files by default
(i.e., spark.conf).
2024-06-06 09:21:01 -05:00
John Colvin d2e9fe8237 Change line endings to crlf in a few cs files 2024-06-06 09:21:00 -05:00
John Colvin b03fec7f79 Change line endings to crlf in a few cs files 2024-06-06 09:20:59 -05:00
Taha 07bade53c0 Fixing AsyncTCPCOnnection_APM's double partial messages for messages > 100 MB 2024-06-06 09:20:05 -05:00
John Colvin 45f02391d1 cleanup: Remove sparkd login creds from PubSubServerConfig
It only needs a Sparkd url to download logging config. TBH, we can probably
get rid of that too, and simply read the local spark.conf.
2024-06-06 09:15:17 -05:00
John Colvin e72e0c0a30 fix: Make PubSubServer actually use sparkd url in config 2024-06-06 09:15:17 -05:00
John Colvin c99d5fe24c Add support for core PubSub config via json 2024-06-06 09:15:16 -05:00
John Colvin 6240d89a08 Map PubSubServerConfig json to historic PubSub3Config 2024-06-03 13:46:51 +00:00
John Colvin 306c572b53 improvement: Polish clis in DataStreamSuite
Specifically PubSubServer, DataStreamPublisher and DSQA

* Use CommandLineParser (nuget pkg)
* Clean up csproj files a bit while we're at it
2024-06-03 13:46:51 +00:00
John Colvin e0459c54fd build: Fix 'pubsub' -> 'PubSub' typo in csproj ProjectReferences
Now all DataStream projects build on Linux (except DSQA, which is a winforms app)
2024-05-25 15:37:33 +00:00
Taha Afzal 5a7b2965be PubUBState cleaned up and modernized 2024-05-25 15:37:33 +00:00
Taha Afzal 287ee291cc Adding a stream that send a TCP packet to the echo server and monitors the n_sends and n_recvs 2024-01-25 13:37:41 -05:00
Taha Afzal 8cbbbc6da3 Merging 2024-01-13 23:45:51 -05:00
Taha Afzal d50e8cda50 Adding AsyncTCPConnectionAcceptor_APM and AsyncTCPConnector_APM to create ITCPConnection 2024-01-11 16:10:51 -05:00
Taha Afzal 694f8b8d11 AsyncTCPConnection_APM works now 2024-01-09 14:34:15 -05:00
Taha Afzal 5b4b41d4eb Adding MongoDB astream integration 2024-01-02 00:44:52 -05:00
Taha Afzal 1ceea4af5a Changing TCPUtils to use SocketAsyncEventArgs instead of TAP functions, simple client<->server works with echoing messages 2023-12-20 11:32:53 -05:00
Taha 7bb7f1a5e2 PubSub is working now 2023-09-30 12:53:51 -04:00
Taha 404698152e Merging, and adding test project 2023-09-30 12:52:37 -04:00
Taha 797816be7e Merging changes 2023-09-04 16:25:40 -04:00
John Colvin ecbae62d07 feat: Add bones for DataStreamSuite
TODO (Taha): Migrate source files from edge
2023-08-22 15:04:08 -04:00