John Colvin
27720a3127
Increase SQLiteEnumerator page size to 65536
...
This performs better, and doesn't seem to be too large in practice
2024-11-26 18:53:33 +00:00
John Colvin
6f352a5c95
fix Add some nifty DataPacket/TableUpdate IObservable
2024-11-26 18:53:33 +00:00
John Colvin
68b42bdb01
Add async SQLiteEnumerator to read sqlite tables as IBlueRowCollection
2024-11-26 18:53:32 +00:00
John Colvin
db0df50b53
Add some nifty DataPacket/TableUpdate IObservable utils
2024-11-26 18:53:32 +00:00
John Colvin
bb6662a2c1
fix: Remove query from literal error stream
...
The parser can't handle colons in the expression (it throws/crashes)
2024-11-26 18:53:32 +00:00
John Colvin
64bd5eef4f
Load IDataStremFactory impls with dependency injection
...
So we can create ViewStreams that depend on other services, config, etc.
Previously, the PrimitiveManager relied purely on static construction of types
through reflection.
2024-11-26 18:53:32 +00:00
John Colvin
50611fe4a3
Add generic ViewStream, BlueRow types/interfaces
2024-11-26 18:53:32 +00:00
John Colvin
b6f64b4804
Propagate ViewStream errors to webapp frontend
2024-11-25 15:26:58 +00:00
John Colvin
fa5cc6ab38
Add ErrorChanged event to propagate DataStream exceptions
2024-11-25 15:26:58 +00:00
John Colvin
61e61e8bb8
refactor: Run RemoteQueryManager as an IHostedService
...
This paradigm more accurately represents the RemoteQueryManager as a
Service that can be started/stopped.
2024-11-25 15:26:58 +00:00
John Colvin
1486c1185c
fix: Add null check to NonIncGroupByStream
2024-11-25 15:26:58 +00:00
John Colvin
a65eaf1cc4
fix: Add null check in TimeBasedFilter.cs
2024-11-25 15:26:58 +00:00
John Colvin
cc153a1ced
Add DataStreamPublisher service modules designed for Dependency Injection
...
* Change DataStreamPublisher json fields to match cs class property names verbatim
* Rename Publishers -> PublisherNodes in DataStreamPublisherConfig
2024-11-25 15:26:57 +00:00
John Colvin
b19f720e42
refactor: Decouple ViewPublisher from IPubTableBroker impl
2024-11-25 15:26:57 +00:00
John Colvin
51d934de27
Add PubSub service modules designed for Dependency Injection
...
* Get rid of JsonPropertyName attributes in PubSubConfig
- The Dependency Injection framework (based around dotnet ApplicationBuilder)
does not support this attribute, and maps property names verbatim to json fields.
* Change PubSubServer json fields to match cs class property names verbatim
2024-11-25 15:26:57 +00:00
John Colvin
23e47f1749
refactor: Decouple RemoteQueryManager from IManagedSubscriber, IPubTableBroker impl
2024-11-25 15:26:57 +00:00
John Colvin
48b3852e6f
refactor: Rename CloudStreamManager.cs -> RemoteQueryManager.cs
...
This matches the file name
2024-11-25 15:26:57 +00:00
John Colvin
7ebafd8246
cleanup: Fix numerous build warnings in DataStreamSuite
...
Nearly all warnings were due to nullability issues, since the concept of
nullable reference types did not exist when the DataStream code was
originally written.
* PubSubClientMessage
- Normalize indentation/whitespace
- Fix some typos
- Address nullability issues to fix build warnings
- Modernize code
* Compression related files
- ZipEntry
- ZipFile
- ZipInputStream
- TarHeader
* RemoteMessage
- make 'message' nullable to resolve warnings elsewhere (in calling code)
* PubSubSingleTagStateEvent.cs
* PubSubMessageReactor
* PubSubState
* SystemHeartbeatPublisher
* SystemStatsPublisher
* SingleTagData
* ManagedTCPMessageHub
* PubsubDeserializationController
* PublisherCore
* InflaterHuffmanTree
* AsyncTCPConnection_APM
* ThreadSafeQueue
* ManagedSubscriber
* SubscriberCore
* PubSubServerManagedTCPConnection
* AsyncTCPConnection_APM
* SuperMapStream
* MapStream
* TradeBarsStream
2024-11-25 15:26:57 +00:00
John Colvin
8f2e37d644
cleanup: Disable nullability warnings
...
* BZip2InputStream
* TarInputStream
* Inflater
* InflaterDynHeader
* BZip2OutputStream
* DeflaterHuffman
* ParallelJoinTest
2024-11-25 15:26:57 +00:00
Taha
6163887cd3
Bunch of CME queries are now working
2024-10-30 13:36:23 -04:00
Taha
f9aa9dd982
Merge branch 'development' of bitbucket.org:bluetradesys/spark into development
2024-10-15 23:48:34 -04:00
Taha
22f1ab44f9
Basic set of streams that can talks ot the LLM and get DSL back from few-shot prompt
2024-10-15 23:38:49 -04:00
Taha
288b7712db
Added and modified more streams to address teh CME NL queries
2024-10-09 13:59:01 -04:00
Taha
322c06bfbe
Fixing build relted to BCG vs bts namespace
2024-08-21 16:24:22 -04:00
Taha
b68fceb362
Cleaning up FileStreams
2024-08-21 16:11:57 -04:00
Taha
f06beff22f
Added new benzinga streams
2024-07-17 02:02:01 -04:00
Taha
050e21e22b
Adding TableMetaDataStream
2024-07-04 23:39:04 -04:00
Taha Afzal
672ad6d237
Cleaning Config 1.0 (not completely done)
2024-06-30 23:17:20 -04:00
Taha
393f304be4
some buf gixes + cleanup
2024-06-26 00:23:19 -04:00
Taha
87b6a2044c
PubSubServer's linux issues fixed
2024-06-23 21:13:37 -04:00
Taha
fac0a9c25b
RemoteQueryStream
2024-06-06 09:20:05 -05:00
John Colvin
015d846941
feat: Add Pandas(Script)Stream
2024-06-05 15:44:35 +00:00
John Colvin
1824d5f9a4
feat: Add [incremental] MongoCollectionStream
2024-06-03 13:46:51 +00:00
John Colvin
d6fae238f9
Add IObservable ViewStream wrapper
2024-06-03 13:46:51 +00:00
John Colvin
5741ec9ab2
cleanup: Remove unnecessary usings (namespaces)
2024-06-03 13:46:51 +00:00
John Colvin
9050318ceb
refactor: Replace HighPerformanceTimer w/ Stopwatch
...
This is needed to build DataStreamSuite on Linux.
'Stopwatch' is a platform independent timer class included with the
System.Runtime library in modern dotnet. 'HighPerformanceTimer' is a BTS
class that predates Stopwatch, and is NOT platform independent as it
relies on Windows kernel32.dll QueryPerformanceCounter/Frequency fns.
2024-06-03 13:46:51 +00:00
John Colvin
e95997ea17
cleanup: Chip away a few nullability warnings in DataStreamSuite
...
This is a start, but there're still > 700 of them!
2024-05-25 15:37:33 +00:00
Taha Afzal
90ec64d25c
deleting old serializers
2024-05-25 15:37:33 +00:00
Taha Afzal
061481419e
Streams can be published and subscribed now
2024-05-25 15:37:33 +00:00
Taha Afzal
8035255868
Cleaned up the cashe held by PublisherCore
2024-05-25 15:37:33 +00:00
Taha Afzal
7feaf869dc
Stream serialization revamp, all tests pass (TestsStream.s)
2024-05-25 15:37:33 +00:00
Taha Afzal
7104e90fc9
Adding tests for stream serialization
2024-05-25 15:37:33 +00:00
Taha Afzal
07274f897a
Pubsub stats
2024-05-25 15:37:33 +00:00
Taha Afzal
f626e53415
CPublisherCore, SubscriberCore, PubsubMessageReactor claned
2024-05-25 15:37:33 +00:00
Taha Afzal
888c06262b
PublisherCore cleaned up
2024-05-25 15:37:33 +00:00
Taha Afzal
90bcf5a455
Adding testss for PubSubMessages
2024-05-25 15:37:33 +00:00
Taha Afzal
4510e120ab
enabling some streams like max, that were dependent on AddCOlumns (thta John fixed in the last commit)
2024-02-12 13:21:28 -05:00
Taha Afzal
da2af06ee6
Merging
2024-02-08 15:48:51 -05:00
John Colvin
8660dea85a
build: Downgrade new DataStream code from net8 -> net7
...
The AddColumnsStream, SortStream, and requisite yacc/lex grammar were
originally added to a new branch, off of development, which is on net8.
I forgot to change it to net7 when I cherry-picked the commits for the
regular development branch.
It's wasn't quite as simple as flipping net8 -> net7 in csproj files,
because I was eager to use new language features only available in net8!
Namely, primary constructors and collection expressions.
2024-02-08 14:24:31 -06:00
Taha Afzal
8c1342230b
RowDiff added
2024-02-08 14:11:10 -05:00