We really can't define them at compile time anymore as we build multiple
exes that depend on libspark in the same dir (e.g., spark api server,
pubsub, ds-publisher, webapp, etc.). That means one libspark.so (or dll)
must serve all exes. We should compile a consistent libspark to mitigate
the risk of running an exe with the wrong version (of libspark). For
example, running pubsub with a libspark configured for spark api server,
leftover from an earlier build.
Bugs (fixed):
* md-publisher silently fails if it tries to connect before those services are "ready".
The Docker compose "depends_on" property ensures sparkd and pubsub
containers are up, but doesn't prevent startup race bugs (e.g., if
md-publisher and pubsub start around the same time, and md-publisher
tries to connect before pubsub has started accepting connections). To
address these race conditions, we use a nifty tool called "dockerize"
References:
* https://github.com/jwilder/dockerize
Since it spawns a new thread on each attempt, uncaught exceptions were previously undetected,
and caused silent failures down the pipe. This should make it easier to track such bugs down.
We can't read spark protobuf Config until it has been initialized,
and since ViewStream doesn't have the configServer param handy it
isn't able to initialize config if necessary. In that case, it would simply
block until config was initialized by someone else. Such a risk doesn't
seem justified for this particular configuration dependency
(DataStreamConfig.UseUncopiedNewRowsInFire). For now, its hardcoded to false.
As a static function that could be called from anywhere in the code
(directly or indirectly), 'Configuration.Initialize' was vulnerable to
race bugs. The "Configuration.get<T>()" method, whose calls are
ubiquitous in our code, would initialize config if it hadn't been done
so already. Without a reference to the configServer param, this param
would effectively be ignored/overridden with null.
It's important that login only be performed once, for obvious reasons
(our exes that connect to sparkd are generally designed to process data
for a single user). Furthermore, not every routine that requires a
successful login has a reference to the credentials needed to do it. To
address this, we now pass the ISparkdLoginService singleton to these
routines so they can wait until the singular login task has completed
before they begin.
John Colvin