Some time ago, in a discussion on one of SObjectizer's releases, we were asked: "Is it possible to make a DSL to describe a data-processing pipeline?" In other words, is it possible to write something like that:
A | B | C | D
and get a working pipeline where messages are going from A to B, and then to C, and then to D. With control that B receives exactly that type that A returns. And C receives exactly that type that B returns. And so on.
It was an interesting task with a surprisingly simple solution. For example, that's how the creation of a pipeline can look like:
auto pipeline = make_pipeline(env, stage(A) | stage(B) | stage(C) | stage(D));Or, in a more complex case (that will be discussed below):
auto pipeline = make_pipeline( sobj.environment(),
stage(validation) | stage(conversion) | broadcast(
stage(archiving),
stage(distribution),
stage(range_checking) | stage(alarm_detector{}) | broadcast(
stage(alarm_initiator),
stage( []( const alarm_detected & v ) {
alarm_distribution( cerr, v );
} )
)
) );In this article, we'll speak about the implementation of such pipeline DSL. We'll discuss mostly parts related to stage(), broadcast() and operator|() functions with several examples of usage of C++ templates. So I hope it will be interesting even for readers who don't know about SObjectizer (if you never heard of SObjectizer here is an overview of this tool).