If you utilize Apache Spark, you probably have a few applications that consume some data from external sources and produce some intermediate result, that is about to be consumed by some applications further down the processing chain, and so on until you get a final result.
We suspect that because we have a similar pipeline with lots of processes like this one:
Click here for a bit larger version
Each rectangle is a Spark application with a set of their own execution parameters, and each arrow is an equally parametrized dataset (externally stored highlighted with a color; note the number of intermediate ones). This example is not the most complex of our processes, it’s fairly a simple one. And we don’t assemble such workflows manually, we generate them from Process Templates (outlined as groups on this flowchart).
So here comes the One Ring, a Spark pipelining framework with very robust configuration abilities, which makes it easier to compose and execute a most complex Process as a single large Spark job.
And we just made it open source. Perhaps, you’re interested in the details.
There are a lot of command line tips and trics in the internet. Most of them discribe the trivials like "learn the hotkeys" or "sudo !! will run previous command with sudo". Instead of that, I will tell you what to do when you have already learned the hotkeys and know about sudo !!.
After hitting the keyboard buttons for the past 20 minutes, as if he was typing for his life, ghostinushanka turns to me with a half-mad look in his eyes and a sly smile, “Dude, I think I got it.
Look at this” — as he points to one of the characters on screen — “I bet my red hat that if we add what I’ve just sent you here” — as he points to another place in the code — “there will be no error anymore.”
Slightly puzzled and tired I modify the sed expression we’ve been figuring out for some time now, save the file and run systemctl varnish reload. Error message gone…
“Those emails I’ve exchanged with the candidate,” my colleague continues, as his smile changes to a wide and genuine grin, “It suddenly struck me that this is the very same exact problem!”
