Debugging *

For 30 days, the IDE stayed closed after 8 PM. I expected the experiment to affect sleep, but the more interesting result appeared somewhere else: in Git history. The month produced fewer bug-fix commits, shorter debugging sessions, and a surprisingly different defect pattern. So I wrote a couple of scripts, reconstructed where several bugs had actually come from, compared the results with the previous month, and ended up changing the way I think about late-night coding.

We released the new version of Chipmunk, software for viewing/analyzing log files. V.3 is wholly reworked with an accent to performance and rethought considering usability. Below short list of the most important changes and a general description.

Have you ever heard of Image File Execution Options (IFEO)? It is a registry key under HKEY_LOCAL_MACHINE that controls things like Global Flags and Mitigation Policies on a per-process basis. One of its features that drew my attention is a mechanism designed to help developers debug multi-process applications. Imagine a scenario where some program creates a child process that crashes immediately. In case you cannot launch this child manually (that can happen for various reasons), you might have a hard time troubleshooting this problem. With IFEO, however, you can instruct the system to launch your favorite debugger right when it's about to start this troublesome process. Then you can single-step through the code and figure what goes wrong. Sounds incredibly useful, right?

I don't know about you, but I immediately saw this feature as a mechanism for executing arbitrary code when someone creates a new process. Even more importantly, it happens synchronously, i.e., the target won't start unless we allow it. Internally, the system swaps the path to the image file with the debugger's location, passing the former as a parameter. Therefore, it becomes the debugger's responsibility to start the application and then attach itself to it.

So, are there any limitations on what we can do if we register ourselves as a debugger? Let's push this opportunity to the limits and see what we can achieve.

What is distributed tracing? Distributed tracing is a method used to profile and monitor applications, especially those built using a microservices architecture. Distributed tracing helps pinpoint where failures occur and what causes poor performance.

Let’s have a look at a simple prototype. A user fetches information about a shipment from `logistic` service. logistic service does some computation and fetches the data from a database. logistic service doesn’t know the actual status of the shipment, so it has to fetch the updated status from another service `tracking`. `tracking` service also needs to fetch the data from a database and to do some computation.

In the screenshot below, we see a whole life cycle of the request issued to `logistics` service:

After hitting the keyboard buttons for the past 20 minutes, as if he was typing for his life, ghostinushanka turns to me with a half-mad look in his eyes and a sly smile, “Dude, I think I got it.

Look at this” — as he points to one of the characters on screen — “I bet my red hat that if we add what I’ve just sent you here” — as he points to another place in the code — “there will be no error anymore.”
Slightly puzzled and tired I modify the sed expression we’ve been figuring out for some time now, save the file and run systemctl varnish reload. Error message gone…

“Those emails I’ve exchanged with the candidate,” my colleague continues, as his smile changes to a wide and genuine grin, “It suddenly struck me that this is the very same exact problem!”