All streams

I'm part of a systems support team, where alongside diagnostics and incident troubleshooting I also do development work. One of the systems we support ingests telemetry from a large fleet of IoT trackers installed on vehicles. Every few seconds each device reports its coordinates and status parameters. An internal processing service turns that raw stream into higher level business objects: events, incidents, trips, that the rest of the platform consumes.

The system has been in pure maintenance mode for years. No development budget, no vendor to call. It still has to be supported though, people use it every day.

Every so often one device develops a hardware fault and starts “spamming”: emitting an abnormal volume of points or malformed events in a short window. That degrades the processing pipeline not only for that device, but for everyone sharing it.

Finding the culprit used to mean a first line engineer manually pulling several raw tables for the relevant day (events, incidents, raw points), cross referencing them by timestamp and device ID, and eyeballing the result for anomalous patterns.

With the events table alone running 2 to 3 million rows a day, that was a 6 to 7 hour job, basically an entire shift, with no guarantee of actually finding the source.

Benchmarks exist so that engineers can test their projects, observe competitors, and compare their performance. Each benchmark serves a specific purpose.

ARC-AGI-3 was created by the ARC Prize Foundation, founded by François Chollet, to evaluate general intelligence and the learning capabilities of AI agents. The premise behind its complex, game-like interactive tasks was that they could only be solved by an artificial intelligence capable of exploring an unfamiliar environment, grasping rules on the fly, planning actions, and adapting to new conditions. In other words, an AI that truly knows how to learn.

A number of projects claim a 100% success rate on this benchmark. But this is not a victory. In this article, I will explain why.