The first fix being correct is what made the second one hard. You measured it, you verified it on two devices, so when the same complaint came back the padding was the one area you already cleared.
Fixes that work is always a good reason not to look there again: which is really wrong when the second complaint has a different cause and the same words!
40 minutes on the two render copies, and both were drawing 1080×1920 correctly the whole time.
The 62.7 → 99.9 gap on ARC-AGI-3 is the number I'd want explained before anything else. If the adapted harness holds opaque state between requests and compresses long sessions, the two runs aren't measuring the same system — one is the model, the other is the model plus somebody's scaffolding. Both are worth measuring. They just aren't comparable, and only one of them is reproducible by anyone outside.
Predictable which one ends up in the announcement.
The metric I'd actually watch is hallucinations about its own capabilities. A model that's confidently wrong about what it can do costs you more than one that scores badly, because the failure looks like a result.
The audit-versus-modelling split is a good framing. Checking that a practice exists is cheap; checking that it does anything is a different job.
On a much smaller scale, I had a rule about how we publish, wrote it down, and got on with it. Weeks later I compared it to what had actually gone out. Nothing matched — and not because things had drifted. The rule described a practice that had never existed, including on the day I wrote it.
So I'm curious how the assessments handle that: do you get at the artefacts a process leaves behind, or mostly the process as described? Asking because the described version always looks healthier than mine did.
The skipped list being where the progress showed is the part I'll keep. An absence doesn't record itself — you had to make it write down what it didn't do before you could see it getting better at anything.
Same thing got me somewhere much more boring than Reddit. A lookup fell back to a default when the key missed, so the output looked fine for eleven weeks. Nothing logs a miss unless you make it.
The status manager in Sheets is the part I’d have expected you to skip, and it’s doing more work than a database would have. Keying updates by source_url means a re-run recognises what’s already been through, which is the thing most of these pipelines get wrong.
One question about the edge of that. The Instagram container polls FINISHED for 12 × 5s, then you write PUBLISHED to Sheets. If the publish succeeds but the Sheets write fails, or the polling times out on a container that actually completed, does the next run see DRAFT and publish again? Asking because that gap between “the side effect happened” and “we recorded that it happened” is where I’ve been bitten, and it only ever shows up in production.
Also liked “you never invent API parameters; when unsure, say so explicitly” plus the VERIFY MANUALLY list. Though it’s still an instruction rather than a guarantee — the model can comply 97% of the time and the 3% arrives as an importable JSON with a plausible field name that doesn’t exist. Did you end up validating generated node configs against the n8n schema before import, or was reading them enough at this scale?
The first fix being correct is what made the second one hard. You measured it, you verified it on two devices, so when the same complaint came back the padding was the one area you already cleared.
Fixes that work is always a good reason not to look there again: which is really wrong when the second complaint has a different cause and the same words!
40 minutes on the two render copies, and both were drawing 1080×1920 correctly the whole time.
Lol. It's my own story though: the rule was mine and so was the mess.
The 62.7 → 99.9 gap on ARC-AGI-3 is the number I'd want explained before anything else. If the adapted harness holds opaque state between requests and compresses long sessions, the two runs aren't measuring the same system — one is the model, the other is the model plus somebody's scaffolding. Both are worth measuring. They just aren't comparable, and only one of them is reproducible by anyone outside.
Predictable which one ends up in the announcement.
The metric I'd actually watch is hallucinations about its own capabilities. A model that's confidently wrong about what it can do costs you more than one that scores badly, because the failure looks like a result.
The audit-versus-modelling split is a good framing. Checking that a practice exists is cheap; checking that it does anything is a different job.
On a much smaller scale, I had a rule about how we publish, wrote it down, and got on with it. Weeks later I compared it to what had actually gone out. Nothing matched — and not because things had drifted. The rule described a practice that had never existed, including on the day I wrote it.
So I'm curious how the assessments handle that: do you get at the artefacts a process leaves behind, or mostly the process as described? Asking because the described version always looks healthier than mine did.
The skipped list being where the progress showed is the part I'll keep. An absence doesn't record itself — you had to make it write down what it didn't do before you could see it getting better at anything.
Same thing got me somewhere much more boring than Reddit. A lookup fell back to a default when the key missed, so the output looked fine for eleven weeks. Nothing logs a miss unless you make it.
The status manager in Sheets is the part I’d have expected you to skip, and it’s doing more work than a database would have. Keying updates by source_url means a re-run recognises what’s already been through, which is the thing most of these pipelines get wrong.
One question about the edge of that. The Instagram container polls FINISHED for 12 × 5s, then you write PUBLISHED to Sheets. If the publish succeeds but the Sheets write fails, or the polling times out on a container that actually completed, does the next run see DRAFT and publish again? Asking because that gap between “the side effect happened” and “we recorded that it happened” is where I’ve been bitten, and it only ever shows up in production.
Also liked “you never invent API parameters; when unsure, say so explicitly” plus the VERIFY MANUALLY list. Though it’s still an instruction rather than a guarantee — the model can comply 97% of the time and the 3% arrives as an importable JSON with a plausible field name that doesn’t exist. Did you end up validating generated node configs against the n8n schema before import, or was reading them enough at this scale?