How results here get checked, and what a full recheck after an AI model switch found

I do research with AI agents: they write code, run computations and draft text, and I direct and check the work. Agents can produce work that looks right and is not, so every result on this site goes through up to five levels of checks before it is published. The more a result matters, the more levels it goes through.

1. Rereading. The agent that produced the work rereads it. This catches typos and little else.
2. Consistency. The numbers have to agree with each other and with the source files.
3. A second program. A second implementation, written from scratch without looking at the first, has to produce the same numbers. This level catches a lot, and AI agents make it cheap enough to do every time.
4. An outside source. The numbers are compared with something I do not control that already has the answer: a published table, or the OEIS (the Online Encyclopedia of Integer Sequences, a public database of number sequences).
5. An expert looking for flaws. For example, when I extended the published tables for a set of octal games (two-player games in which players take turns removing counters from heaps), Achim Flammenkamp, who maintains the reference tables for these games, reviewed my solver and reran parts of the computation on his own machines with his own programs: one game over his old table's range, and two others over their full new ranges. Then he listed the results on his page.

The recheck after the model switch#

In mid-July, partway through a session, I found that my sessions had been running on a different AI model from the one I had chosen, for about two and a half days, without my knowledge. The session record shows only that the model changed. I still don't know why. Because the work had come from a model I had not picked, I asked for a full recheck of everything done under it.

The session transcripts record which model wrote each message, so I could find exactly when the switch started and ended and list every file produced in between. Items about to go out to other people, such as draft emails and text prepared for the OEIS, were checked first. Then fresh agents redid the work instead of rereading it: they wrote new code, went back to the original sources, and were told not to trust the documents being checked.

The computed numbers held up. The data files for the octal games were recomputed from scratch and came out identical, byte for byte.

The errors were elsewhere:

• A finding nobody had marked. The recomputed data for one game, called 0.314, contained a new rare value, the kind of unusual value that people who study these games watch for. It was a real result. It had sat in data that had already passed its checks, and nobody had noticed it, so the write-up undersold the work.
• A wrong draft reply. A draft reply to Flammenkamp got the status of some games wrong. It said the work of extending one game's table was essentially finished when it was only a little over half done.
• A search done the wrong way. An agent had searched the OEIS to see whether the lists of nim values for ten of my octal games were already there, and reported that none of them were. My lists started counting at position 0, while the OEIS lists for these games start at position 1, so the search never found a match. Done again correctly, all ten were already in the database. So the right step was to add my new values to those existing entries rather than propose new ones; the OEIS entries that came out of this work are mostly those existing entries, extended. This one was found only on a deeper second pass that I asked for after the first recheck.

All three were caught by reading the work against the source records: recomputing the data, comparing the draft with the actual progress of each game, and noticing that an older note on my project status page already listed one of the games in the OEIS, which contradicted the search.

What I changed#

I now check the checks as well as the results. For every check that has passed, I ask what method it used and whether that method could miss something, such as a different starting position, a different version of the data, or a different encoding. I scan data that has already passed its checks for findings nobody has marked. And every archived agent report now states which AI model produced it.

Scope

This covers one model switch and one person's program, and it does not establish a general rule.

For readers in software:how the software was built, and which parts you could reuse

Treat AI output like a binary from a build server you do not control: rebuild it from source, compare it with known-good outputs, checksum long runs, and make sure the system that produced a result is never the only one that checked it.

@misc{orden2026verificationladder,
  author       = {Tyler Satchel Orden},
  title        = {How results here get checked, and what a full recheck after an AI model switch found},
  year         = {2026},
  howpublished = {Cyborg Mathematics},
  url    = {https://cyborgmathematics.com/writing/verification-ladder}
}