The record I keep of errors my checks caught, and two that changed how I work

I call this record my own-goal ledger. In football, an own goal is one a team scores against itself. Here it means an error that my own work introduced, as opposed to an error I found in someone else's published work. A ledger is just a running list.

Most of the ledger was compiled from a second record: a dated log of the whole project, which is where each error and its catch were first written down. Every time something real happens, such as a result, a decision, or an attempt that failed, I add one line to the log the same day. Lines are never edited. If a line turns out to be wrong, I add a new line with the correction, so the log always shows what was believed at the time.

Two entries in the ledger changed how all the work here is done.

The first was a false alarm that looked like a disproof. In July I was testing a conjecture of Alavi, Malde, Schwenk and Erdős about trees, which are sets of dots joined by lines with no loops. For each tree you can write down a list of counts, and the conjecture says that list always rises and then falls, with no second peak. A program built every tree of one size and tested each one. For any tree that broke the pattern, it would print a line starting with the word VIOLATION. A simple text search watched the output for that word.

The search found the word. A real match would have meant a tree that breaks the conjecture, which would disprove an open problem of Erdős. For 20 minutes that is what it looked like. Then the match turned out to be inside the program's own summary line at the end of the run, the line reporting that no tree had broken the pattern. The mathematics was fine. The error was in the text search, a tool. Since then, automated searches in my work match the warning word only at the start of a line, where a real report puts it, so a summary line that mentions the word no longer counts. The conjecture has since been checked on every tree up to a new size limit, and it held; that check is on its own page.

The second was two programs that agreed with each other and were both wrong. Some results here count shapes made of hexagons joined edge to edge. Before I report a count, two separately written programs have to give the same number. In August two such programs agreed, and both were short of the true count for the larger shapes. They agreed because both used the same step to identify each shape by its corner points, and that step loses information for larger shapes, so two different shapes could look identical and be counted once. AI agents in my project had written both programs, and the error was in the step they shared.

What showed the error was a table in the OEIS, the online database of integer sequences, computed by other people without either of my programs. My counts for the larger shapes fell short of it, so they were thrown out before any of them was sent to the OEIS. Since then, every count here has to match an outside source wherever one exists, in addition to my two programs matching each other.

Over time the ledger shows which checks actually catch things. In the paper, no single kind of check caught a majority of the errors.

The dated log addresses a memory problem. Without a same-day record that is only ever appended to, a long project's understanding of itself drifts toward whatever its latest summary says, and summaries tend to drop the caveats that mattered. The error record addresses a different problem. "We check everything" is a claim about a process, and it can only be tested if the checks' findings are recorded as carefully as the results. An error enters the record only if the primary record documents both the error and its catch, so errors that no check caught are absent by construction.

Two entries changed the standing rules. In July a text search for the word VIOLATION, run over the output of an exhaustive tree search for a counterexample to the Alavi, Malde, Schwenk and Erdős unimodality conjecture, matched the run's own all-clear summary line; for 20 minutes it read as a counterexample. Detectors are now anchored to a marker that normal output cannot contain. In August two separately written polyhex enumerators agreed with each other below the published OEIS values at the larger sizes, because both identified shapes through the same vertex-set canonicalization step, which loses cycles for larger shapes. The per-term comparison with the published sequence caught it before any wrong term was staged. Agreement between implementations is no longer accepted on its own wherever an external value exists.

Two more entries are errors in descriptions rather than in results, which a pipeline that checks only computed values would not detect.

In mid-August the lead agent and a helper agent explained that same polyhex shortfall as a failure at "pinched" boundaries, where the outline of a shape touches itself at a single corner. On the hexagonal lattice, where every corner meets exactly three edges, such boundaries cannot occur. The correct cause, the canonicalization step losing cycles, had been written in the project's own verification notes in July, and the August session had not read them. In early September a helper agent ran a full edge-level enumeration that reproduced the published values exactly, which the pinched-boundary explanation did not allow, and then found the July note. The wrong explanation had reached internal status pages only. The rule I now follow: an explanation of a failure is tested like a result. If a method without the suspected step still reproduces the trusted values, that step was not the cause.

In September the lead agent wrote, in a project file, that a folder was excluded from version control, and only then ran the command that checks this. The same command printed a warning that the folder was in fact tracked, and the line was corrected in the next command. The rule now: a sentence that depends on a tool's output is written only after the output exists.

Neither of these two affected the mathematics. The numbers were right; the descriptions of them needed their own check. The paper reads the full record of 58 caught errors as data and asks where along the way, from a working file to a submission, each one was stopped.