Flakebound

Problem

The imaging lab’s cell-segmentation notebook produced a figure folder that a second machine could not regenerate. The env.yaml was pinned in name only: conda solved a different build two weeks later, a shared GPU box had leftover packages, and the results directory had no record of which commit produced the PNGs. Two paper drafts almost cited numbers nobody could rerun.

Contribution

I built Flakebound, a small runner that checks out a notebook repo, builds a lockfile-pinned container, executes one notebook headlessly, and writes a provenance manifest beside the figures. The manifest records the git SHA, lockfile hash, command, wall time, and output hashes. I did not build a general workflow engine or a lab portal. The lab’s existing self-hosted GPU runner is the only machine this targets.

Decisions

• Containers instead of conda-in-CI. The lab already had Docker on the GPU box, and the env.yaml history was the thing that had drifted.
• One notebook per job, not a DAG. The failure we actually had was a single figure folder, and a workflow engine would have hidden that.
• The manifest is a single JSON file on a bot branch, not a database, so a reviewer can open it on GitHub without an account in our system.
• I turned down Buildkite. It was another login and a bill. GitHub Actions plus the lab’s existing self-hosted runner was enough.

Tests

A fixture notebook writes a known PNG. The test asserts the output hash and the manifest fields: git SHA, lockfile hash, command, and a non-zero wall time. A second test mutates the lockfile and expects a non-zero exit. A third test opens a socket and expects the run to fail closed, because the job drops network egress. I also ran the lab’s real segmentation notebook twice on the GPU box; the figure hashes matched. Those runs are noted in tests/fixtures/README.md, not only in this writeup.

AI use

I used Cursor’s chat to draft the first GitHub Actions workflow and to explain nbclient’s timeout arguments. That draft did not mount the lab data volume, so I rewrote the job. I wrote the provenance schema, the hash checks, and the egress test myself. I did not paste notebook outputs or figure files into a model. Claude turned my sketch notes into a Mermaid diagram, and I edited the diagram before it landed in the README.

Limitations

One notebook per run, on a single self-hosted GPU machine, so this does not show that the lab can schedule a queue or a second box. The manifest misses weights a cell downloads at runtime. The fixture test bans network egress, and I have not watched anyone else’s lab use the runner. A matching figure hash does not mean the scientific claim in the notebook is true.

Learning

Lockfiles lie when the notebook can still reach the network. The test that mattered was the one that fails closed on egress, not the happy-path hash. I also learned that a manifest nobody opens is just another log. I put the six fields a reviewer needs in the first screen of the Actions summary, and I stopped adding fields that only I would read.