The Test That Failed Correctly

The most useful thing my new browser tests did on their first run was fail. Two of three went red: one because part of my homepage is invisible until you scroll, and one because the runner refused to guess which of three post cards I meant by “the first.” Neither was a bug in the site. Both were places where a model-judged test could have quietly passed, and didn’t, because the judge, Cloudflare’s Clef decision model, returns a probability against a threshold. No paragraph to interpret. A number below a line, and a red X.

That’s the case for putting a decision model, not a chat model, behind a test. It isn’t that the model is smarter. It’s that the verdict is a probability someone set a threshold on, so the failure lands where code can read it.

Two days ago, in The Pull Request Is the Ticket, I said the weak link in our plain-English QA check at work was the judge, and that the next experiment was to swap it for a decision model. This is that experiment, on my own site first, where breaking things costs nobody but me.

Three tests, one folder, a third of a cent

The setup is Sedum, an open-source test runner built on Playwright that takes steps in plain English, with Clef on Workers AI as the model. The tests are YAML files with one sentence per step:

description: Contact page reveals an email address after JavaScript runs
url: /contact
steps:
  - verify the page shows an email address for contacting Kevin

Three tests cover the homepage (name, navigation, latest posts), a click path from Writing to a post, and the contact page. The contact test earns its place: earlier today I moved my email address out of the HTML and into JavaScript to keep scrapers off it, so the only proof it still renders is a browser that runs the script.

I picked Clef over TypeSafe’s Jev, which is Sedum’s default, because the site already runs on Cloudflare Workers. One account, one token with two permissions (Workers AI Read and Edit), no new vendor.

Everything lives in an e2e/ folder with its own package.json. That’s the infrastructure habit from my day job: a test harness that ships inside the production build is a test harness that will one day break the production build. Cloudflare’s build installs the site’s dependencies and never sees Playwright.

The workflow runs only when I start it. A full run of all three tests made 9 model calls and about 12,000 input tokens, which Sedum’s own cost meter put at $0.0029 on Clef and $0.0011 on Clef-flash, the smaller model. Both passed everything.

Two failures were my tests, not my site

The first local run passed one test of three, and both failures were about how I wrote the tests.

The homepage test said the Latest Writing section lists recent posts. Clef scored that 0.12, and it was right. My homepage fades sections in as you scroll (data-reveal, a bit of polish I’d forgotten about), and Sedum judges only visible text. To a browser that hasn’t scrolled, the section is not there. Two scroll down steps fixed it, and the claim scored 0.96.

The navigation test said “click the first blog post title.” Sedum refused to click. The three newest post cards scored 0.19, 0.19 and 0.13 as the target, nothing cleared the bar, and the run stopped with ambiguous: several elements match it; name the one you mean more precisely. Clef sees candidate elements as text, not positions, so “first” means nothing to it. I named an older post instead, one that will always be on the list.

A step that refuses to guess is the behavior I want from anything that gates a merge. A click that lands on the wrong card and passes anyway is worse than no test.

A false claim has to fail too

A test suite that passes is evidence of nothing until you’ve watched it fail on purpose; verification is the bottleneck, and a check nobody has seen fail hasn’t been verified either. So I added a fourth test that claimed the contact page “shows a pricing table with monthly subscription plans.” It scored 0.0107 and failed.

The real claims scored between 0.88 and 0.97. That gap, from almost nothing to almost certain, is what I’d want to see before trusting the threshold. Three tests is not a calibration study. It is enough to know the judge isn’t stamping everything green. In the chart below, the gold bar is the one to look at: the same true claim, scored before the section had scrolled into view.

What I got wrong two days ago

In the last post I wrote that Clef accepts images, “which matters for QA: a screenshot” could be the input. That’s true of Clef. It isn’t true of Clef inside Sedum. The provider docs say Clef “is text-only in this release,” so screenshots and the vision fallback are off, and every test has to be judgeable from visible text. That constraint shaped two of the fixes above.

I also said I’d run Jev and Clef side by side. I ran Clef and Clef-flash side by side. The Jev comparison is still owed.

The case for not doing this

Playwright’s ordinary assertions are free, exact and fast. expect(page.getByText("Kevin P. Davison")).toBeVisible() will never return 0.88. For three pages on a static site, a shell script with curl and grep would catch most of what these tests catch, minus the JavaScript.

The model adds failure modes that have nothing to do with the site. Clef’s free allocation is 10,000 neurons a day, shared with everything else on the account, and Sedum stops the run when a provider rate-limits it. Sedum itself is pre-alpha, version 0.1.0-alpha.18, and says its format may change before 1.0. I pinned the exact version for that reason.

And a probability is not a truth. Clef’s 0.88 on the contact page is calibrated against Clef’s training, not against my page. It’s a better signal than a paragraph, because it can be compared to a line. It is still a model’s opinion.

Where the case falls short is authorship. The person writing verify the page shows an email address for contacting Kevin doesn’t need to know a selector, a role or a locator strategy. At work, that person is QA, and the sentence goes in the pull request body where the reviewer can read it.

Fail closed, then argue

Every failure today was correct: a section I forgot was hidden, and a sentence that was ambiguous. Neither showed up as a pass. That is the property I was after: when the judge can’t see what it was asked about, the answer is a low number and a stopped pipeline, not a confident paragraph. Next is the same three sentences on Jev, and then the same idea in the pull request at work, where the reader of the verdict is a merge button.

Resources

• Sedum, documentation, Clef provider (text-only scope, rates, free allocation), Running in CI, and sedum-dev/sedum.
• Cloudflare, Clef launch post and changelog, October 1, 2026 (vendor).
• TypeSafe AI, “Introducing System One Models & Jev”, September 15, 2026 (vendor).
• Playwright, Assertions.
• Scores, token counts and costs: Sedum run reports from my local runs and GitHub Actions runs, October 16, 2026.