The page said the question set had NEVER BEEN RUN. That was wrong twice
over, and the second one only surfaced because the first was challenged.
Wrong the first time: commit 7ea8636 records that "the 67-question set
passed for weeks without catching any of them" - it was run repeatedly
against the stand-in model and passed. That is precisely why it caught
none of the eight failures that appeared minutes after the first real
model call on 27 August.
Wrong the second time, and worse: Langfuse holds 251 traces from
1 September 01:01-02:08 UTC covering 84 distinct questions, and every one
of the 75 distinct questions in the set is among them. Nothing was
skipped. That is the set being run end to end against the live model,
nine hours before the seed was re-keyed - and nobody knew, because
eval/results is git-ignored and no scorecard was kept. A run that leaves
no artefact is indistinguishable from one that never happened, which is
how a page assembled from the host ended up asserting the opposite of
what the host could prove.
So the exam is now amber, not red, and says what is actually wrong with
it as a gate: it scored a system that no longer exists, and its result was
not kept. A run against the current build is planned.
The rest is shape. The opening was three paragraphs of running commentary;
it is now one sentence plus a three-row revision table. The KPI band had
three separate tiles for open work - 1 request, 0 findings, 3 issues - so
a reader saw 0 beside 3 and had to work out they meant different things;
one tile now reads 4 open items, 1 blocking. A new section lists those
four in plain words with whose they are and whether they block anything,
because a number with nothing behind it is not status. "Phase" is gone
from the visible text: nine steps, each ending in a check.
Five container uptimes were removed rather than corrected. "Up 26 h" was
four days stale and how long since a restart changes no decision - the
same reasoning as not quoting counts in prose.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
52a26f1d6e