YAU.POC/yau-plant-assistant
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Stop quoting the eval-set size, and fix two counts that were wrong

Browse source 
Five documents named the size of eval/testset.jsonl and two were wrong:
spec/REBUILD.md said 62 cases and status/current-state.html said 75. The
file holds 78. Both were corrected earlier today in the two places I had
grepped for, and these two were missed - the same staleness failure
repeating inside the same session that found it.

Rather than correct them to 78 and wait for the next divergence, the
count is gone from every sentence that states a current total. README.md,
REBUILD.md, current-state.html and workflow-map.html now name the eval set
without sizing it. The file is the source of truth and it cannot go stale.

The four remaining mentions are historical and stay: "the exam grew from
67 questions to 75 to cover them" records what happened when the eight
live-model failures were pinned, and REQUESTS.md's "67-question acceptance
test" is inside a dated request. Both are correct as history.

This is the same reasoning as dropping "twelve assumptions" from the
stand-in historian pointer, and it is now a rule in CLAUDE.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude 2026-09-01 21:37:58 +10:00
parent b08fe60772
commit 7d6cc8985c
4 changed files with 5 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -239,7 +239,7 @@ api/ FastAPI, classifier, agent, contracts, guardrails
ingest/ Docling → chunk → embed → pg-ai, plus the
Phase 9 upload worker
web/ React + Vite operator UI
eval/ 78-case test set and the scorecard runner
eval/ the eval set and the scorecard runner
scripts/ deploy.sh, verify.sh
```

2
spec/REBUILD.md
View file

@ -181,7 +181,7 @@ build loads on a control-room PC and then fails every question on DNS. Only
python eval/run_eval.py --api https://api.yokogawa.tech
```
62 engineer-reviewable cases in [`eval/testset.jsonl`](../eval/testset.jsonl), every
Every case in [`eval/testset.jsonl`](../eval/testset.jsonl) is engineer-reviewable, and every
data-dependent one with a **pinned time window** — `imh` is live, and an
unpinned question gives a different answer each run.

2
status/current-state.html
View file

@ -600,7 +600,7 @@ footer { margin-top: 72px; padding-top: 22px; border-top: 1px solid var(--hair);
<span class="pn">08</span>
<div class="pbody"><h4>The exam</h4></div>
<span class="chip c-red"><i class="lamp l-red"></i>Never run</span>
<div class="pnote">75 engineer-reviewable questions, every data-dependent one with a pinned time
<div class="pnote">Engineer-reviewable questions, every data-dependent one with a pinned time
window. Nothing blocks running it &mdash; the model account arrived on 27 August &mdash; and it
has never been run as a gate. No scorecard exists. Two cases fail by design until the historian
lands.</div>

4
status/workflow-map.html
View file

@ -651,7 +651,7 @@ shows the control-room machine opening the page on 28 August, the day the rule w
three questions on 31 August, each answered.</td></tr>
<tr><td class="ph">8</td>
<td><b>The exam</b> &mdash; 78 engineer-checked questions run end to end and scored</td>
<td><b>The exam</b> &mdash; every engineer-checked question run end to end and scored</td>
<td><span class="badge b-blocked">Runnable, and never run</span></td>
<td><b>Nothing is blocking this and it has still not been done.</b> The model account arrived on 27
August, so both AI steps can be scored; no scorecard has ever been produced. The set covers 28
@ -714,7 +714,7 @@ returns that document&rsquo;s heading, purpose and prerequisites and withholds t
The rulebook already refused instruction language and still does; this removes the temptation
rather than relying on catching it.</li>
<li><b>What it still does not prove:</b> whether the labelling and wording are <em>good enough</em>.
They work; they are not yet scored. The 78-question exam can now be run and has not been. And no
They work; they are not yet scored. The exam can now be run and has not been. And no
operator has yet sat at a control-room PC and got an answer &mdash; the one check that cannot be
done from here.</li>
<li><b>Found today, and worth repeating:</b> the most obvious question at this station &mdash;