Two retrieval faults, both only visible once a real model ran.
find_procedure ranked a procedure's chunks by similarity to the question. For
"how do I lift the interlock on Pump 02" the closest chunks ARE the step list -
so the branch whose entire purpose is not reproducing steps was handing the
answer writer nothing but steps, while omitting the header block carrying the
title and the authorising role. The model was being asked for a title it had
never been shown, and returned "".
Once the document is identified, WHICH document it is settles what to send:
the header and the prerequisites, in document order, never the steps.
STEP_SECTION_RE is a second line behind ProceduralAnswer's instruction-language
check, not a replacement for it - the contract still rejects instruction
language whatever arrives here. This removes the temptation rather than relying
on catching it.
Separately, rerank did 0.75 * chunk.similarity where similarity is NULL for a
chunk ingested with --no-embed: `1 - (NULL <=> vec)` is NULL, so it raised
TypeError and 500'd the whole question rather than ranking that chunk last. It
now degrades to the lexical half - an unembedded chunk is still findable, just
not by meaning - and Chunk.similarity is typed honestly as float | None.
find_procedure_lexical takes the same identify-then-expand shape, so the stub
keeps testing the shape it always did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Azure OpenAI is pending and imh is pending, so POST /ask could not return
anything at all - which left the entire chain either side of the model
unproven: the browser, the API, entity resolution, Cube, retrieval, the
contracts, the banners, the error paths. All of it is testable now, and waiting
for a key to find out whether it works is a choice to find out later.
NO_LLM_STUB=true substitutes the two steps that need a model and nothing else.
- Classification: the caller supplies the class, from a dropdown in the UI.
NOT a keyword classifier. A crude keyword classifier produces a PLAUSIBLE
label, and a plausible wrong label is the exact failure this system exists
to prevent - "how do I reset it" landing in Historical is how a synthesised
procedure reaches an operator. Choosing by hand is honest about what is
happening and drives each branch deliberately. apply_safety_rules() still
runs over the result.
- Prose: a fixed placeholder per class, in stub.py.
Everything else is the real path. This is possible because generate() already
kept the factual fields away from the model: rows, counts, citations, the
fixture flag and the class are attached from evidence, and only prose comes
from the generator. Splitting that into _generate_prose() and _assemble() makes
the seam explicit - the stub feeds _assemble() exactly as the model does, so
this is a fair test of the assembly path rather than a mock of it.
The contracts are the point. A stub payload goes through enforce_contract()
unchanged, and it FAILED first time on two classes: the "nothing found" wording
did not match the not-found detectors, so Reference and Procedural returned 422
rather than an uncited answer. That is the contract doing its job against text
no model wrote. Retries are pointless on deterministic output, and a 422 is a
real result here, not a stub bug.
Retrieval is lexical (retrieval.lexical_search), because embedding the question
needs the model. Kept beside search() and never called on the normal path, so
nobody reads a trace and mistakes a lexical hit for a semantic one. It matches
what the operator typed, not what they meant.
What it does not prove: whether the classifier would have labelled correctly -
a person did; whether retrieval finds the RIGHT chunk; and nothing about prose.
It also cannot fill prerequisites_verbatim - extracting them with a regex would
be the "synthesised from fragments" failure the Procedural contract forbids, so
the list is empty and the answer says so.
Every answer carries stub_mode: true in the contract, not decorated on by the
UI, and a banner beside the fixture banner. Same reasoning: an answer nobody
generated must not be indistinguishable from one that was.
Also here:
- demo/ai-docs: three fabricated documents, numbered WRPS-DEMO-00x so header
extraction is genuinely exercised against a number no real WRPS document
can have. Their setpoints contradict tags.csv on purpose.
- VITE_API_BASE build arg, for a tunnelled build before DNS exists. The
tunnel origin is allowed in CORS only while NO_LLM_STUB is on, so it
disappears with the flag. Proxying /api through ai-web's nginx would have
been easier and was rejected: it creates a second route to the API that
bypasses the api.yokogawa.tech Caddy block, where the Phase 9 publisher
rule lives.
Verified on lin001 with no Azure key set at all: all five classes return 200
through the real UI in a browser, over an SSH tunnel, with citations from the
demo documents, real Cube numbers, and both banners showing.
Turning it off: NO_LLM_STUB=false in ~/ai/api.env, restart ai-api.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Build spec and host brief carried in from C:\Claude and WRPS/02-env; the
plant model (equipment, tags, alarm bitmask, enums, unit conversions) is
derived from WRPS/04-plc/register-map.csv, WRPS/05-scada/modbus/scada-points.csv
and WRPS-CTL-003.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
