Claude 885d8e31e2 Add NO_LLM_STUB: the whole chain, working, without a model

Azure OpenAI is pending and imh is pending, so POST /ask could not return
anything at all - which left the entire chain either side of the model
unproven: the browser, the API, entity resolution, Cube, retrieval, the
contracts, the banners, the error paths. All of it is testable now, and waiting
for a key to find out whether it works is a choice to find out later.

NO_LLM_STUB=true substitutes the two steps that need a model and nothing else.

  - Classification: the caller supplies the class, from a dropdown in the UI.
    NOT a keyword classifier. A crude keyword classifier produces a PLAUSIBLE
    label, and a plausible wrong label is the exact failure this system exists
    to prevent - "how do I reset it" landing in Historical is how a synthesised
    procedure reaches an operator. Choosing by hand is honest about what is
    happening and drives each branch deliberately. apply_safety_rules() still
    runs over the result.

  - Prose: a fixed placeholder per class, in stub.py.

Everything else is the real path. This is possible because generate() already
kept the factual fields away from the model: rows, counts, citations, the
fixture flag and the class are attached from evidence, and only prose comes
from the generator. Splitting that into _generate_prose() and _assemble() makes
the seam explicit - the stub feeds _assemble() exactly as the model does, so
this is a fair test of the assembly path rather than a mock of it.

The contracts are the point. A stub payload goes through enforce_contract()
unchanged, and it FAILED first time on two classes: the "nothing found" wording
did not match the not-found detectors, so Reference and Procedural returned 422
rather than an uncited answer. That is the contract doing its job against text
no model wrote. Retries are pointless on deterministic output, and a 422 is a
real result here, not a stub bug.

Retrieval is lexical (retrieval.lexical_search), because embedding the question
needs the model. Kept beside search() and never called on the normal path, so
nobody reads a trace and mistakes a lexical hit for a semantic one. It matches
what the operator typed, not what they meant.

What it does not prove: whether the classifier would have labelled correctly -
a person did; whether retrieval finds the RIGHT chunk; and nothing about prose.
It also cannot fill prerequisites_verbatim - extracting them with a regex would
be the "synthesised from fragments" failure the Procedural contract forbids, so
the list is empty and the answer says so.

Every answer carries stub_mode: true in the contract, not decorated on by the
UI, and a banner beside the fixture banner. Same reasoning: an answer nobody
generated must not be indistinguishable from one that was.

Also here:
  - demo/ai-docs: three fabricated documents, numbered WRPS-DEMO-00x so header
    extraction is genuinely exercised against a number no real WRPS document
    can have. Their setpoints contradict tags.csv on purpose.
  - VITE_API_BASE build arg, for a tunnelled build before DNS exists. The
    tunnel origin is allowed in CORS only while NO_LLM_STUB is on, so it
    disappears with the flag. Proxying /api through ai-web's nginx would have
    been easier and was rejected: it creates a second route to the API that
    bypasses the api.yokogawa.tech Caddy block, where the Phase 9 publisher
    rule lives.

Verified on lin001 with no Azure key set at all: all five classes return 200
through the real UI in a browser, over an SSH tunnel, with citations from the
demo documents, real Cube numbers, and both banners showing.

Turning it off: NO_LLM_STUB=false in ~/ai/api.env, restart ai-api.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-21 15:24:25 +10:00

1.8 KiB

Raw Blame History

DEMO DOCUMENT — NOT A CONTROLLED DOCUMENT — DO NOT USE ON PLANT

Document number: WRPS-DEMO-001 Title: Temporary Bypass of Pump Motor Protection Interlock Revision: 0 Effective: 2026-01-01 Authorising role: Station Maintenance Supervisor

1. Purpose

This fabricated document exists so that the Procedural branch of the plant assistant has a procedure-shaped thing to locate during a no-LLM demonstration. It describes, in invented terms, the temporary bypass of the motor protection interlock on pump PU-302 at the Waterloo Road Pump Station.

Nothing in this document is a real instruction. It has not been reviewed, it has not been approved, and no hazard assessment sits behind it.

2. Prerequisites

The following must all be confirmed before this fabricated procedure begins.

A permit to work has been issued and is held by the person carrying out the work.
PU-302 has been electrically isolated at MCC-501 and the isolation proven dead.
The duty selection has been transferred so that PU-301 and PU-303 can meet inflow.
The wet well WW-101 level is below the start-duty level and falling.
The Station Maintenance Supervisor has authorised the bypass in writing.

3. Procedure

Confirm every prerequisite in section 2 is satisfied and recorded.
Log in to the station HMI with a maintenance-level account.
Navigate to the PU-302 faceplate and select the protection tab.
Set the motor protection bypass to ENABLED and confirm the prompt.
Record the bypass in the station log with the time and the permit number.
Carry out the authorised work.
Set the motor protection bypass to DISABLED before returning PU-302 to service.

4. Restoration

Restore the duty selection and confirm PU-302 responds to a test start. Close the permit. Record the restoration time in the station log.

1.8 KiB Raw Blame History