A shift log is continuously updated, which rules out both stores we already
have. Not doc_chunks: that store is built on doc_number + revision +
effective_date and nothing in it is citable until a human confirms the header,
so a record changing every shift would wear out the gate that makes procedure
citations trustworthy. Not the historian: we hold read-only on imh and may not
write to it at all, it is keyed on the CI Server item and a log entry has no
item, and its seven-day retention would expire the one source that could
outlive it.
So a table in pg-ai, read on the historical lane because that lane's contract
already fits it - a time window, rows, and "no records found" when there are
none. Entries are rows, NOT citations: a citation is a controlled document with
a confirmed revision and an operator's note is not one however true it is. The
Citation contract is untouched.
Routed on the question text rather than a classifier label. Adding a sixth
class to the five-way classifier - the most safety-relevant component in the
stack - to reach one demo source would be a poor trade. Case S01 pins the
trigger.
Demo entries are anchored to the start of today in site local time, so a
re-run always lands them inside the rolling seven days and the shift label
always matches the clock. The alarm fixtures are absolute and have gone stale;
this cannot. Future-dated entries are dropped, and the load asserts the window
and the count rather than trusting them.
Eval, 78 -> 80 cases:
S01, S02 the shift log, whole and filtered by equipment
N04 repointed at the environmental discharge licence. The old wording
asked about the shift log, which now exists, so it had stopped
testing "no such source" - a different answer from "no records
found", and the distinction is the point of the case.
H02 window changed to a relative one. It still fails, on a false
positive in _contains_quantity: "the last 3 days" reads as a
fabricated figure, so the correct zero-row answer is rejected.
Left for its own change.
80 cases: 97.5% overall, 100% classification, p95 5257 ms. One contract
violation (H02), so the Phase 8 gate is still not met.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
metrics.run() computed outside_retention but nothing consumed it, so the flag
was dead and the distinction it exists to carry never reached an answer. Both
gather_historical and gather_advisory now pass it, with retention_days, as
EVIDENCE - not as an instruction in a prompt.
Also corrects eval case H29 to record what it actually is: a deliberately
failing case, pinned before the fix per the house convention. It cannot pass
yet for a reason that predates this change - gather_historical always queries
a rolling 7 days and never parses the window the question asks about, so
outside_retention can never be true on that path. Observed today:
"No records were found for June 2026. The data provided is for the window
from 2026-08-24 to 2026-08-31."
Honest, and it states the window, but it leads with "no records were found"
and never says the historian keeps only seven days - so an operator cannot
tell a retention limit from a quiet month. Question-window parsing is the
missing piece and is a separate change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three open Phase 5 findings were one defect: the stand-in was keyed on
CI Server POINT names (PS_STN_WET_WELL_LEVEL) when the historian is keyed on
CI Server ITEM names (AID.WRPS.STN.LEVEL). Modbus carries register numbers,
not names, so those two layers are free to differ - and do. Reconciling
against the register map, as planned, would only have proved the first three
namespaces agreed with each other.
Rebuilt from WRPS/05-scada/modbus, so item names, sample rates, retention and
timestamp semantics come from the machine rather than from a guess.
(a) Level tag does not join. PS_STN_WET_WELL_LEVEL becomes a tag row in its
own right; LIT-101 is marked NOT HISTORISED - a field input on %IW0 that
never reaches SCADA. It was the only seed row carrying two addresses.
public.historian_items holds the item-to-tag mapping, generated by
scripts/gen_historian_items.py and enforced non-empty at generate, at
deploy and at verify.
(b) first_alarm/last_alarm returned UTC. Converted inside the measure, so it
stays in Cube and happens once. Aggregate first, convert after - the other
order picks the wrong row across a DST fall-back. Returned as a formatted
string with a companion site_timezone measure. Storage being UTC is now
confirmed, not assumed: all 49 points carry TIME_ZONE "Date+time GMT" and
every history group CORRECT_DAYLIGHT=0. This answers Phase 4 task 4.
(c) High level alarm filed against the wrong equipment. Both sides were right
about different things; the defect was asserting equipment twice. The
history now carries no equipment column at all - faithful, since CI
Server's section tree stops at the station and three pumps. Equipment is
reached bit -> tag -> equipment via public.alarm_bits.
Alarms are derived, not stored: CI Server's ALARM_HISTORY group is empty
because every item imports with alarming off. Decomposing the alarm word needs
no configuration that does not exist.
Three things the SCADA config changed that were never filed as faults:
- retention is 7 days, not 30. The advisory path was reporting a month of
evidence drawn from a week of data
- the analogue rate is 5 s, not 60. Two measures multiplied sample counts by
a hardcoded 60 - a twelvefold overstatement that read as plausible
- the deadband warning in process_values.yml was wrong and was steering
people away from the correct measure
db/002_fixtures.sql now asserts its own counts at load and cross-checks the
alarm derivation against two independent signals. Those prove the pipeline,
not the plant.
db/README-standin-historian.md documents removal: the seam between generation
and contract, and twelve assumptions about imh that are NOT confirmed. Two of
them fail silently.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four faults, all surfaced within an hour of the first live Azure OpenAI call
on 2026-08-27, all invisible under NO_LLM_STUB because the stub supplied the
very fields that turned out to be missing.
1. The classifier few-shot showed eight replies of {"question_class": ...}
alone. A few-shot reply is a shape the model copies, so it omitted
confidence, which defaulted to 0.0, fell below the 0.7 threshold, and EVERY
non-procedural question downgraded to UNCLEAR. The replies now carry the
complete payload the system prompt asks for. Confidences are varied and the
traps carry alternatives: a constant teaches the model to emit that
constant, and the tie rule in apply_safety_rules only has something to work
with if the runners-up are populated.
2. procedure{} was the one part of the procedural payload not assembled from
evidence, contrary to _assemble's own stated rule. The model returned
effective_date "" - neither a date nor None - so ProceduralAnswer rejected
the answer, the single regeneration failed identically, and every procedural
question returned 422.
3. title and authorising_role came back "" for the same reason: the model was
asked for header fields it had never been shown.
4. documented_limits[].citation arrived as the string "WRPS-DEMO-003, Section
4" where a Citation was required, because the schema hint said only
"documented_limits": [] and told the model nothing about the shape.
procedure_identity now takes no `generated` argument at all: there is no path
by which a model can name a revision an operator does not hold. documented_
limits attaches the real Citation by matching source_file against what was
actually retrieved, and DROPS a limit matching nothing - a limit carries the
authority of the document behind it, and misattributing one is worse than
omitting it.
Both live in contracts.py rather than agent.py because they are contract
rules, and because agent.py imports langgraph, which would make the test suite
unrunnable on a bare checkout.
The prompt also now separates two things it was conflating: retrieval
returning nothing (say so and stop) from retrieval returning a document marked
draft, demo or superseded (identify it, quote it, and state the marking).
Including the header chunk made the model read "NOT A CONTROLLED DOCUMENT" and
answer "no controlled procedure was retrieved" while citing one. The marking is
information the operator needs, not a reason to withhold what was found.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Azure OpenAI is pending and imh is pending, so POST /ask could not return
anything at all - which left the entire chain either side of the model
unproven: the browser, the API, entity resolution, Cube, retrieval, the
contracts, the banners, the error paths. All of it is testable now, and waiting
for a key to find out whether it works is a choice to find out later.
NO_LLM_STUB=true substitutes the two steps that need a model and nothing else.
- Classification: the caller supplies the class, from a dropdown in the UI.
NOT a keyword classifier. A crude keyword classifier produces a PLAUSIBLE
label, and a plausible wrong label is the exact failure this system exists
to prevent - "how do I reset it" landing in Historical is how a synthesised
procedure reaches an operator. Choosing by hand is honest about what is
happening and drives each branch deliberately. apply_safety_rules() still
runs over the result.
- Prose: a fixed placeholder per class, in stub.py.
Everything else is the real path. This is possible because generate() already
kept the factual fields away from the model: rows, counts, citations, the
fixture flag and the class are attached from evidence, and only prose comes
from the generator. Splitting that into _generate_prose() and _assemble() makes
the seam explicit - the stub feeds _assemble() exactly as the model does, so
this is a fair test of the assembly path rather than a mock of it.
The contracts are the point. A stub payload goes through enforce_contract()
unchanged, and it FAILED first time on two classes: the "nothing found" wording
did not match the not-found detectors, so Reference and Procedural returned 422
rather than an uncited answer. That is the contract doing its job against text
no model wrote. Retries are pointless on deterministic output, and a 422 is a
real result here, not a stub bug.
Retrieval is lexical (retrieval.lexical_search), because embedding the question
needs the model. Kept beside search() and never called on the normal path, so
nobody reads a trace and mistakes a lexical hit for a semantic one. It matches
what the operator typed, not what they meant.
What it does not prove: whether the classifier would have labelled correctly -
a person did; whether retrieval finds the RIGHT chunk; and nothing about prose.
It also cannot fill prerequisites_verbatim - extracting them with a regex would
be the "synthesised from fragments" failure the Procedural contract forbids, so
the list is empty and the answer says so.
Every answer carries stub_mode: true in the contract, not decorated on by the
UI, and a banner beside the fixture banner. Same reasoning: an answer nobody
generated must not be indistinguishable from one that was.
Also here:
- demo/ai-docs: three fabricated documents, numbered WRPS-DEMO-00x so header
extraction is genuinely exercised against a number no real WRPS document
can have. Their setpoints contradict tags.csv on purpose.
- VITE_API_BASE build arg, for a tunnelled build before DNS exists. The
tunnel origin is allowed in CORS only while NO_LLM_STUB is on, so it
disappears with the flag. Proxying /api through ai-web's nginx would have
been easier and was rejected: it creates a second route to the API that
bypasses the api.yokogawa.tech Caddy block, where the Phase 9 publisher
rule lives.
Verified on lin001 with no Azure key set at all: all five classes return 200
through the real UI in a browser, over an SSH tunnel, with citations from the
demo documents, real Cube numbers, and both banners showing.
Turning it off: NO_LLM_STUB=false in ~/ai/api.env, restart ai-api.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Build spec and host brief carried in from C:\Claude and WRPS/02-env; the
plant model (equipment, tags, alarm bitmask, enums, unit conversions) is
derived from WRPS/04-plc/register-map.csv, WRPS/05-scada/modbus/scada-points.csv
and WRPS-CTL-003.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude