A shift log is continuously updated, which rules out both stores we already
have. Not doc_chunks: that store is built on doc_number + revision +
effective_date and nothing in it is citable until a human confirms the header,
so a record changing every shift would wear out the gate that makes procedure
citations trustworthy. Not the historian: we hold read-only on imh and may not
write to it at all, it is keyed on the CI Server item and a log entry has no
item, and its seven-day retention would expire the one source that could
outlive it.
So a table in pg-ai, read on the historical lane because that lane's contract
already fits it - a time window, rows, and "no records found" when there are
none. Entries are rows, NOT citations: a citation is a controlled document with
a confirmed revision and an operator's note is not one however true it is. The
Citation contract is untouched.
Routed on the question text rather than a classifier label. Adding a sixth
class to the five-way classifier - the most safety-relevant component in the
stack - to reach one demo source would be a poor trade. Case S01 pins the
trigger.
Demo entries are anchored to the start of today in site local time, so a
re-run always lands them inside the rolling seven days and the shift label
always matches the clock. The alarm fixtures are absolute and have gone stale;
this cannot. Future-dated entries are dropped, and the load asserts the window
and the count rather than trusting them.
Eval, 78 -> 80 cases:
S01, S02 the shift log, whole and filtered by equipment
N04 repointed at the environmental discharge licence. The old wording
asked about the shift log, which now exists, so it had stopped
testing "no such source" - a different answer from "no records
found", and the distinction is the point of the case.
H02 window changed to a relative one. It still fails, on a false
positive in _contains_quantity: "the last 3 days" reads as a
fabricated figure, so the correct zero-row answer is rejected.
Left for its own change.
80 cases: 97.5% overall, 100% classification, p95 5257 ms. One contract
violation (H02), so the Phase 8 gate is still not met.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The query window is a rolling seven days by design, and stays that way against
the real SQL historian - it is not parsed out of the question. Two things I
wrote assumed otherwise.
H29 was wrong in two ways. It demanded the words "retention" and "seven days",
and it BANNED "no records found" - which contradicts CLAUDE.md, where that is
the required wording for zero rows. With a fixed one-week window June genuinely
has zero rows in what was queried, so the phrase is correct rather than evasive.
It also described the missing piece as question-window parsing, which is not a
gap but the design.
Rewritten to pin the risk that actually exists: SUBSTITUTION. A question naming
June must never be answered with this week's figure wearing June's label.
Answering "there were 14" would be exactly that, and nothing downstream could
catch it. The observed answer refuses the substitution and states the window it
used, so the case now passes on its merits rather than being red by default.
must_not is the only half run_eval enforces, so the banned phrases are ones
that appear only on a substitution.
outside_retention is reframed as what it is: a GUARD, always false while every
caller asks for seven days, there so a caller who later passes a longer window
cannot get an empty result that reads as "nothing happened".
REQUESTS.md and the removal guide now say that extending retention on its own
is inert - the window would still ask for a week of a longer history. Both
halves are needed, or neither.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
metrics.run() computed outside_retention but nothing consumed it, so the flag
was dead and the distinction it exists to carry never reached an answer. Both
gather_historical and gather_advisory now pass it, with retention_days, as
EVIDENCE - not as an instruction in a prompt.
Also corrects eval case H29 to record what it actually is: a deliberately
failing case, pinned before the fix per the house convention. It cannot pass
yet for a reason that predates this change - gather_historical always queries
a rolling 7 days and never parses the window the question asks about, so
outside_retention can never be true on that path. Observed today:
"No records were found for June 2026. The data provided is for the window
from 2026-08-24 to 2026-08-31."
Honest, and it states the window, but it leads with "no records were found"
and never says the historian keeps only seven days - so an operator cannot
tell a retention limit from a quiet month. Question-window parsing is the
missing piece and is a separate change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three open Phase 5 findings were one defect: the stand-in was keyed on
CI Server POINT names (PS_STN_WET_WELL_LEVEL) when the historian is keyed on
CI Server ITEM names (AID.WRPS.STN.LEVEL). Modbus carries register numbers,
not names, so those two layers are free to differ - and do. Reconciling
against the register map, as planned, would only have proved the first three
namespaces agreed with each other.
Rebuilt from WRPS/05-scada/modbus, so item names, sample rates, retention and
timestamp semantics come from the machine rather than from a guess.
(a) Level tag does not join. PS_STN_WET_WELL_LEVEL becomes a tag row in its
own right; LIT-101 is marked NOT HISTORISED - a field input on %IW0 that
never reaches SCADA. It was the only seed row carrying two addresses.
public.historian_items holds the item-to-tag mapping, generated by
scripts/gen_historian_items.py and enforced non-empty at generate, at
deploy and at verify.
(b) first_alarm/last_alarm returned UTC. Converted inside the measure, so it
stays in Cube and happens once. Aggregate first, convert after - the other
order picks the wrong row across a DST fall-back. Returned as a formatted
string with a companion site_timezone measure. Storage being UTC is now
confirmed, not assumed: all 49 points carry TIME_ZONE "Date+time GMT" and
every history group CORRECT_DAYLIGHT=0. This answers Phase 4 task 4.
(c) High level alarm filed against the wrong equipment. Both sides were right
about different things; the defect was asserting equipment twice. The
history now carries no equipment column at all - faithful, since CI
Server's section tree stops at the station and three pumps. Equipment is
reached bit -> tag -> equipment via public.alarm_bits.
Alarms are derived, not stored: CI Server's ALARM_HISTORY group is empty
because every item imports with alarming off. Decomposing the alarm word needs
no configuration that does not exist.
Three things the SCADA config changed that were never filed as faults:
- retention is 7 days, not 30. The advisory path was reporting a month of
evidence drawn from a week of data
- the analogue rate is 5 s, not 60. Two measures multiplied sample counts by
a hardcoded 60 - a twelvefold overstatement that read as plausible
- the deadband warning in process_values.yml was wrong and was steering
people away from the correct measure
db/002_fixtures.sql now asserts its own counts at load and cross-checks the
alarm derivation against two independent signals. Those prove the pipeline,
not the plant.
db/README-standin-historian.md documents removal: the seam between generation
and contract, and twelve assumptions about imh that are NOT confirmed. Two of
them fail silently.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
None of these were reachable by the tests as they stood, and all three were
silent - the screen looked correct in every case. An 8-page control philosophy
found all of them in one upload.
1. THE WHOLE DOCUMENT BECAME ONE CHUNK. pypdf emits one line per line of the
PDF and no blank lines at all: 416 lines, none blank. Section splitting looks
for Markdown headings and paragraph splitting looks for blank lines, so the
chunker was a no-op on PDF text - one 18,307-character chunk, a single
embedding vector for eight pages, and every citation reading "(untitled),
page 1". A longer document would have exceeded the embedding model's input
limit and failed to publish at all.
convert.py now recovers structure: headings from numbered and capitalised
lines, paragraphs by reflowing on line width. Heading detection is
deliberately narrow, because the dangerous direction is promoting a numbered
STEP to a heading and splitting a step sequence - so a heading must be short,
a few words, and without terminal punctuation. "1. Purpose" qualifies;
"1. Open the isolation valve and confirm zero pressure." does not.
chunking.py gains a ceiling no chunk may exceed whatever the input looks
like, falling back to line and then word boundaries. The step-sequence
refusal still holds below it and is unchanged for any realistic procedure;
past it, splitting is the lesser harm, because an embeddings call that fails
protects nobody. Two heuristics found only by running the real file:
"SCADA" and "WRPS-PRO-001" were being promoted to headings, which cut real
sections in half and re-titled the remainder with something meaningless, and
"11 August 2026" was parsing as section 11.
19 chunks now, largest 574 tokens, sections matching the document.
2. EVERY CHUNK CARRIED doc_title = "Revision". TITLE_RE used [\s:]+ for the gap
after the label, and \s includes the newline. A cover page flattens to a
label column then a value column - Title / Revision / Date - so it matched a
bare "Title" line, consumed the line break and captured the next line. Now
[ \t:]+, the same trap AUTHORISING_ROLE_RE was fixed for once already. The
document's title is now null, which is the honest answer: a citation falls
back to the section title, and a confidently wrong title falls back to
nothing. Inherited, so fixed in ingest.py too.
3. RE-PUBLISHING A DOCUMENT DUPLICATED IT. approve deleted prior chunks by
source_file, which carries the upload_id and is new on every upload -
so approving the same revision twice left 38 live chunks and the same
passage citable twice. Invisible on screen, because live_documents groups by
(doc_number, revision) and only the count moved. Now deletes by document and
revision as well, and logs how many chunks it replaced.
The two chunkers are now provably in step rather than asked to be. The header
of chunking.py claimed drift in ingest.py could not be detected from the test
suite; that was wrong, both files are on disk. The new test compares the source
of chunk_section, _split_on_lines, _split_on_words, extract_header and
approx_tokens character for character. Writing it found that one earlier edit to
ingest.py had silently not applied, leaving the two genuinely divergent, and
then that extract_header's docstring had drifted. Both fixed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 9's operator path, built ahead of Phase 8 at the customer's direction and
live at api.yokogawa.tech/documents. Upload, convert, review, approve, withdraw
and restore. The pool screen is explicitly out of scope.
Served by ai-api rather than ai-web, and mounted at /documents rather than
/docs. ai.yokogawa.tech is SCADA-only since 2026-08-28 and passes through no
Authelia, so it has no identity to record; publishers arrive on
api.yokogawa.tech where the forward-auth headers still do. /docs stays with
Swagger, which the customer is keeping - two things under one prefix with two
different access policies is what gets misread during a later edit.
Conversion is text extraction, not document parsing: pypdf, python-docx and
openpyxl. Docling would be better at this and pulls torch, which lin001 has
neither the memory to install nor the business running next to the demo plant's
PLC. The cost is real - no layout, no table structure, and a scan cannot be read
at all, so it is refused rather than stored empty. It is acceptable only because
the converted text is shown to a person before the document can be cited, which
is the same safety net the design already required for the header. convert.py is
the one file to change if that stops being true.
Chunking is mirrored from ingest.py rather than shared, because the two live in
different images. They must stay identical: if they drift, the same document
chunks differently depending on who loaded it, and the assistant answers or
fails to answer depending on that. The step-sequence rule is locked by a test.
Identity is self-asserted for the demo - the actor is typed on the form, which
section 16 forbids, and the publisher list is one name with no password. Rows are
written as `demo:<name>` with actor_groups = 'DEMO-UNVERIFIED' so that when real
auth goes on, a name somebody typed stays tellable from a name Authelia proved.
doc_actions cannot be deleted from, so an ambiguity there would be permanent.
Two rules the code enforces rather than documents: uploading is open to anyone
who reaches the page, because uploading changes nothing an operator can see -
approving does, and that is what is gated; and an empty publisher list means
nobody, not everybody.
Verified on the host end to end: withdraw as a non-publisher 403s, with a short
reason 400s, and as admin flips 5 chunks and writes a complete audit row;
restore puts them back and keeps both rows. The corpus is unchanged afterwards.
Requirements are split so the document dependencies install in their own layer -
a change there costs four small wheels instead of re-resolving fastapi,
langgraph and langfuse on a 2 vCPU shared host.
The five divergences from section 16 are recorded in section 14. The one with
teeth: files published through the UI stay in the inbox, so `ai-ingest --all`
cannot see them and the two paths must not be used on the same document.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four faults, all surfaced within an hour of the first live Azure OpenAI call
on 2026-08-27, all invisible under NO_LLM_STUB because the stub supplied the
very fields that turned out to be missing.
1. The classifier few-shot showed eight replies of {"question_class": ...}
alone. A few-shot reply is a shape the model copies, so it omitted
confidence, which defaulted to 0.0, fell below the 0.7 threshold, and EVERY
non-procedural question downgraded to UNCLEAR. The replies now carry the
complete payload the system prompt asks for. Confidences are varied and the
traps carry alternatives: a constant teaches the model to emit that
constant, and the tie rule in apply_safety_rules only has something to work
with if the runners-up are populated.
2. procedure{} was the one part of the procedural payload not assembled from
evidence, contrary to _assemble's own stated rule. The model returned
effective_date "" - neither a date nor None - so ProceduralAnswer rejected
the answer, the single regeneration failed identically, and every procedural
question returned 422.
3. title and authorising_role came back "" for the same reason: the model was
asked for header fields it had never been shown.
4. documented_limits[].citation arrived as the string "WRPS-DEMO-003, Section
4" where a Citation was required, because the schema hint said only
"documented_limits": [] and told the model nothing about the shape.
procedure_identity now takes no `generated` argument at all: there is no path
by which a model can name a revision an operator does not hold. documented_
limits attaches the real Citation by matching source_file against what was
actually retrieved, and DROPS a limit matching nothing - a limit carries the
authority of the document behind it, and misattributing one is worse than
omitting it.
Both live in contracts.py rather than agent.py because they are contract
rules, and because agent.py imports langgraph, which would make the test suite
unrunnable on a bare checkout.
The prompt also now separates two things it was conflating: retrieval
returning nothing (say so and stop) from retrieval returning a document marked
draft, demo or superseded (identify it, quote it, and state the marking).
Including the header chunk made the model read "NOT A CONTROLLED DOCUMENT" and
answer "no controlled procedure was retrieved" while citing one. The marking is
information the operator needs, not a reason to withhold what was found.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two retrieval faults, both only visible once a real model ran.
find_procedure ranked a procedure's chunks by similarity to the question. For
"how do I lift the interlock on Pump 02" the closest chunks ARE the step list -
so the branch whose entire purpose is not reproducing steps was handing the
answer writer nothing but steps, while omitting the header block carrying the
title and the authorising role. The model was being asked for a title it had
never been shown, and returned "".
Once the document is identified, WHICH document it is settles what to send:
the header and the prerequisites, in document order, never the steps.
STEP_SECTION_RE is a second line behind ProceduralAnswer's instruction-language
check, not a replacement for it - the contract still rejects instruction
language whatever arrives here. This removes the temptation rather than relying
on catching it.
Separately, rerank did 0.75 * chunk.similarity where similarity is NULL for a
chunk ingested with --no-embed: `1 - (NULL <=> vec)` is NULL, so it raised
TypeError and 500'd the whole question rather than ranking that chunk last. It
now degrades to the lexical half - an unembedded chunk is still findable, just
not by meaning - and Chunk.similarity is typed honestly as float | None.
find_procedure_lexical takes the same identify-then-expand shape, so the stub
keeps testing the shape it always did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ProcedureIdentity requires a title and an authorising role, and neither was
stored anywhere. The answer writer was asked for both, read them off whatever
chunk retrieval happened to return, and returned "" whenever the header chunk
was not among them.
They belong in the row for the same reason doc_number and revision do: they
are facts about the controlled document, established once when a human
confirms the header, not something to re-derive per question from whatever
text was retrieved. Denormalised onto every chunk exactly as the existing
header fields are - ingest replaces every chunk of a source_file in one
transaction, so they cannot drift within a document.
complete() deliberately still requires only doc_number, revision and
effective_date. A missing title makes an answer less useful; a wrong revision
sends somebody to the wrong document. --assume-yes must keep refusing on the
second and tolerate the first.
controlled_copy_location is NOT in the schema. It is a site fact, identical on
every row, and the one field where an invented value sends a person to a place
that does not exist. It is CONTROLLED_COPY_LOCATION in api.env, defaulting to
a string that names who to ask.
The authorising-role pattern requires the colon: without it the lazy gap
swallowed the field name and captured "role: Station Maintenance Supervisor"
as the value, which the first run caught.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
api.yokogawa.tech has a public A record but no pinpoint record on the DC, so
it does not resolve from inside the VNet at all. The browser called it by
hostname, which means an operator on cicore1 would have loaded the page and
had every question fail on DNS - the exact gap Phase 7's gate exists to catch,
and one an engineer's laptop cannot see.
Caddy now routes /ask under ai.yokogawa.tech to ai-api, inside a route block
so import authelia still runs first: forward_auth sorts after handle in the
default directive order, and outside a route the handles would be terminal and
the gate would never run. Only /ask is routed - the Phase 9 publisher rule is
scoped to api.yokogawa.tech and a wider route here would leave it inert.
Also fixes the fallback it replaces. The build arg defaults to "", and
`?? "https://api.yokogawa.tech"` does not catch an empty string, so the
documented real-deployment build resolved the API base to "" and posted /ask
at ai-web, which 404s it. verify.sh and deploy.sh now check both the route and
whether the built bundle carries the hostname.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Azure OpenAI is pending and imh is pending, so POST /ask could not return
anything at all - which left the entire chain either side of the model
unproven: the browser, the API, entity resolution, Cube, retrieval, the
contracts, the banners, the error paths. All of it is testable now, and waiting
for a key to find out whether it works is a choice to find out later.
NO_LLM_STUB=true substitutes the two steps that need a model and nothing else.
- Classification: the caller supplies the class, from a dropdown in the UI.
NOT a keyword classifier. A crude keyword classifier produces a PLAUSIBLE
label, and a plausible wrong label is the exact failure this system exists
to prevent - "how do I reset it" landing in Historical is how a synthesised
procedure reaches an operator. Choosing by hand is honest about what is
happening and drives each branch deliberately. apply_safety_rules() still
runs over the result.
- Prose: a fixed placeholder per class, in stub.py.
Everything else is the real path. This is possible because generate() already
kept the factual fields away from the model: rows, counts, citations, the
fixture flag and the class are attached from evidence, and only prose comes
from the generator. Splitting that into _generate_prose() and _assemble() makes
the seam explicit - the stub feeds _assemble() exactly as the model does, so
this is a fair test of the assembly path rather than a mock of it.
The contracts are the point. A stub payload goes through enforce_contract()
unchanged, and it FAILED first time on two classes: the "nothing found" wording
did not match the not-found detectors, so Reference and Procedural returned 422
rather than an uncited answer. That is the contract doing its job against text
no model wrote. Retries are pointless on deterministic output, and a 422 is a
real result here, not a stub bug.
Retrieval is lexical (retrieval.lexical_search), because embedding the question
needs the model. Kept beside search() and never called on the normal path, so
nobody reads a trace and mistakes a lexical hit for a semantic one. It matches
what the operator typed, not what they meant.
What it does not prove: whether the classifier would have labelled correctly -
a person did; whether retrieval finds the RIGHT chunk; and nothing about prose.
It also cannot fill prerequisites_verbatim - extracting them with a regex would
be the "synthesised from fragments" failure the Procedural contract forbids, so
the list is empty and the answer says so.
Every answer carries stub_mode: true in the contract, not decorated on by the
UI, and a banner beside the fixture banner. Same reasoning: an answer nobody
generated must not be indistinguishable from one that was.
Also here:
- demo/ai-docs: three fabricated documents, numbered WRPS-DEMO-00x so header
extraction is genuinely exercised against a number no real WRPS document
can have. Their setpoints contradict tags.csv on purpose.
- VITE_API_BASE build arg, for a tunnelled build before DNS exists. The
tunnel origin is allowed in CORS only while NO_LLM_STUB is on, so it
disappears with the flag. Proxying /api through ai-web's nginx would have
been easier and was rejected: it creates a second route to the API that
bypasses the api.yokogawa.tech Caddy block, where the Phase 9 publisher
rule lives.
Verified on lin001 with no Azure key set at all: all five classes return 200
through the real UI in a browser, over an SSH tunnel, with citations from the
demo documents, real Cube numbers, and both banners showing.
Turning it off: NO_LLM_STUB=false in ~/ai/api.env, restart ai-api.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The UI calls the API with credentials: "include", because ai.yokogawa.tech and
api.yokogawa.tech are different origins and the Authelia session cookie has to
be attached explicitly. The CORS middleware never set allow_credentials, and a
browser refuses a credentialed cross-origin request unless the response says
Access-Control-Allow-Credentials: true. It fails at the preflight, so the real
request is never sent:
Access to fetch at '.../ask' has been blocked by CORS policy: the value of
the 'Access-Control-Allow-Credentials' header in the response is '' which
must be 'true' when the request's credentials mode is 'include'.
Every question from the UI would have failed at Phase 7 with "Could not reach
the assistant" - the app's network-error branch, which says nothing about CORS
and points at the wrong layer entirely. The API is fine; curl against it passes,
because curl is not a browser and does not enforce this.
Found driving the built UI in a browser. It is not reachable by any test that
does not involve a browser, which is the useful part: the Phase 7 gate says an
operator reaches the UI and gets an answer end to end, and that gate is the
first thing that would have caught it - at the point where DNS, Caddy and
Authelia are all new too, and any of them a plausible suspect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every Historical and Advisory answer stated a window in AEST and queried one
shifted by ten hours.
rolling_window() builds its boundary strings in SITE_TIMEZONE - that is the
whole point of it, and its docstring says so. metrics.run() then posted the
query to Cube with no timezone at all, and Cube defaults to UTC. So
"2026-08-14T15:22:13" meant 15:22 Sydney to the code that produced it and 15:22
UTC to the engine that ran it, and MetricResult.time_window reported
SITE_TIMEZONE from config rather than whatever the query actually used, so the
two could not disagree visibly.
Measured on the fixtures, same dateRange, one field changed:
timezone UTC 8019 samples
timezone Australia/Sydney 8619 samples
600 samples. One per minute, ten hours, exactly the offset.
Nothing about the answer looked wrong. The prose was right, the count was a
real count, the window description was correctly formatted and correctly named
AEST. It was only visible by reading the Cube query in the UI's "show working"
panel - which is an argument for that panel existing, and an argument for
looking at the thing in a browser rather than trusting curl against the API.
- check_cube_query() now takes site_timezone and pins it onto the query, at
the single point every Cube query passes through. Per-query-builder is the
wrong place: "remember to set the timezone" is not a control, and this
defect is what forgetting looks like. An explicit timezone already on the
query is left alone.
- time_window now reports capped["timezone"] - the timezone the query ran in,
not the one it should have run in.
An unpinned timezone belongs in the same guardrail as an unpinned date range,
and for the same reason: both make an answer unreproducible. The difference is
that an unpinned date range is obvious in the query and an unpinned timezone
is invisible.
eval case H28 records it. Two unit tests: the timezone is pinned, and an
explicit one is not overridden.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Build spec and host brief carried in from C:\Claude and WRPS/02-env; the
plant model (equipment, tags, alarm bitmask, enums, unit conversions) is
derived from WRPS/04-plc/register-map.csv, WRPS/05-scada/modbus/scada-points.csv
and WRPS-CTL-003.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude