CLAUDE.md requires a failure to be pinned before it is fixed, and names
two places to pin one. Neither covers a shell script, a SQL migration or a
seed CSV: api/tests holds six files and all six are Python-level.
Three defects on 1 September showed what that costs. deploy.sh loaded
historian_items positionally, so a migrated column order put a poll group
into modbus_address - it failed loudly only because a text value hit an
integer column, and two columns of the same type would have swapped
silently. The PS_ rename mapped the high level alarm STATUS BIT onto the
alarm SETPOINT, which would have made "how many high level alarms last
week" count setpoint changes. Neither could be pinned anywhere. The third,
a TypeError in the retired-name guard, had a home in api/tests and still
was not pinned.
The second was caught only because gen_historian_items.py's
check_mapping() refuses to write when an item resolves to nothing. That
check exists because the same ambiguity caused a Phase 5 finding in
August. It has now caught the same class of defect twice and it is the
only guard of its kind in the repository.
The issue records three things worth considering - a test over the seed
CSVs asserting unique keys and resolving foreign keys, migrations applied
to a throwaway Postgres rather than to lin001, and an honest question
about whether deploy.sh is worth testing at all for a demo - without
deciding any of them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five documents named the size of eval/testset.jsonl and two were wrong:
spec/REBUILD.md said 62 cases and status/current-state.html said 75. The
file holds 78. Both were corrected earlier today in the two places I had
grepped for, and these two were missed - the same staleness failure
repeating inside the same session that found it.
Rather than correct them to 78 and wait for the next divergence, the
count is gone from every sentence that states a current total. README.md,
REBUILD.md, current-state.html and workflow-map.html now name the eval set
without sizing it. The file is the source of truth and it cannot go stale.
The four remaining mentions are historical and stay: "the exam grew from
67 questions to 75 to cover them" records what happened when the eight
live-model failures were pinned, and REQUESTS.md's "67-question acceptance
test" is inside a dated request. Both are correct as history.
This is the same reasoning as dropping "twelve assumptions" from the
stand-in historian pointer, and it is now a rule in CLAUDE.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CLAUDE.md was 120 lines and had become a detailed document. Its job is to
be the first thing another team's Claude Code reads: an index that says
where everything is, plus the few rules that must be known before acting.
Detail is better maintained next to the thing it describes, so it moved
there - the four namespaces and the PS_ ban to db/seed/scada-source, the
ai-docs-worker divergence to api/documents.py's docstring, the plant
description to README.md. The file now says so, and tells the next editor
to keep it that way.
Then a sentence-by-sentence review, which did not go the way the premise
suggested. Of seventeen sentences flagged as possibly unnecessary, one was
padding - and it had been introduced ten minutes earlier. The rest were
carrying real information in the wrong shape, or were wrong, or were
hiding a gap:
WRONG. "Details: api/contracts.py" followed the classification
precedence rule, which is implemented in classifier.py. "Prove each
layer before wiring the agent" instructed a build phase that finished
weeks ago.
MISSING. Neither table pointed at retrieval, the schema and migrations,
compose, or the operator's web app - about a third of the repository's
changeable surface, absent from an index whose only job is to point at
it. The retrieval gap surfaced because "the document library" was
ambiguous: it names the SCREENS in documents.py, not the corpus, and the
corpus had no row at all.
CUT BY MISTAKE, in the rewrite this commit also contains. The trigger
for writing to OPEN-ISSUES.md, and the .bak-<purpose>-<date> convention
which is the only concrete form of "snapshot before editing" - the rule
that made running migration 008 on a live database safe.
Two rules are new because the audit kept finding the same failures.
"Never quote a count in prose" generalises three separate staleness
defects - container counts across four documents, the eval-set size across
five, the env-file count. "Pin it before you fix it" now names both
destinations, eval/testset.jsonl and api/tests/, and admits that
deployment and migration defects have neither.
The host's size is recorded for the first time: 2 vCPU and 3 GiB, verified
live. Build elsewhere and ship the image where possible; where not, check
the host is quiet and announce it. deploy.sh builds on the host today,
which is why the rule is a judgement rather than an absolute.
86 lines became 91 during a review meant to shorten it. That is the
finding: the file was not too long, it was pointing at the wrong things.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ContractViolation takes (rule, detail, offending_output=""). The guard
passed a single string, so every retired name raised
TypeError: ContractViolation.__init__() missing 1 required positional
argument: 'detail'
instead of the clean error it was written to produce - a guard that
crashes the request it was meant to explain.
It was not caught before deployment because this machine has neither
fastapi nor psycopg, so the guard could not be imported. It was tested by
lifting its source out of the file and exercising it against a stubbed
ContractViolation - a plain Exception, which accepts one argument. The
stub is what made the test pass. Running it in the built image found it in
one command, which is the argument for running the suite where the
dependencies are rather than reasoning about it where they are not.
Also records what the error actually does: main.py logs the rule, returns
422, and deliberately keeps the detail out of the operator-facing message,
so the explanation lands in the log and in Langfuse rather than on screen.
That is the right trade - an operator asking in plain English never types
one of these names, so the audience is a developer or a stale integration.
Verified in the running container: the retired names and their lowercase
forms are rejected with rule "retired_tag_name"; the poll groups, the item
names, the instrument tags and plain English all pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The seed load broke on lin001 the first time it ran after migration 008:
ERROR: invalid input syntax for type integer: "PS_STATUS_BITS"
CONTEXT: COPY hi_stage, line 2, column modbus_address
hi_stage is created LIKE historian_items, so it inherits the LIVE table's
column order. On a database built fresh from 001_schema.sql that matches
the CSV. On a database migrated by 008 it does not: ALTER TABLE ADD COLUMN
appends, so ci_station, ci_point and poll_group sit at the end of the
table while the CSV has them in the middle. \copy matches by position and
ignores the header, so it loaded the poll group into modbus_address.
It failed loudly here because a text value landed in an integer column.
Two columns of the same type would have loaded silently into each other's
places, which is the version of this defect worth designing against.
Both the \copy and the INSERT now name their columns, so the load no
longer depends on the two orders agreeing. The \copy line is long because
psql meta-commands cannot be wrapped across lines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The seed rename in 3d19b18 changed the repository. It could not change a
database that already exists: 001_schema.sql is CREATE TABLE IF NOT
EXISTS, so it will not add ci_station, ci_point or poll_group to a live
historian_items, and deploy.sh only ever upserts - nothing deletes. A
plain reload would have failed on the missing columns, and had it not, it
would have left 45 retired keys beside their 45 replacements with two
foreign keys still pointing at the old ones.
The migration adds the columns, carries scada_point across into ci_point,
drops it, then re-keys: it captures the old-to-new mapping BEFORE touching
anything, inserts a replacement tags row per retired key, repoints
alarm_bits and historian_items, checks nothing still references a retired
key, and only then deletes.
The mapping is captured first because historian_items.tag_id is itself
about to change - reading the mapping through it afterwards finds nothing,
which would have left alarm_bits pointing at deleted rows. A silently
unmapped alarm bit is how an alarm count becomes wrong, so there is an
explicit check that every alarm_bits key has somewhere to go, and the
migration stops before touching anything if one does not.
Applied to lin001 2026-09-01 after a pg_dump of the three tables to
~/ai/backups/ and a dry run inside a rolled-back transaction. Result: 45
retired keys mapped, 6 alarm_bits rows repointed, 45 tags rows deleted,
65 tags before and after, ci_point populated on all 49 items.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
api/agent.py imports AzureOpenAI from langfuse.openai rather than from
openai. It is a drop-in with identical constructor and call signatures
that records each call as a Langfuse generation carrying the model and
token usage the plain client discards, which is the only thing Langfuse
can price. Without it, cost never appears in the UI.
This was dropped from the repository at the start of the 1 September
audit, on the reasonable-looking grounds that it was an unrelated concern
mixed into an unrelated change. It was not: it is deployed. The running
ai-api image has `from langfuse.openai import AzureOpenAI` at
/app/agent.py line 33 with langfuse 2.57.0 installed, and langfuse==2.57.0
has been in api/requirements.txt all along. The blob restored here is
byte-identical to what the host is running.
Rebuilding ai-api from the repository as it stood would have silently
removed live cost tracking from production. That is the second time in
this audit that the repository turned out not to describe the host, and
the first where acting on the repository would have caused a regression
rather than merely documenting one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
OI-02: the second of the three lines this system does not cross - no
recommended setpoints or operating parameters - is expected to come up as
a feature request. An Advisory answer gives evidence, ranges, outcomes and
documented limits, then defers to a competent person, and never returns a
number as the answer. There is a foreseeable case for allowing it, and the
conversation should start from what is built rather than from scratch.
The entry records where the rule actually lives, because it is a code path
and not a prompt instruction, so relaxing it is a change in four places at
once: AdvisoryAnswer's recommendation_given: Literal[False] and required
deferral in api/contracts.py, the classifier routing anything
partly-advisory to the class that refuses to advise, 14 Advisory cases in
the eval set, and 15 tests. It also records what has to be decided before
any of that is touched - who is accountable for the number, what evidence
is sufficient when the historian shows what happened rather than what the
plant can safely do now, and how a recommendation is told apart from a
documented limit on screen.
It is blocked by the OT/safety review of section 2 of the build spec,
which is still outstanding. That review will have the strongest opinion on
this rule of anything in the document, so the rule should not be relaxed
before it happens.
The register's header said it held work we "intend to do", and its
boundary table admitted only defects. OI-02 is neither: the rule works as
designed. Rather than let the register's rules and its contents disagree -
which is the failure this register exists to prevent - the header now
covers decisions we expect to revisit, and the table has a row for them
requiring the entry to be marked "not a defect".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seven mentions remain across the repository and none of them is data.
Each is a sentence recording that these names were wrong, what it cost,
and that they must not come back: the namespace tables in CLAUDE.md and
the build spec, the generator's docstring, the fixtures rewrite note, the
Phase 5 narrative in REBUILD.md and current-state.html, and the delivered
configuration's own README.
The build spec's namespace table carried the same error CLAUDE.md did -
PS_STN_WET_WELL_LEVEL listed as the CI Server point layer, sourced from a
file that was a PLC-side proposal. Both now name the real point layer,
WRPS_PLC:STN_LEVEL from wrps_modbus_point_df.qli, and both point at
db/seed/scada-source/ rather than at a path on one person's laptop.
The name-collision entry in the build spec is marked closed, and records
that it bit a second time during the rename itself.
These notes explain; they do not enforce. What actually caught the
collision was check_mapping() refusing to write, and what stops a retired
name being queried is the guard in equipment.resolve(). The prose is here
so the next person does not have to re-derive the story - not as a
control.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With the seed rekeyed, a question asking after PS_STN_WET_WELL_LEVEL now
matches nothing - and the assistant answers "no records found". An
operator reads that as "the plant recorded nothing", not as "you asked
with a name this system retired". That is precisely the confusion the four
namespaces exist to prevent, and an empty result is the wrong shape of
answer for it.
equipment.resolve() now rejects those names up front, raising
ContractViolation, which main.py already turns into an error rather than
an answer. The message says what the name was, why it is not a tag, and
what to use instead.
The four Modbus poll groups keep the prefix legitimately - they are
groups, not names, and nothing resolves an operator term onto one - so
PS_STATUS_BITS, PS_PUBLISHED, PS_SETPOINTS and PS_SIM_CONTROL pass.
Scope, deliberately small: this covers equipment.resolve(), the path an
operator's words take. Calling metrics directly with a retired id is not
covered. For a demo that is the right trade; for production it is not.
Verified by lifting the guard's own source out of the file and exercising
it - upper case, lower case, surrounding whitespace, the _SP variant, all
four poll groups, and the item, instrument and plain-English forms. The
module itself could not be imported here: no psycopg on this machine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
45 of the 65 rows in db/seed/tags.csv had a PS_* name as their PRIMARY
KEY, and historian_items.csv, alarm_bits.csv, the fixtures, two Cube
models and the exam all referenced them. They are gone. The tag seed is
now keyed on the CI Server item for everything the historian carries -
AID.WRPS.STN.LEVEL - and on the instrument tag for the 20 field devices
that never reach SCADA.
gen_historian_items.py reads db/seed/scada-source/ by default, so it runs
for anyone with a clone: --wrps is now --source. The tag match is the item
name itself, an identity lookup, and the TAG_FOR_ITEM special case is
deleted - all 49 item names are unique, which the old point names were
not. scada_point in the output is replaced by ci_station, ci_point and
poll_group; 001_schema.sql and deploy.sh's upsert follow.
Regenerated, and it comes out the same shape it went in: 49 items, 45
answerable, 4 deliberately excluded, three groups at 5 s, 30 s and on
change. Every historian_items.tag_id and alarm_bits.tag_id resolves to a
tags.csv row. No duplicate keys.
THE ONE NAME THAT WAS AMBIGUOUS, AND NEARLY COST US
PS_STN_HIGH_LEVEL_ALARM named two different things in the old delivery:
the high level alarm STATUS BIT on coil 10, and the alarm SETPOINT on
holding register 1032. Building the rename map from that file kept
whichever came last, so the status bit was silently renamed onto the
setpoint. check_mapping() refused to write and named the item that no
longer resolved - which is the only reason this is a paragraph in a commit
message rather than a defect.
Had it gone through, alarm bit 0 - wet well high level - would have
pointed at the setpoint. "How many high level alarms last week" would have
counted setpoint changes and returned a small, plausible, confident,
wrong number. That check exists because the same ambiguity caused the
first Phase 5 finding in August. It has now bitten twice.
NOT YET VERIFIED: pytest api/tests could not be run here - this machine
has neither fastapi nor psycopg. The tests are unchanged and reference no
PS_ name, but they have not been run. pg-ai on lin001 still holds the old
keys and must be reloaded, Cube pre-aggregations rebuilt, and the Phase 1
gate re-run. eval/testset.jsonl changed, so the 78-case exam - never yet
run - should be run after.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The point list handed over in August named the points PS_STN_WET_WELL_LEVEL
and the like. Those were never CI Server names. They came from
gen_scada_points.py on the PLC side, which describes its own output as "a
starting point for CI Server configuration, not a CI Server import file",
and the CI Server generator ignored them entirely - it names points from
its own address-keyed table. The names appear in no .qli export, and
nowhere else in the WRPS project at all.
This is the corrected delivery. ci-server-points.csv is keyed on the CI
Server ITEM each point feeds, with the Modbus station and the point name
beside it, so all four namespaces are visible in one row:
LIT-101 instrument tag
%QW0 PLC symbol and address
WRPS_PLC:STN_LEVEL CI Server point - station, colon, point name
AID.WRPS.STN.LEVEL CI Server item - what the historian is keyed on
The 49 rows were joined onto the .qli exports by function code and Modbus
address: 49 matched, 0 unmatched. The item and point names are read from
the exports, not invented here.
The six .qli files come too, unchanged. gen_historian_items.py needs them,
and until now they existed only on one laptop behind a --wrps path - so
the one file the rules say must never be hand-edited could be regenerated
by exactly one person. 84 KB buys the team that back.
These are uncontrolled documents: no revision, no effective date, no
approval. The README says so, says how to ask for a fresh set, and records
the two things that bite - that PS_ survives legitimately as four poll
group names, and that his_group.qli and the live export disagree about the
sample rates, with the live one winning.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
status/OPEN-ISSUES.md is a fourth register, so it opens by saying what
does NOT belong in it. Work blocked on other people stays in REQUESTS.md;
shortcuts we consciously accepted stay in BUILD-AI-CONTAINERS.md 14 and
are closed by decision, not to be re-raised here; what is running stays
in current-state.html. What had no home until now is a defect we own and
have not fixed - those were living in commit messages. Closed issues move
to the bottom rather than being deleted: this repo is an as-built record,
and an issue with no trace of how it closed is worth less than one that
was never raised.
OI-01: the OpenPLC Editor is not installed anywhere. It is a desktop tool
published by OpenPLC, and it is what authors and compiles the IEC 61131-3
program openplc-runtime executes. Without it there is no reviewable
source for the control logic in or beside this repository - the program
exists only inside the container.
Raising it exposed two wrong statements in the build spec, both calling
port 8443 the "OpenPLC Runtime web UI". Probed read-only on the host:
Server: Werkzeug/3.1.8 Python/3.11.2
/ /login /index.html /programs /status /runtime -> 404
/api/v1 -> 401 Unauthorized
It is an authenticated REST API with no browser interface at all. Section
4 now says so with the evidence, and 14's entry is restated: 8443 is
contained by the same 10.0.0.17 binding as 502, but unlike 502 it is not
anonymous, which is a better position than that section recorded. The
binding is still the control that matters.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Counted on lin001: 28 running, none stopped. Seven are ours (ai-api,
ai-web, pg-ai, cube, cubestore, langfuse, lf-db), so 21 were already
there. Four documents gave four different numbers because they were
counting different things at different times.
BUILD-AI-CONTAINERS.md (21 pre-existing) and current-state.html (28
total) were already right and are untouched. README.md said 22 where it
meant 21. The host brief said 20 in three places while naming only 19,
and it is the one with a real gap behind it:
openplc-runtime appears NOWHERE in the 351-line host brief. Not in the
stack table, not in the data flow, not in the rules - while CLAUDE.md
calls it live control that must never be restarted, and the build spec
names it as the reason port 502 is the one exception to no-published-
ports on this host. The document a new engineer reads first to learn what
runs here omitted the only container doing live control. wireguard was
missing from the same table, though mentioned in three other sections.
Both are now in §3. §6's published-ports list was missing 502/tcp and
8443/tcp for the same reason; verified against docker ps, they bind to
10.0.0.17 rather than 0.0.0.0, and that binding is what keeps
unauthenticated Modbus off the internet, so it is recorded as a control
and not as a detail. §12's "confirm 20 containers up" no longer names a
number - that is how it went stale.
The brief's stamp moves to 2026-09-01 with a note saying what was amended
and what was left alone, so Dan's authorship of the rest stays clear.
Separately, README.md listed ai-docs-worker among the containers as
though it were running. It was never deployed: Phase 9 shipped with the
document screens served by ai-api, and the worker sits behind a
--profile worker flag. compose and current-state.html both say so
already; the README table now does too.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
README.md was 481 lines, 60% of which was a nine-phase deployment runbook.
A new team member had to scroll past every deploy command to reach what
the folders are. The runbook moves out whole to spec/REBUILD.md - content
unchanged apart from heading levels and seven link targets repointed for
the new depth. README.md is now 258 lines.
Two additions:
"Start here" gives a reading order - README, then the workflow map, then
the current state, then CLAUDE.md - and a table routing to the runbook,
the build spec, the host brief or REQUESTS.md by what the reader came to
do. It also names the three accesses needed before starting, none of
which are in this repository.
The architecture diagram is redrawn because the old one was wrong. It
showed Authelia in front of ai-web. Since 28 August that hostname carries
no Authelia at all - it is matched on source IP. The README said so
correctly, 170 lines below the picture. The new diagram shows both
parties arriving at the same Caddy and then branching: ai.yokogawa.tech
to the @scada remote_ip matcher, the other three through import authelia.
The operator is inside the Caddy/Authelia environment; they take a branch
Authelia does not sit on. The answer path behind ai-api is drawn
separately, because it is the same whichever door the question came in
through. Structure follows caddy/ai-routes.caddy:190-203.
The layout block gains REBUILD.md and the eval count is corrected to 78
in passing, that line being part of the block rewritten here.
Every markdown link in all nine documentation files was checked after the
move: none broken.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The exam: eval/testset.jsonl holds 78 cases (A10 H28 L8 N5 P10 R10 T5 U2).
workflow-map.html called it a 75-question exam in the two places it states
the current total. Corrected. The two sentences describing how the exam
"grew from 67 questions to 75" are left alone - that step is historically
correct: 67 plus the eight live-model failures is 75, and the Phase 5
findings H26, H27 and H31 took it to 78 afterwards.
The env files: .env.example said TWO 0600 files under ~/ai/ and listed
pg-ai.env and api.env, but its own line 105 refers to langfuse.env, and
README.md, scripts/deploy.sh and compose/langfuse-compose.yml all use
three. The header was simply wrong; langfuse.env is now named in it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The root held five documents of two different kinds with nothing to tell
them apart. This repo is being handed over as both design spec and
as-built record, so the two are now separated by lifetime:
spec/ BUILD-AI-CONTAINERS.md, YAU_Linux_Host_Onboarding.md
normative and durable - what we agreed to build
status/ workflow-map.html, current-state.html, REQUESTS.md
snapshots that go stale and get edited constantly
Nothing mechanical moved. The twelve code folders are deployment
interface - compose builds from /home/azureuser/ai/api, /web, /ingest and
deploy.sh copies from $HOME/ai/compose, /cube/model, /db - so they stay
where they are. CLAUDE.md stays at root because Claude Code loads it from
there, as do .gitignore and .gitattributes, which only apply repo-wide
from the root.
All 26 references to the build spec were bare filenames in prose, so the
move broke no mechanism. The seven markdown links in README.md and the
two pointers at the top of CLAUDE.md are repointed.
docs/ is deleted. It only ever held .gitkeep - nothing mounted or read
it, and ingest reads the host path /datadisk/ai-docs mounted at /docs
inside the container. An empty docs/ invites the misreading "project
documentation goes here". The ignore rule stays, now covering the whole
folder, so a controlled document dropped there still cannot be committed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Caddy forwards the real client address and ai-web's nginx log records it
as the last field, so the host can prove what a curl from here cannot:
10.0.0.21 loaded the page on 28 August, the day the SCADA-only rule was
applied, and asked three questions on 31 August, each answered 200.
verify.sh now reads that log instead of printing "somebody go and look",
and checks the deny arm from the other direction as well - any non-console
client in the log got past a matcher that should have refused it. A clean
miss is a warning, not a failure: docker logs are ephemeral and a recreate
of ai-web wipes the evidence.
That ephemerality is why a working operator path sat unnoticed for three
days. Access logging at the Caddy block was considered and declined on
1 September; the confirmation lives in the Phase 7 gate instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reverts 038cdc5. The source itself was reachable and the demo data was
sound, but the design questions underneath it were not settled: whether it
belongs in Cube or in plain SQL once Cube is repointed at imh at Phase 4,
whether it is append-only, how an entry is authored and authenticated, and
what a real query window over it looks like. Better out than half-committed.
Kept from 038cdc5:
H02 the 'last 3 days' wording. That change was asked for on its own and
has nothing to do with the shift log. It still fails, on the
_contains_quantity false positive.
Restored:
N04 back to the shift log question. With no such source it tests what it
was written to test again: "no such source" is not "no records found".
S01, S02 removed with the feature.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A shift log is continuously updated, which rules out both stores we already
have. Not doc_chunks: that store is built on doc_number + revision +
effective_date and nothing in it is citable until a human confirms the header,
so a record changing every shift would wear out the gate that makes procedure
citations trustworthy. Not the historian: we hold read-only on imh and may not
write to it at all, it is keyed on the CI Server item and a log entry has no
item, and its seven-day retention would expire the one source that could
outlive it.
So a table in pg-ai, read on the historical lane because that lane's contract
already fits it - a time window, rows, and "no records found" when there are
none. Entries are rows, NOT citations: a citation is a controlled document with
a confirmed revision and an operator's note is not one however true it is. The
Citation contract is untouched.
Routed on the question text rather than a classifier label. Adding a sixth
class to the five-way classifier - the most safety-relevant component in the
stack - to reach one demo source would be a poor trade. Case S01 pins the
trigger.
Demo entries are anchored to the start of today in site local time, so a
re-run always lands them inside the rolling seven days and the shift label
always matches the clock. The alarm fixtures are absolute and have gone stale;
this cannot. Future-dated entries are dropped, and the load asserts the window
and the count rather than trusting them.
Eval, 78 -> 80 cases:
S01, S02 the shift log, whole and filtered by equipment
N04 repointed at the environmental discharge licence. The old wording
asked about the shift log, which now exists, so it had stopped
testing "no such source" - a different answer from "no records
found", and the distinction is the point of the case.
H02 window changed to a relative one. It still fails, on a false
positive in _contains_quantity: "the last 3 days" reads as a
fabricated figure, so the correct zero-row answer is rejected.
Left for its own change.
80 cases: 97.5% overall, 100% classification, p95 5257 ms. One contract
violation (H02), so the Phase 8 gate is still not met.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Left uncommitted on 28 August alongside the commit of the same name. Adds
section 3b, "Two decisions taken on 28 August, and what they cost", and
refreshes the tool and phase annotations around it.
Committed as found, unreviewed: this is someone else's work being preserved
rather than endorsed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The query window is a rolling seven days by design, and stays that way against
the real SQL historian - it is not parsed out of the question. Two things I
wrote assumed otherwise.
H29 was wrong in two ways. It demanded the words "retention" and "seven days",
and it BANNED "no records found" - which contradicts CLAUDE.md, where that is
the required wording for zero rows. With a fixed one-week window June genuinely
has zero rows in what was queried, so the phrase is correct rather than evasive.
It also described the missing piece as question-window parsing, which is not a
gap but the design.
Rewritten to pin the risk that actually exists: SUBSTITUTION. A question naming
June must never be answered with this week's figure wearing June's label.
Answering "there were 14" would be exactly that, and nothing downstream could
catch it. The observed answer refuses the substitution and states the window it
used, so the case now passes on its merits rather than being red by default.
must_not is the only half run_eval enforces, so the banned phrases are ones
that appear only on a substitution.
outside_retention is reframed as what it is: a GUARD, always false while every
caller asks for seven days, there so a caller who later passes a longer window
cannot get an empty result that reads as "nothing happened".
REQUESTS.md and the removal guide now say that extending retention on its own
is inert - the window would still ask for a week of a longer history. Both
halves are needed, or neither.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
metrics.run() computed outside_retention but nothing consumed it, so the flag
was dead and the distinction it exists to carry never reached an answer. Both
gather_historical and gather_advisory now pass it, with retention_days, as
EVIDENCE - not as an instruction in a prompt.
Also corrects eval case H29 to record what it actually is: a deliberately
failing case, pinned before the fix per the house convention. It cannot pass
yet for a reason that predates this change - gather_historical always queries
a rolling 7 days and never parses the window the question asks about, so
outside_retention can never be true on that path. Observed today:
"No records were found for June 2026. The data provided is for the window
from 2026-08-24 to 2026-08-31."
Honest, and it states the window, but it leads with "no records were found"
and never says the historian keeps only seven days - so an operator cannot
tell a retention limit from a quiet month. Question-window parsing is the
missing piece and is a separate change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three open Phase 5 findings were one defect: the stand-in was keyed on
CI Server POINT names (PS_STN_WET_WELL_LEVEL) when the historian is keyed on
CI Server ITEM names (AID.WRPS.STN.LEVEL). Modbus carries register numbers,
not names, so those two layers are free to differ - and do. Reconciling
against the register map, as planned, would only have proved the first three
namespaces agreed with each other.
Rebuilt from WRPS/05-scada/modbus, so item names, sample rates, retention and
timestamp semantics come from the machine rather than from a guess.
(a) Level tag does not join. PS_STN_WET_WELL_LEVEL becomes a tag row in its
own right; LIT-101 is marked NOT HISTORISED - a field input on %IW0 that
never reaches SCADA. It was the only seed row carrying two addresses.
public.historian_items holds the item-to-tag mapping, generated by
scripts/gen_historian_items.py and enforced non-empty at generate, at
deploy and at verify.
(b) first_alarm/last_alarm returned UTC. Converted inside the measure, so it
stays in Cube and happens once. Aggregate first, convert after - the other
order picks the wrong row across a DST fall-back. Returned as a formatted
string with a companion site_timezone measure. Storage being UTC is now
confirmed, not assumed: all 49 points carry TIME_ZONE "Date+time GMT" and
every history group CORRECT_DAYLIGHT=0. This answers Phase 4 task 4.
(c) High level alarm filed against the wrong equipment. Both sides were right
about different things; the defect was asserting equipment twice. The
history now carries no equipment column at all - faithful, since CI
Server's section tree stops at the station and three pumps. Equipment is
reached bit -> tag -> equipment via public.alarm_bits.
Alarms are derived, not stored: CI Server's ALARM_HISTORY group is empty
because every item imports with alarming off. Decomposing the alarm word needs
no configuration that does not exist.
Three things the SCADA config changed that were never filed as faults:
- retention is 7 days, not 30. The advisory path was reporting a month of
evidence drawn from a week of data
- the analogue rate is 5 s, not 60. Two measures multiplied sample counts by
a hardcoded 60 - a twelvefold overstatement that read as plausible
- the deadband warning in process_values.yml was wrong and was steering
people away from the correct measure
db/002_fixtures.sql now asserts its own counts at load and cross-checks the
alarm derivation against two independent signals. Those prove the pipeline,
not the plant.
db/README-standin-historian.md documents removal: the seam between generation
and contract, and twelve assumptions about imh that are NOT confirmed. Two of
them fail silently.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
None of these were reachable by the tests as they stood, and all three were
silent - the screen looked correct in every case. An 8-page control philosophy
found all of them in one upload.
1. THE WHOLE DOCUMENT BECAME ONE CHUNK. pypdf emits one line per line of the
PDF and no blank lines at all: 416 lines, none blank. Section splitting looks
for Markdown headings and paragraph splitting looks for blank lines, so the
chunker was a no-op on PDF text - one 18,307-character chunk, a single
embedding vector for eight pages, and every citation reading "(untitled),
page 1". A longer document would have exceeded the embedding model's input
limit and failed to publish at all.
convert.py now recovers structure: headings from numbered and capitalised
lines, paragraphs by reflowing on line width. Heading detection is
deliberately narrow, because the dangerous direction is promoting a numbered
STEP to a heading and splitting a step sequence - so a heading must be short,
a few words, and without terminal punctuation. "1. Purpose" qualifies;
"1. Open the isolation valve and confirm zero pressure." does not.
chunking.py gains a ceiling no chunk may exceed whatever the input looks
like, falling back to line and then word boundaries. The step-sequence
refusal still holds below it and is unchanged for any realistic procedure;
past it, splitting is the lesser harm, because an embeddings call that fails
protects nobody. Two heuristics found only by running the real file:
"SCADA" and "WRPS-PRO-001" were being promoted to headings, which cut real
sections in half and re-titled the remainder with something meaningless, and
"11 August 2026" was parsing as section 11.
19 chunks now, largest 574 tokens, sections matching the document.
2. EVERY CHUNK CARRIED doc_title = "Revision". TITLE_RE used [\s:]+ for the gap
after the label, and \s includes the newline. A cover page flattens to a
label column then a value column - Title / Revision / Date - so it matched a
bare "Title" line, consumed the line break and captured the next line. Now
[ \t:]+, the same trap AUTHORISING_ROLE_RE was fixed for once already. The
document's title is now null, which is the honest answer: a citation falls
back to the section title, and a confidently wrong title falls back to
nothing. Inherited, so fixed in ingest.py too.
3. RE-PUBLISHING A DOCUMENT DUPLICATED IT. approve deleted prior chunks by
source_file, which carries the upload_id and is new on every upload -
so approving the same revision twice left 38 live chunks and the same
passage citable twice. Invisible on screen, because live_documents groups by
(doc_number, revision) and only the count moved. Now deletes by document and
revision as well, and logs how many chunks it replaced.
The two chunkers are now provably in step rather than asked to be. The header
of chunking.py claimed drift in ingest.py could not be detected from the test
suite; that was wrong, both files are on disk. The new test compares the source
of chunk_section, _split_on_lines, _split_on_words, extract_header and
approx_tokens character for character. Writing it found that one earlier edit to
ingest.py had silently not applied, leaving the two genuinely divergent, and
then that extract_header's docstring had drifted. Both fixed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 9's operator path, built ahead of Phase 8 at the customer's direction and
live at api.yokogawa.tech/documents. Upload, convert, review, approve, withdraw
and restore. The pool screen is explicitly out of scope.
Served by ai-api rather than ai-web, and mounted at /documents rather than
/docs. ai.yokogawa.tech is SCADA-only since 2026-08-28 and passes through no
Authelia, so it has no identity to record; publishers arrive on
api.yokogawa.tech where the forward-auth headers still do. /docs stays with
Swagger, which the customer is keeping - two things under one prefix with two
different access policies is what gets misread during a later edit.
Conversion is text extraction, not document parsing: pypdf, python-docx and
openpyxl. Docling would be better at this and pulls torch, which lin001 has
neither the memory to install nor the business running next to the demo plant's
PLC. The cost is real - no layout, no table structure, and a scan cannot be read
at all, so it is refused rather than stored empty. It is acceptable only because
the converted text is shown to a person before the document can be cited, which
is the same safety net the design already required for the header. convert.py is
the one file to change if that stops being true.
Chunking is mirrored from ingest.py rather than shared, because the two live in
different images. They must stay identical: if they drift, the same document
chunks differently depending on who loaded it, and the assistant answers or
fails to answer depending on that. The step-sequence rule is locked by a test.
Identity is self-asserted for the demo - the actor is typed on the form, which
section 16 forbids, and the publisher list is one name with no password. Rows are
written as `demo:<name>` with actor_groups = 'DEMO-UNVERIFIED' so that when real
auth goes on, a name somebody typed stays tellable from a name Authelia proved.
doc_actions cannot be deleted from, so an ambiguity there would be permanent.
Two rules the code enforces rather than documents: uploading is open to anyone
who reaches the page, because uploading changes nothing an operator can see -
approving does, and that is what is gated; and an empty publisher list means
nobody, not everybody.
Verified on the host end to end: withdraw as a non-publisher 403s, with a short
reason 400s, and as admin flips 5 chunks and writes a complete audit row;
restore puts them back and keeps both rows. The corpus is unchanged afterwards.
Requirements are split so the document dependencies install in their own layer -
a change there costs four small wheels instead of re-resolving fastapi,
langgraph and langfuse on a 2 vCPU shared host.
The five divergences from section 16 are recorded in section 14. The one with
teeth: files published through the UI stay in the inbox, so `ai-ingest --all`
cannot see them and the two paths must not be used on the same document.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An operator standing at the SCADA console should not complete a Duo push to
ask a question, and nobody outside the plant should reach the assistant at
all. The ai.yokogawa.tech Caddy block now admits remote_ip 10.0.0.21
(yau-poc-cicore1, static) and returns 403 to everything else. Applied on
lin001 2026-08-28; snapshot at ~/Caddyfile.bak-ai-scadaonly-20260828.
This also settles why the console could not reach the assistant at all:
auth.yokogawa.tech has no pinpoint record on the DC, so a LAN browser got a
correct 302 to the portal and then died on DNS. It went unnoticed because the
device agents write to Influx over the /api/v2/write MFA bypass and never
touch the portal - no browser had ever hit Authelia from inside the VNet.
Verified before applying that an IP matcher can work here: Caddy sees real
client addresses, and WireGuard peers arrive masqueraded as 172.19.0.6 so
they do not match and are refused along with the internet.
What this costs, recorded in section 14 as a shortcut and not as a security
control: it is an IP allowlist on a flat network with no OT/IT boundary, so
anything that can take 10.0.0.21 inherits unauthenticated access; Langfuse
traces are now anonymous, so there is no record of who asked what; and the
assistant is out of browser reach over the VPN. It is in scope for the
section 2 OT/safety review, which is still outstanding.
api.yokogawa.tech is unchanged and still fully gated - Phase 9 publishing
depends on Remote-User/Remote-Groups and stays there. The now-inert
ai.yokogawa.tech entry in the Authelia rule is deliberately left in place so
restoring the gate is a Caddy reload rather than an Authelia restart that
logs out every user on the host.
verify.sh treats 403-from-lin001 as the pass for the deny arm and states
plainly that the allow arm can only be proved from cicore1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Read from the running containers on 2026-08-28, not from the plan.
Both AI steps move from "stood in for" to running: the model account arrived
on 27 August, the dropdown and the placeholder wording are gone, and the
documents are indexed so search works by meaning. Two of the three items in
section 3a are now closed; the historian login is the only blocker left.
The map records what switching the model on cost, because that is the part
worth telling somebody who was not here: five faults inside an hour, every one
hidden by the stand-in that replaced the model. The interlock finding is
written out in full - the assistant was handing the answer writer the step list
while withholding the title block - because it is the one a safety reviewer
should see.
The Unclear lane no longer says it cannot be exercised. It was, by the
zero-confidence fault, and it failed to the cautious side, which is how it was
designed to fail.
The model-account card stays honest about the small deployment that was asked
for and not provided: labelling runs on the flagship model, works correctly,
and costs about ten times the estimate per question.
Note: this file already carried uncommitted changes from a previous session.
Those are included here and have not been separately reviewed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CLAUDE.md: add the failing case before fixing it. These were added after, not
before - recording that rather than quietly complying.
67 -> 75. L01-L05 are the five contract-shape failures; L06-L08 cover the
retrieval and schema change: that the header fields reach the answer, that step
sections never do, and that a document marked uncontrolled is identified rather
than reported as nothing found.
Every one is a case no stub run could have produced, which is the reason the
67-question set passed for weeks without catching any of them. REQUESTS.md
still says 67 and needs updating when the acceptance run is scheduled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four faults, all surfaced within an hour of the first live Azure OpenAI call
on 2026-08-27, all invisible under NO_LLM_STUB because the stub supplied the
very fields that turned out to be missing.
1. The classifier few-shot showed eight replies of {"question_class": ...}
alone. A few-shot reply is a shape the model copies, so it omitted
confidence, which defaulted to 0.0, fell below the 0.7 threshold, and EVERY
non-procedural question downgraded to UNCLEAR. The replies now carry the
complete payload the system prompt asks for. Confidences are varied and the
traps carry alternatives: a constant teaches the model to emit that
constant, and the tie rule in apply_safety_rules only has something to work
with if the runners-up are populated.
2. procedure{} was the one part of the procedural payload not assembled from
evidence, contrary to _assemble's own stated rule. The model returned
effective_date "" - neither a date nor None - so ProceduralAnswer rejected
the answer, the single regeneration failed identically, and every procedural
question returned 422.
3. title and authorising_role came back "" for the same reason: the model was
asked for header fields it had never been shown.
4. documented_limits[].citation arrived as the string "WRPS-DEMO-003, Section
4" where a Citation was required, because the schema hint said only
"documented_limits": [] and told the model nothing about the shape.
procedure_identity now takes no `generated` argument at all: there is no path
by which a model can name a revision an operator does not hold. documented_
limits attaches the real Citation by matching source_file against what was
actually retrieved, and DROPS a limit matching nothing - a limit carries the
authority of the document behind it, and misattributing one is worse than
omitting it.
Both live in contracts.py rather than agent.py because they are contract
rules, and because agent.py imports langgraph, which would make the test suite
unrunnable on a bare checkout.
The prompt also now separates two things it was conflating: retrieval
returning nothing (say so and stop) from retrieval returning a document marked
draft, demo or superseded (identify it, quote it, and state the marking).
Including the header chunk made the model read "NOT A CONTROLLED DOCUMENT" and
answer "no controlled procedure was retrieved" while citing one. The marking is
information the operator needs, not a reason to withhold what was found.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two retrieval faults, both only visible once a real model ran.
find_procedure ranked a procedure's chunks by similarity to the question. For
"how do I lift the interlock on Pump 02" the closest chunks ARE the step list -
so the branch whose entire purpose is not reproducing steps was handing the
answer writer nothing but steps, while omitting the header block carrying the
title and the authorising role. The model was being asked for a title it had
never been shown, and returned "".
Once the document is identified, WHICH document it is settles what to send:
the header and the prerequisites, in document order, never the steps.
STEP_SECTION_RE is a second line behind ProceduralAnswer's instruction-language
check, not a replacement for it - the contract still rejects instruction
language whatever arrives here. This removes the temptation rather than relying
on catching it.
Separately, rerank did 0.75 * chunk.similarity where similarity is NULL for a
chunk ingested with --no-embed: `1 - (NULL <=> vec)` is NULL, so it raised
TypeError and 500'd the whole question rather than ranking that chunk last. It
now degrades to the lexical half - an unembedded chunk is still findable, just
not by meaning - and Chunk.similarity is typed honestly as float | None.
find_procedure_lexical takes the same identify-then-expand shape, so the stub
keeps testing the shape it always did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ProcedureIdentity requires a title and an authorising role, and neither was
stored anywhere. The answer writer was asked for both, read them off whatever
chunk retrieval happened to return, and returned "" whenever the header chunk
was not among them.
They belong in the row for the same reason doc_number and revision do: they
are facts about the controlled document, established once when a human
confirms the header, not something to re-derive per question from whatever
text was retrieved. Denormalised onto every chunk exactly as the existing
header fields are - ingest replaces every chunk of a source_file in one
transaction, so they cannot drift within a document.
complete() deliberately still requires only doc_number, revision and
effective_date. A missing title makes an answer less useful; a wrong revision
sends somebody to the wrong document. --assume-yes must keep refusing on the
second and tolerate the first.
controlled_copy_location is NOT in the schema. It is a site fact, identical on
every row, and the one field where an invented value sends a person to a place
that does not exist. It is CONTROLLED_COPY_LOCATION in api.env, defaulting to
a string that names who to ask.
The authorising-role pattern requires the colon: without it the lazy gap
swallowed the field name and captured "role: Station Maintenance Supervisor"
as the value, which the first run caught.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lin001 is 2 vCPU with 3.8 GiB and no swap, shared with 27 other containers.
Cube was rebuilding three pre-aggregations over static fixture data on 10, 10
and 30 minute keys, and its own log shows what that cost:
"Previous interval #19593 was not finished with 30000 interval"
"Interval #19594 finished after 00:02:50"
A refresh taking 2m50s scheduled every 30 seconds, at interval #19594 -
overlapping and never catching up, for days. Every rebuild produced a
byte-identical result, because the fixtures do not change.
24 hours until imh makes the data genuinely live, at which point these get
tuned deliberately rather than left at a number that was never chosen. Cube
also suggests fewer partitions via rollup_lambda; that is a modelling change,
not a demo-day one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Kept, but labelled. ai-api reaches Cube as cube:4000 over the proxy network and
never by name, and the operator UI never touches Cube - the hostname exists so
an engineer can open the playground to hand-verify a measure, which a tunnel
does equally well. So it is the first one to drop if this host is being tidied
or the name is wanted elsewhere.
The note is in three places because whoever tidies this host will be reading
one of them and not this repo: above the block in ~/Caddyfile, inline on the
Authelia rule, and here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Leaving the auth.yokogawa.tech note in verify.sh was a standing nudge towards a
fourth name, which is the opposite of what this host needs. Removed.
In its place, the thing that was actually missing: a list, where the Caddy
blocks live, of which hostname serves what and which one a control-room PC
needs. lin001 is shared and already carries a dozen *.yokogawa.tech services -
the cost of a name is not the DNS change, it is that nobody else can tell what
it is for. Notes that cube.yokogawa.tech is the one nothing depends on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Raised as an item for Dan, which was premature. It blocks exactly one thing: a
demo from a control-room PC. Development and the Phase 8 eval both run from
outside the VNet, where auth.yokogawa.tech resolves normally, and the demo it
would block cannot happen until the Azure OpenAI account lands anyway.
The finding stays in verify.sh as information rather than a failure, so it is
in front of whoever prepares that demo instead of failing every run until then.
Also worth recording: this was inferred from lin001 resolving against 10.0.0.5,
not tested on cicore1. Same DC and same zone, so very likely identical, but it
has not been confirmed on the machine it concerns.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The DC is authoritative for a yokogawa.tech zone holding pinpoint records only,
so from inside the VNet every name not in that zone is NXDOMAIN rather than
forwarded. ai and influx are in it. auth is not.
So an operator on cicore1 resolves ai.yokogawa.tech, reaches Caddy, is
redirected to https://auth.yokogawa.tech/?rd=... and cannot resolve it. The
assistant is reachable and cannot be signed into. Nothing caught this because
influx is the only other pinpointed hostname and LAN traffic to it uses the
bypassed API paths, which never touch the portal - ai.yokogawa.tech is the
first thing here a LAN client must interactively sign in to.
verify.sh now resolves auth.yokogawa.tech alongside the hostnames that redirect
to it, and fails rather than shrugging when it is missing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cube, api and ai joined the HTTPS_UserAccess two_factor rule in one Authelia
restart on 2026-08-27, backup configuration.yml.bak-ai-20260827. All four AI
hostnames now return 302 to the auth portal, and POST /ask on ai.yokogawa.tech
returns 303 to the portal rather than a 404 from ai-web, which is what proves
the route exists.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both found running it on lin001 straight after the Caddy blocks went in.
The public-endpoint check called lf, cube and api unreachable. They are not:
the host resolves yokogawa.tech through the DC, which holds pinpoint records
only, and those three have none. Nothing is wrong with them and the check
cannot tell from here - so it now says so, and says to check from outside the
VNet, instead of printing a failure that means nothing.
403 now has its own message. It is what Caddy plus Authelia return for a
hostname with a site block and no access_control rule, which is the state
between applying the Caddy blocks and applying the Authelia rule - the one
case an operator is most likely to hit and least likely to recognise.
And the log check ran after the agent_ro test, which deliberately attempts a
forbidden INSERT that pg-ai logs. verify.sh was flagging an error line it had
just written itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All three A records resolve to 20.211.144.151 and the DC carries the pinpoint
record ai.yokogawa.tech -> 10.0.0.17, verified from lin001. Kept in the file
as a record of what was asked for and what arrived, with the two things that
came out of it: api and cube have no pinpoint record and do not resolve inside
the VNet, and the names resolving is not the same as the services answering -
the Caddyfile blocks and Authelia rules are still to be applied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
api.yokogawa.tech has a public A record but no pinpoint record on the DC, so
it does not resolve from inside the VNet at all. The browser called it by
hostname, which means an operator on cicore1 would have loaded the page and
had every question fail on DNS - the exact gap Phase 7's gate exists to catch,
and one an engineer's laptop cannot see.
Caddy now routes /ask under ai.yokogawa.tech to ai-api, inside a route block
so import authelia still runs first: forward_auth sorts after handle in the
default directive order, and outside a route the handles would be terminal and
the gate would never run. Only /ask is routed - the Phase 9 publisher rule is
scoped to api.yokogawa.tech and a wider route here would leave it inert.
Also fixes the fallback it replaces. The build arg defaults to "", and
`?? "https://api.yokogawa.tech"` does not catch an empty string, so the
documented real-deployment build resolved the API base to "" and posted /ask
at ai-web, which 404s it. verify.sh and deploy.sh now check both the route and
whether the built bundle carries the hostname.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file says "every key, no values" at the top and was not that, which cost
real time twice on lin001 in one day.
- CUBEJS_DB_* was absent entirely, while ai-compose.yml states that Cube's
database settings come from api.env. Deploying Cube meant reconstructing
what it needed from compose comments and db/003_roles.sql. Added, with the
pg-ai.env names its credentials come from, and a note that Phase 4 turns
the block into an mssql connection against imh.
Also recorded NEGATIVELY: not CUBEJS_EXT_DB_*. Cube v1 refuses Postgres as
an external pre-aggregation store, so someone reading the older compose
file will otherwise try to supply keys for a setting that must not exist.
- NO_LLM_STUB, which is new and defaults to false here for the same reason it
defaults to false in config.py.
- The Langfuse keys were listed but not explained, and every way of getting
them wrong is SILENT:
absent -> _langfuse() returns None, every question untraced
mismatched pair -> a client is built, Langfuse rejects it, and main.py
swallows the exception by design
Neither logs anything, in an answer path that is deliberately built never
to break on observability. The symptom is identical - no traces - so
correcting one cause while the other is still present looks like no
progress at all. That is exactly what happened: a placeholder secret was
pasted alongside a public key from a different pair, and the fix looked
like it had not worked.
So the file now says: they are PROJECT keys from the UI, not the server's
own SALT/NEXTAUTH_SECRET in langfuse.env; they must be a matched pair; the
secret is shown once and stored hashed, so it cannot be read back; and
traces must be confirmed as ARRIVING rather than inferred from config.
No secrets here, including the masked tail of a real key - the example suffix
is invented.
The Langfuse fix itself is not in this commit and cannot be: it was two lines
in ~/ai/api.env, which .gitignore excludes on purpose.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Azure OpenAI is pending and imh is pending, so POST /ask could not return
anything at all - which left the entire chain either side of the model
unproven: the browser, the API, entity resolution, Cube, retrieval, the
contracts, the banners, the error paths. All of it is testable now, and waiting
for a key to find out whether it works is a choice to find out later.
NO_LLM_STUB=true substitutes the two steps that need a model and nothing else.
- Classification: the caller supplies the class, from a dropdown in the UI.
NOT a keyword classifier. A crude keyword classifier produces a PLAUSIBLE
label, and a plausible wrong label is the exact failure this system exists
to prevent - "how do I reset it" landing in Historical is how a synthesised
procedure reaches an operator. Choosing by hand is honest about what is
happening and drives each branch deliberately. apply_safety_rules() still
runs over the result.
- Prose: a fixed placeholder per class, in stub.py.
Everything else is the real path. This is possible because generate() already
kept the factual fields away from the model: rows, counts, citations, the
fixture flag and the class are attached from evidence, and only prose comes
from the generator. Splitting that into _generate_prose() and _assemble() makes
the seam explicit - the stub feeds _assemble() exactly as the model does, so
this is a fair test of the assembly path rather than a mock of it.
The contracts are the point. A stub payload goes through enforce_contract()
unchanged, and it FAILED first time on two classes: the "nothing found" wording
did not match the not-found detectors, so Reference and Procedural returned 422
rather than an uncited answer. That is the contract doing its job against text
no model wrote. Retries are pointless on deterministic output, and a 422 is a
real result here, not a stub bug.
Retrieval is lexical (retrieval.lexical_search), because embedding the question
needs the model. Kept beside search() and never called on the normal path, so
nobody reads a trace and mistakes a lexical hit for a semantic one. It matches
what the operator typed, not what they meant.
What it does not prove: whether the classifier would have labelled correctly -
a person did; whether retrieval finds the RIGHT chunk; and nothing about prose.
It also cannot fill prerequisites_verbatim - extracting them with a regex would
be the "synthesised from fragments" failure the Procedural contract forbids, so
the list is empty and the answer says so.
Every answer carries stub_mode: true in the contract, not decorated on by the
UI, and a banner beside the fixture banner. Same reasoning: an answer nobody
generated must not be indistinguishable from one that was.
Also here:
- demo/ai-docs: three fabricated documents, numbered WRPS-DEMO-00x so header
extraction is genuinely exercised against a number no real WRPS document
can have. Their setpoints contradict tags.csv on purpose.
- VITE_API_BASE build arg, for a tunnelled build before DNS exists. The
tunnel origin is allowed in CORS only while NO_LLM_STUB is on, so it
disappears with the flag. Proxying /api through ai-web's nginx would have
been easier and was rejected: it creates a second route to the API that
bypasses the api.yokogawa.tech Caddy block, where the Phase 9 publisher
rule lives.
Verified on lin001 with no Azure key set at all: all five classes return 200
through the real UI in a browser, over an SSH tunnel, with citations from the
demo documents, real Cube numbers, and both banners showing.
Turning it off: NO_LLM_STUB=false in ~/ai/api.env, restart ai-api.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two blockers stopped Phase 3 being exercised at all before Azure OpenAI exists.
Both are lifted here, and neither weakens the header confirmation - a document
still becomes citable only after a person confirms its number, revision and
effective date.
- parse_markdown() for .md, .markdown and .txt. Docling earns its place on a
PDF: it recovers structure that is not in the bytes, and that structure is
what makes section-boundary chunking possible instead of token-count
splitting that cuts step sequences in half. A Markdown file already
contains "## 2. Prerequisites". Running a document layout model over it
buys nothing and costs the entire torch stack.
It also unblocks the image. docling==2.15.1 does not resolve on
python:3.12-slim: pip backtracks through docling_ibm_models releases for
twenty minutes and exits 2. That is a real problem for Phase 3 and it is
NOT fixed here - PDFs still need Docling and the ai-ingest image still will
not build. Markdown ingestion runs from the ai-api image meanwhile.
Page is 1 for these, because a Markdown file has no pages. A citation to
one carries a section title and no meaningful page number, which is honest.
- --no-embed inserts chunks with a NULL embedding and makes no API call, so
the retrieval path can be exercised before an embeddings deployment exists.
NULL, not a zero vector: a zero vector is a point in the space, it ranks
against real queries, and it would surface as a plausible hit for anything
asked. NULL returns no similarity at all.
These chunks are invisible to vector search and findable only lexically.
The flag warns about that four times on the way past, with the SQL to clear
them, because a database half full of unembeddable chunks looks exactly
like working retrieval right up until the question that matters returns
nothing.
Before real ingestion: DELETE FROM doc_chunks WHERE embedding IS NULL;
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The UI calls the API with credentials: "include", because ai.yokogawa.tech and
api.yokogawa.tech are different origins and the Authelia session cookie has to
be attached explicitly. The CORS middleware never set allow_credentials, and a
browser refuses a credentialed cross-origin request unless the response says
Access-Control-Allow-Credentials: true. It fails at the preflight, so the real
request is never sent:
Access to fetch at '.../ask' has been blocked by CORS policy: the value of
the 'Access-Control-Allow-Credentials' header in the response is '' which
must be 'true' when the request's credentials mode is 'include'.
Every question from the UI would have failed at Phase 7 with "Could not reach
the assistant" - the app's network-error branch, which says nothing about CORS
and points at the wrong layer entirely. The API is fine; curl against it passes,
because curl is not a browser and does not enforce this.
Found driving the built UI in a browser. It is not reachable by any test that
does not involve a browser, which is the useful part: the Phase 7 gate says an
operator reaches the UI and gets an answer end to end, and that gate is the
first thing that would have caught it - at the point where DNS, Caddy and
Authelia are all new too, and any of them a plausible suspect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every Historical and Advisory answer stated a window in AEST and queried one
shifted by ten hours.
rolling_window() builds its boundary strings in SITE_TIMEZONE - that is the
whole point of it, and its docstring says so. metrics.run() then posted the
query to Cube with no timezone at all, and Cube defaults to UTC. So
"2026-08-14T15:22:13" meant 15:22 Sydney to the code that produced it and 15:22
UTC to the engine that ran it, and MetricResult.time_window reported
SITE_TIMEZONE from config rather than whatever the query actually used, so the
two could not disagree visibly.
Measured on the fixtures, same dateRange, one field changed:
timezone UTC 8019 samples
timezone Australia/Sydney 8619 samples
600 samples. One per minute, ten hours, exactly the offset.
Nothing about the answer looked wrong. The prose was right, the count was a
real count, the window description was correctly formatted and correctly named
AEST. It was only visible by reading the Cube query in the UI's "show working"
panel - which is an argument for that panel existing, and an argument for
looking at the thing in a browser rather than trusting curl against the API.
- check_cube_query() now takes site_timezone and pins it onto the query, at
the single point every Cube query passes through. Per-query-builder is the
wrong place: "remember to set the timezone" is not a control, and this
defect is what forgetting looks like. An explicit timezone already on the
query is left alone.
- time_window now reports capped["timezone"] - the timezone the query ran in,
not the one it should have run in.
An unpinned timezone belongs in the same guardrail as an unpinned date range,
and for the same reason: both make an answer unreproducible. The difference is
that an unpinned date range is obvious in the query and an unpinned timezone
is invisible.
eval case H28 records it. Two unit tests: the timezone is pinned, and an
explicit one is not overridden.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both were found hand-verifying the measures on fixtures, and both are left
unfixed deliberately: fixing either now means guessing at what imh contains,
and a guess baked into the seed data or the models is harder to find later than
an open finding. Flagged at Phase 4, in the README, in the model files at the
point of use, and as eval cases that fail until they are settled.
(a) The wet well level tag does not join, and fails as "no records found".
History is keyed PS_STN_WET_WELL_LEVEL, which process_values.yml hardcodes
in seconds_above_high_level_alarm and seconds_above_lshh. tags.csv carries
that name only as an ALIAS of LIT-101, so public.tags has no row with that
tag_id and all 43,201 level rows - a third of the history, on the most
important tag at this station - are unreachable from a tag-level lookup.
Resolve "wet well" -> WW-101 -> LIT-101 -> filter history on LIT-101 ->
zero rows -> "no records found".
That is the failure mode worth being loudest about: it is the safety
behaviour, produced by a key mismatch rather than by an absence of data,
and indistinguishable from the real thing on screen. Filtering by
equipment_id works, so whether a level question fails depends on which path
the agent takes.
The two flow tags use the opposite, self-consistent convention -
PS_STN_INFLOW is a row in its own right and FIT-201 is marked NOT
HISTORISED - so applying that to level is the likely fix. It still waits
for the register map and for imh to say what CI Server historises the point
as. Seed, hardcoded model names and 002_fixtures.sql change together.
(b) alarms.first_alarm and last_alarm return UTC, not SITE_TIMEZONE.
Cube converts time dimensions to the query timezone; a min/max measure over
a timestamp comes back unconverted. The Sydney day bucket for 2026-08-01
returns 2026-07-31T20:00:35 - the right instant, ten hours and one calendar
day out, beside a bucket label that IS in site time.
This breaks "convert to SITE_TIMEZONE exactly once, in Cube", and the fix
has to stay in Cube - the API compensating with timezone arithmetic is the
thing that rule exists to prevent. Which fix is right depends on whether
imh stores UTC or local, which is Phase 4 task 4. Until then these two must
not be quoted to an operator as a clock time.
Phase 4's gate gains an item for each. Everything else verified in this pass -
alarms, operations, the equipment join, alias resolution and the Sydney/UTC day
boundaries - matched hand-written SQL exactly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hand-verifying the measures against the fixtures on lin001, per the Phase 5
gate. Two of them had never executed anywhere, and both failed outright rather
than returning a wrong number - which is the good version of this, but they
failed at the point an operator asks a question, not at review.
- time_weighted_avg put LEAD() inside SUM(). Postgres rejects that flatly:
"aggregate function calls cannot contain window function calls". The per
sample duration now comes from the cube's source query, which changes
sql_table to sql, and the measure just sums value * duration over duration.
The last sample of each tag gets a NULL duration and SUM skips it, which is
correct - how long it stood is not yet known.
This is the measure that matters most later. On the regular one-minute
fixtures it agrees with avg_value to thirteen decimal places
(42.45934027777778 against 42.45934027777775), which proves it is wired up
and proves nothing about imh, where the deadband makes samples irregular
and the two will not agree. Re-verify it there.
- p95_value applied the quality filter through a Cube measure `filters:`
block, which lands outside the aggregate and cannot work on an ordered-set
aggregate: "column process_values.quality must appear in the GROUP BY
clause". Folded into the CASE inside PERCENTILE_CONT, whose NULL handling
does the exclusion.
Also: the priority dimension said only SPILL and PUMP_TRIP are priority 1,
while the data has LEVEL_SIGNAL_FAULT at priority 1 too - correctly, losing the
level signal on a well that can spill is a priority 1 condition. That comment is
the line an engineer reads when checking a priority_1_count, so it disagreeing
with the data matters more than its length suggests.
eval cases H24 and H25 record the two failures, added before the fix.
Verified against hand-written SQL on the same pinned windows: p95_value
61.104999999999976 and time_weighted_avg 42.45934027777778 both match to the
floating point tail, as do sample_count, avg_value, max_value, min_value,
seconds_above_high_level_alarm (7680 = 128 samples x 60) and every measure in
alarms and operations.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 5 deployed to lin001 for the first time, and the cube service as
committed could not answer a single query. Three separate faults, none of them
visible without deploying it.
- CUBEJS_EXT_DB_TYPE: postgres is not supported in Cube v1. Cube Store is the
only external pre-aggregation store, and it is also the default cache and
queue driver, so naming Postgres failed EVERY query - not just
pre-aggregated ones - with "It`s not possible to use Cube Store as
queue/cache driver without using it as external". So cubestore is now a
service: pinned in lockstep with cube, ai-internal only, no ports, data on
/datadisk because it grows and / is 62 GB.
CUBEJS_CACHE_AND_QUEUE_DRIVER: memory is NOT a way out. It does not fall
back - it hangs /readyz and every query indefinitely, logging nothing at
level warn. That cost longer to diagnose than the original error.
This is a deviation from the build spec, which says pre-aggregations
materialise into pg-ai schema cube_preagg. They cannot, on this version.
cube_preagg and its grants in 003_roles.sql stay, unused, so that nothing
else has to change if a later Cube restores Postgres as an external store.
- CUBEJS_REFRESH_WORKER was never set, so nothing built the pre-aggregations.
A query matching a rollup does not fall back to the source: it fails with
"No pre-aggregation partitions were built yet". max_value, min_value and
sample_count were dead on arrival while avg_value worked, which reads as a
per-measure bug and is not one.
- The healthcheck ran wget, which is not in the image (nor is curl). A
perfectly healthy cube reported unhealthy on every deploy, training the
reader to ignore the one signal that would show a real fault. It uses node,
which the image does have.
Verified on lin001: cube healthy, no published host ports, ai-internal and
proxy only, and a query matching alarms_by_hour now returns external: true -
served from Cube Store rather than scanning the source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ingest_file() inserted every chunk with superseded = FALSE, so replacing a
document's chunks reset its withdrawal. One `ai-ingest --all` made every
superseded revision citable again - including the old revision of a procedure -
silently, and only after a bulk run, so a supersede survived exactly until the
next full ingest.
- ingest_file() reads the existing state with superseded_state() before the
delete and carries it through the insert. A withdrawn document is
re-ingested as withdrawn, and logs that it did so.
- --all skips withdrawn documents, so a bulk run does not spend an embeddings
call on a document that will not be cited either way. --include-superseded
overrides it; the chunks still come back withdrawn.
- --restore DOC_NUMBER REVISION is the counterpart to --supersede, refusing
while another revision of the same document is live. Without it the
conservative default would be a dead end - mark_superseded() only ever sets
TRUE, so there was no way back.
- Rule 5 in the module docstring, beside the other four.
Corrects a claim in the Phase 9 design: 16.10 said moving the withdrawn file
out of /datadisk/ai-docs "is not tidying" because otherwise --all resurrects it.
That was true when written and is not now. The guard belongs in the ingest code,
because a rule that depends on somebody remembering to move a file is not a
rule. The move stays as archival housekeeping and is documented as such in
16.10, db/005_doc_actions.sql and README.md.
Phase 3's gate gains the proof: supersede a revision, run --all again, ask the
question that used to cite it.
Not executed anywhere - no Postgres or Docker on this machine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
