An operator standing at the SCADA console should not complete a Duo push to
ask a question, and nobody outside the plant should reach the assistant at
all. The ai.yokogawa.tech Caddy block now admits remote_ip 10.0.0.21
(yau-poc-cicore1, static) and returns 403 to everything else. Applied on
lin001 2026-08-28; snapshot at ~/Caddyfile.bak-ai-scadaonly-20260828.
This also settles why the console could not reach the assistant at all:
auth.yokogawa.tech has no pinpoint record on the DC, so a LAN browser got a
correct 302 to the portal and then died on DNS. It went unnoticed because the
device agents write to Influx over the /api/v2/write MFA bypass and never
touch the portal - no browser had ever hit Authelia from inside the VNet.
Verified before applying that an IP matcher can work here: Caddy sees real
client addresses, and WireGuard peers arrive masqueraded as 172.19.0.6 so
they do not match and are refused along with the internet.
What this costs, recorded in section 14 as a shortcut and not as a security
control: it is an IP allowlist on a flat network with no OT/IT boundary, so
anything that can take 10.0.0.21 inherits unauthenticated access; Langfuse
traces are now anonymous, so there is no record of who asked what; and the
assistant is out of browser reach over the VPN. It is in scope for the
section 2 OT/safety review, which is still outstanding.
api.yokogawa.tech is unchanged and still fully gated - Phase 9 publishing
depends on Remote-User/Remote-Groups and stays there. The now-inert
ai.yokogawa.tech entry in the Authelia rule is deliberately left in place so
restoring the gate is a Caddy reload rather than an Authelia restart that
logs out every user on the host.
verify.sh treats 403-from-lin001 as the pass for the deny arm and states
plainly that the allow arm can only be proved from cicore1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Read from the running containers on 2026-08-28, not from the plan.
Both AI steps move from "stood in for" to running: the model account arrived
on 27 August, the dropdown and the placeholder wording are gone, and the
documents are indexed so search works by meaning. Two of the three items in
section 3a are now closed; the historian login is the only blocker left.
The map records what switching the model on cost, because that is the part
worth telling somebody who was not here: five faults inside an hour, every one
hidden by the stand-in that replaced the model. The interlock finding is
written out in full - the assistant was handing the answer writer the step list
while withholding the title block - because it is the one a safety reviewer
should see.
The Unclear lane no longer says it cannot be exercised. It was, by the
zero-confidence fault, and it failed to the cautious side, which is how it was
designed to fail.
The model-account card stays honest about the small deployment that was asked
for and not provided: labelling runs on the flagship model, works correctly,
and costs about ten times the estimate per question.
Note: this file already carried uncommitted changes from a previous session.
Those are included here and have not been separately reviewed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CLAUDE.md: add the failing case before fixing it. These were added after, not
before - recording that rather than quietly complying.
67 -> 75. L01-L05 are the five contract-shape failures; L06-L08 cover the
retrieval and schema change: that the header fields reach the answer, that step
sections never do, and that a document marked uncontrolled is identified rather
than reported as nothing found.
Every one is a case no stub run could have produced, which is the reason the
67-question set passed for weeks without catching any of them. REQUESTS.md
still says 67 and needs updating when the acceptance run is scheduled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four faults, all surfaced within an hour of the first live Azure OpenAI call
on 2026-08-27, all invisible under NO_LLM_STUB because the stub supplied the
very fields that turned out to be missing.
1. The classifier few-shot showed eight replies of {"question_class": ...}
alone. A few-shot reply is a shape the model copies, so it omitted
confidence, which defaulted to 0.0, fell below the 0.7 threshold, and EVERY
non-procedural question downgraded to UNCLEAR. The replies now carry the
complete payload the system prompt asks for. Confidences are varied and the
traps carry alternatives: a constant teaches the model to emit that
constant, and the tie rule in apply_safety_rules only has something to work
with if the runners-up are populated.
2. procedure{} was the one part of the procedural payload not assembled from
evidence, contrary to _assemble's own stated rule. The model returned
effective_date "" - neither a date nor None - so ProceduralAnswer rejected
the answer, the single regeneration failed identically, and every procedural
question returned 422.
3. title and authorising_role came back "" for the same reason: the model was
asked for header fields it had never been shown.
4. documented_limits[].citation arrived as the string "WRPS-DEMO-003, Section
4" where a Citation was required, because the schema hint said only
"documented_limits": [] and told the model nothing about the shape.
procedure_identity now takes no `generated` argument at all: there is no path
by which a model can name a revision an operator does not hold. documented_
limits attaches the real Citation by matching source_file against what was
actually retrieved, and DROPS a limit matching nothing - a limit carries the
authority of the document behind it, and misattributing one is worse than
omitting it.
Both live in contracts.py rather than agent.py because they are contract
rules, and because agent.py imports langgraph, which would make the test suite
unrunnable on a bare checkout.
The prompt also now separates two things it was conflating: retrieval
returning nothing (say so and stop) from retrieval returning a document marked
draft, demo or superseded (identify it, quote it, and state the marking).
Including the header chunk made the model read "NOT A CONTROLLED DOCUMENT" and
answer "no controlled procedure was retrieved" while citing one. The marking is
information the operator needs, not a reason to withhold what was found.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two retrieval faults, both only visible once a real model ran.
find_procedure ranked a procedure's chunks by similarity to the question. For
"how do I lift the interlock on Pump 02" the closest chunks ARE the step list -
so the branch whose entire purpose is not reproducing steps was handing the
answer writer nothing but steps, while omitting the header block carrying the
title and the authorising role. The model was being asked for a title it had
never been shown, and returned "".
Once the document is identified, WHICH document it is settles what to send:
the header and the prerequisites, in document order, never the steps.
STEP_SECTION_RE is a second line behind ProceduralAnswer's instruction-language
check, not a replacement for it - the contract still rejects instruction
language whatever arrives here. This removes the temptation rather than relying
on catching it.
Separately, rerank did 0.75 * chunk.similarity where similarity is NULL for a
chunk ingested with --no-embed: `1 - (NULL <=> vec)` is NULL, so it raised
TypeError and 500'd the whole question rather than ranking that chunk last. It
now degrades to the lexical half - an unembedded chunk is still findable, just
not by meaning - and Chunk.similarity is typed honestly as float | None.
find_procedure_lexical takes the same identify-then-expand shape, so the stub
keeps testing the shape it always did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ProcedureIdentity requires a title and an authorising role, and neither was
stored anywhere. The answer writer was asked for both, read them off whatever
chunk retrieval happened to return, and returned "" whenever the header chunk
was not among them.
They belong in the row for the same reason doc_number and revision do: they
are facts about the controlled document, established once when a human
confirms the header, not something to re-derive per question from whatever
text was retrieved. Denormalised onto every chunk exactly as the existing
header fields are - ingest replaces every chunk of a source_file in one
transaction, so they cannot drift within a document.
complete() deliberately still requires only doc_number, revision and
effective_date. A missing title makes an answer less useful; a wrong revision
sends somebody to the wrong document. --assume-yes must keep refusing on the
second and tolerate the first.
controlled_copy_location is NOT in the schema. It is a site fact, identical on
every row, and the one field where an invented value sends a person to a place
that does not exist. It is CONTROLLED_COPY_LOCATION in api.env, defaulting to
a string that names who to ask.
The authorising-role pattern requires the colon: without it the lazy gap
swallowed the field name and captured "role: Station Maintenance Supervisor"
as the value, which the first run caught.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lin001 is 2 vCPU with 3.8 GiB and no swap, shared with 27 other containers.
Cube was rebuilding three pre-aggregations over static fixture data on 10, 10
and 30 minute keys, and its own log shows what that cost:
"Previous interval #19593 was not finished with 30000 interval"
"Interval #19594 finished after 00:02:50"
A refresh taking 2m50s scheduled every 30 seconds, at interval #19594 -
overlapping and never catching up, for days. Every rebuild produced a
byte-identical result, because the fixtures do not change.
24 hours until imh makes the data genuinely live, at which point these get
tuned deliberately rather than left at a number that was never chosen. Cube
also suggests fewer partitions via rollup_lambda; that is a modelling change,
not a demo-day one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Kept, but labelled. ai-api reaches Cube as cube:4000 over the proxy network and
never by name, and the operator UI never touches Cube - the hostname exists so
an engineer can open the playground to hand-verify a measure, which a tunnel
does equally well. So it is the first one to drop if this host is being tidied
or the name is wanted elsewhere.
The note is in three places because whoever tidies this host will be reading
one of them and not this repo: above the block in ~/Caddyfile, inline on the
Authelia rule, and here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Leaving the auth.yokogawa.tech note in verify.sh was a standing nudge towards a
fourth name, which is the opposite of what this host needs. Removed.
In its place, the thing that was actually missing: a list, where the Caddy
blocks live, of which hostname serves what and which one a control-room PC
needs. lin001 is shared and already carries a dozen *.yokogawa.tech services -
the cost of a name is not the DNS change, it is that nobody else can tell what
it is for. Notes that cube.yokogawa.tech is the one nothing depends on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Raised as an item for Dan, which was premature. It blocks exactly one thing: a
demo from a control-room PC. Development and the Phase 8 eval both run from
outside the VNet, where auth.yokogawa.tech resolves normally, and the demo it
would block cannot happen until the Azure OpenAI account lands anyway.
The finding stays in verify.sh as information rather than a failure, so it is
in front of whoever prepares that demo instead of failing every run until then.
Also worth recording: this was inferred from lin001 resolving against 10.0.0.5,
not tested on cicore1. Same DC and same zone, so very likely identical, but it
has not been confirmed on the machine it concerns.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The DC is authoritative for a yokogawa.tech zone holding pinpoint records only,
so from inside the VNet every name not in that zone is NXDOMAIN rather than
forwarded. ai and influx are in it. auth is not.
So an operator on cicore1 resolves ai.yokogawa.tech, reaches Caddy, is
redirected to https://auth.yokogawa.tech/?rd=... and cannot resolve it. The
assistant is reachable and cannot be signed into. Nothing caught this because
influx is the only other pinpointed hostname and LAN traffic to it uses the
bypassed API paths, which never touch the portal - ai.yokogawa.tech is the
first thing here a LAN client must interactively sign in to.
verify.sh now resolves auth.yokogawa.tech alongside the hostnames that redirect
to it, and fails rather than shrugging when it is missing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cube, api and ai joined the HTTPS_UserAccess two_factor rule in one Authelia
restart on 2026-08-27, backup configuration.yml.bak-ai-20260827. All four AI
hostnames now return 302 to the auth portal, and POST /ask on ai.yokogawa.tech
returns 303 to the portal rather than a 404 from ai-web, which is what proves
the route exists.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both found running it on lin001 straight after the Caddy blocks went in.
The public-endpoint check called lf, cube and api unreachable. They are not:
the host resolves yokogawa.tech through the DC, which holds pinpoint records
only, and those three have none. Nothing is wrong with them and the check
cannot tell from here - so it now says so, and says to check from outside the
VNet, instead of printing a failure that means nothing.
403 now has its own message. It is what Caddy plus Authelia return for a
hostname with a site block and no access_control rule, which is the state
between applying the Caddy blocks and applying the Authelia rule - the one
case an operator is most likely to hit and least likely to recognise.
And the log check ran after the agent_ro test, which deliberately attempts a
forbidden INSERT that pg-ai logs. verify.sh was flagging an error line it had
just written itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All three A records resolve to 20.211.144.151 and the DC carries the pinpoint
record ai.yokogawa.tech -> 10.0.0.17, verified from lin001. Kept in the file
as a record of what was asked for and what arrived, with the two things that
came out of it: api and cube have no pinpoint record and do not resolve inside
the VNet, and the names resolving is not the same as the services answering -
the Caddyfile blocks and Authelia rules are still to be applied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
api.yokogawa.tech has a public A record but no pinpoint record on the DC, so
it does not resolve from inside the VNet at all. The browser called it by
hostname, which means an operator on cicore1 would have loaded the page and
had every question fail on DNS - the exact gap Phase 7's gate exists to catch,
and one an engineer's laptop cannot see.
Caddy now routes /ask under ai.yokogawa.tech to ai-api, inside a route block
so import authelia still runs first: forward_auth sorts after handle in the
default directive order, and outside a route the handles would be terminal and
the gate would never run. Only /ask is routed - the Phase 9 publisher rule is
scoped to api.yokogawa.tech and a wider route here would leave it inert.
Also fixes the fallback it replaces. The build arg defaults to "", and
`?? "https://api.yokogawa.tech"` does not catch an empty string, so the
documented real-deployment build resolved the API base to "" and posted /ask
at ai-web, which 404s it. verify.sh and deploy.sh now check both the route and
whether the built bundle carries the hostname.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file says "every key, no values" at the top and was not that, which cost
real time twice on lin001 in one day.
- CUBEJS_DB_* was absent entirely, while ai-compose.yml states that Cube's
database settings come from api.env. Deploying Cube meant reconstructing
what it needed from compose comments and db/003_roles.sql. Added, with the
pg-ai.env names its credentials come from, and a note that Phase 4 turns
the block into an mssql connection against imh.
Also recorded NEGATIVELY: not CUBEJS_EXT_DB_*. Cube v1 refuses Postgres as
an external pre-aggregation store, so someone reading the older compose
file will otherwise try to supply keys for a setting that must not exist.
- NO_LLM_STUB, which is new and defaults to false here for the same reason it
defaults to false in config.py.
- The Langfuse keys were listed but not explained, and every way of getting
them wrong is SILENT:
absent -> _langfuse() returns None, every question untraced
mismatched pair -> a client is built, Langfuse rejects it, and main.py
swallows the exception by design
Neither logs anything, in an answer path that is deliberately built never
to break on observability. The symptom is identical - no traces - so
correcting one cause while the other is still present looks like no
progress at all. That is exactly what happened: a placeholder secret was
pasted alongside a public key from a different pair, and the fix looked
like it had not worked.
So the file now says: they are PROJECT keys from the UI, not the server's
own SALT/NEXTAUTH_SECRET in langfuse.env; they must be a matched pair; the
secret is shown once and stored hashed, so it cannot be read back; and
traces must be confirmed as ARRIVING rather than inferred from config.
No secrets here, including the masked tail of a real key - the example suffix
is invented.
The Langfuse fix itself is not in this commit and cannot be: it was two lines
in ~/ai/api.env, which .gitignore excludes on purpose.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Azure OpenAI is pending and imh is pending, so POST /ask could not return
anything at all - which left the entire chain either side of the model
unproven: the browser, the API, entity resolution, Cube, retrieval, the
contracts, the banners, the error paths. All of it is testable now, and waiting
for a key to find out whether it works is a choice to find out later.
NO_LLM_STUB=true substitutes the two steps that need a model and nothing else.
- Classification: the caller supplies the class, from a dropdown in the UI.
NOT a keyword classifier. A crude keyword classifier produces a PLAUSIBLE
label, and a plausible wrong label is the exact failure this system exists
to prevent - "how do I reset it" landing in Historical is how a synthesised
procedure reaches an operator. Choosing by hand is honest about what is
happening and drives each branch deliberately. apply_safety_rules() still
runs over the result.
- Prose: a fixed placeholder per class, in stub.py.
Everything else is the real path. This is possible because generate() already
kept the factual fields away from the model: rows, counts, citations, the
fixture flag and the class are attached from evidence, and only prose comes
from the generator. Splitting that into _generate_prose() and _assemble() makes
the seam explicit - the stub feeds _assemble() exactly as the model does, so
this is a fair test of the assembly path rather than a mock of it.
The contracts are the point. A stub payload goes through enforce_contract()
unchanged, and it FAILED first time on two classes: the "nothing found" wording
did not match the not-found detectors, so Reference and Procedural returned 422
rather than an uncited answer. That is the contract doing its job against text
no model wrote. Retries are pointless on deterministic output, and a 422 is a
real result here, not a stub bug.
Retrieval is lexical (retrieval.lexical_search), because embedding the question
needs the model. Kept beside search() and never called on the normal path, so
nobody reads a trace and mistakes a lexical hit for a semantic one. It matches
what the operator typed, not what they meant.
What it does not prove: whether the classifier would have labelled correctly -
a person did; whether retrieval finds the RIGHT chunk; and nothing about prose.
It also cannot fill prerequisites_verbatim - extracting them with a regex would
be the "synthesised from fragments" failure the Procedural contract forbids, so
the list is empty and the answer says so.
Every answer carries stub_mode: true in the contract, not decorated on by the
UI, and a banner beside the fixture banner. Same reasoning: an answer nobody
generated must not be indistinguishable from one that was.
Also here:
- demo/ai-docs: three fabricated documents, numbered WRPS-DEMO-00x so header
extraction is genuinely exercised against a number no real WRPS document
can have. Their setpoints contradict tags.csv on purpose.
- VITE_API_BASE build arg, for a tunnelled build before DNS exists. The
tunnel origin is allowed in CORS only while NO_LLM_STUB is on, so it
disappears with the flag. Proxying /api through ai-web's nginx would have
been easier and was rejected: it creates a second route to the API that
bypasses the api.yokogawa.tech Caddy block, where the Phase 9 publisher
rule lives.
Verified on lin001 with no Azure key set at all: all five classes return 200
through the real UI in a browser, over an SSH tunnel, with citations from the
demo documents, real Cube numbers, and both banners showing.
Turning it off: NO_LLM_STUB=false in ~/ai/api.env, restart ai-api.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two blockers stopped Phase 3 being exercised at all before Azure OpenAI exists.
Both are lifted here, and neither weakens the header confirmation - a document
still becomes citable only after a person confirms its number, revision and
effective date.
- parse_markdown() for .md, .markdown and .txt. Docling earns its place on a
PDF: it recovers structure that is not in the bytes, and that structure is
what makes section-boundary chunking possible instead of token-count
splitting that cuts step sequences in half. A Markdown file already
contains "## 2. Prerequisites". Running a document layout model over it
buys nothing and costs the entire torch stack.
It also unblocks the image. docling==2.15.1 does not resolve on
python:3.12-slim: pip backtracks through docling_ibm_models releases for
twenty minutes and exits 2. That is a real problem for Phase 3 and it is
NOT fixed here - PDFs still need Docling and the ai-ingest image still will
not build. Markdown ingestion runs from the ai-api image meanwhile.
Page is 1 for these, because a Markdown file has no pages. A citation to
one carries a section title and no meaningful page number, which is honest.
- --no-embed inserts chunks with a NULL embedding and makes no API call, so
the retrieval path can be exercised before an embeddings deployment exists.
NULL, not a zero vector: a zero vector is a point in the space, it ranks
against real queries, and it would surface as a plausible hit for anything
asked. NULL returns no similarity at all.
These chunks are invisible to vector search and findable only lexically.
The flag warns about that four times on the way past, with the SQL to clear
them, because a database half full of unembeddable chunks looks exactly
like working retrieval right up until the question that matters returns
nothing.
Before real ingestion: DELETE FROM doc_chunks WHERE embedding IS NULL;
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The UI calls the API with credentials: "include", because ai.yokogawa.tech and
api.yokogawa.tech are different origins and the Authelia session cookie has to
be attached explicitly. The CORS middleware never set allow_credentials, and a
browser refuses a credentialed cross-origin request unless the response says
Access-Control-Allow-Credentials: true. It fails at the preflight, so the real
request is never sent:
Access to fetch at '.../ask' has been blocked by CORS policy: the value of
the 'Access-Control-Allow-Credentials' header in the response is '' which
must be 'true' when the request's credentials mode is 'include'.
Every question from the UI would have failed at Phase 7 with "Could not reach
the assistant" - the app's network-error branch, which says nothing about CORS
and points at the wrong layer entirely. The API is fine; curl against it passes,
because curl is not a browser and does not enforce this.
Found driving the built UI in a browser. It is not reachable by any test that
does not involve a browser, which is the useful part: the Phase 7 gate says an
operator reaches the UI and gets an answer end to end, and that gate is the
first thing that would have caught it - at the point where DNS, Caddy and
Authelia are all new too, and any of them a plausible suspect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every Historical and Advisory answer stated a window in AEST and queried one
shifted by ten hours.
rolling_window() builds its boundary strings in SITE_TIMEZONE - that is the
whole point of it, and its docstring says so. metrics.run() then posted the
query to Cube with no timezone at all, and Cube defaults to UTC. So
"2026-08-14T15:22:13" meant 15:22 Sydney to the code that produced it and 15:22
UTC to the engine that ran it, and MetricResult.time_window reported
SITE_TIMEZONE from config rather than whatever the query actually used, so the
two could not disagree visibly.
Measured on the fixtures, same dateRange, one field changed:
timezone UTC 8019 samples
timezone Australia/Sydney 8619 samples
600 samples. One per minute, ten hours, exactly the offset.
Nothing about the answer looked wrong. The prose was right, the count was a
real count, the window description was correctly formatted and correctly named
AEST. It was only visible by reading the Cube query in the UI's "show working"
panel - which is an argument for that panel existing, and an argument for
looking at the thing in a browser rather than trusting curl against the API.
- check_cube_query() now takes site_timezone and pins it onto the query, at
the single point every Cube query passes through. Per-query-builder is the
wrong place: "remember to set the timezone" is not a control, and this
defect is what forgetting looks like. An explicit timezone already on the
query is left alone.
- time_window now reports capped["timezone"] - the timezone the query ran in,
not the one it should have run in.
An unpinned timezone belongs in the same guardrail as an unpinned date range,
and for the same reason: both make an answer unreproducible. The difference is
that an unpinned date range is obvious in the query and an unpinned timezone
is invisible.
eval case H28 records it. Two unit tests: the timezone is pinned, and an
explicit one is not overridden.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both were found hand-verifying the measures on fixtures, and both are left
unfixed deliberately: fixing either now means guessing at what imh contains,
and a guess baked into the seed data or the models is harder to find later than
an open finding. Flagged at Phase 4, in the README, in the model files at the
point of use, and as eval cases that fail until they are settled.
(a) The wet well level tag does not join, and fails as "no records found".
History is keyed PS_STN_WET_WELL_LEVEL, which process_values.yml hardcodes
in seconds_above_high_level_alarm and seconds_above_lshh. tags.csv carries
that name only as an ALIAS of LIT-101, so public.tags has no row with that
tag_id and all 43,201 level rows - a third of the history, on the most
important tag at this station - are unreachable from a tag-level lookup.
Resolve "wet well" -> WW-101 -> LIT-101 -> filter history on LIT-101 ->
zero rows -> "no records found".
That is the failure mode worth being loudest about: it is the safety
behaviour, produced by a key mismatch rather than by an absence of data,
and indistinguishable from the real thing on screen. Filtering by
equipment_id works, so whether a level question fails depends on which path
the agent takes.
The two flow tags use the opposite, self-consistent convention -
PS_STN_INFLOW is a row in its own right and FIT-201 is marked NOT
HISTORISED - so applying that to level is the likely fix. It still waits
for the register map and for imh to say what CI Server historises the point
as. Seed, hardcoded model names and 002_fixtures.sql change together.
(b) alarms.first_alarm and last_alarm return UTC, not SITE_TIMEZONE.
Cube converts time dimensions to the query timezone; a min/max measure over
a timestamp comes back unconverted. The Sydney day bucket for 2026-08-01
returns 2026-07-31T20:00:35 - the right instant, ten hours and one calendar
day out, beside a bucket label that IS in site time.
This breaks "convert to SITE_TIMEZONE exactly once, in Cube", and the fix
has to stay in Cube - the API compensating with timezone arithmetic is the
thing that rule exists to prevent. Which fix is right depends on whether
imh stores UTC or local, which is Phase 4 task 4. Until then these two must
not be quoted to an operator as a clock time.
Phase 4's gate gains an item for each. Everything else verified in this pass -
alarms, operations, the equipment join, alias resolution and the Sydney/UTC day
boundaries - matched hand-written SQL exactly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hand-verifying the measures against the fixtures on lin001, per the Phase 5
gate. Two of them had never executed anywhere, and both failed outright rather
than returning a wrong number - which is the good version of this, but they
failed at the point an operator asks a question, not at review.
- time_weighted_avg put LEAD() inside SUM(). Postgres rejects that flatly:
"aggregate function calls cannot contain window function calls". The per
sample duration now comes from the cube's source query, which changes
sql_table to sql, and the measure just sums value * duration over duration.
The last sample of each tag gets a NULL duration and SUM skips it, which is
correct - how long it stood is not yet known.
This is the measure that matters most later. On the regular one-minute
fixtures it agrees with avg_value to thirteen decimal places
(42.45934027777778 against 42.45934027777775), which proves it is wired up
and proves nothing about imh, where the deadband makes samples irregular
and the two will not agree. Re-verify it there.
- p95_value applied the quality filter through a Cube measure `filters:`
block, which lands outside the aggregate and cannot work on an ordered-set
aggregate: "column process_values.quality must appear in the GROUP BY
clause". Folded into the CASE inside PERCENTILE_CONT, whose NULL handling
does the exclusion.
Also: the priority dimension said only SPILL and PUMP_TRIP are priority 1,
while the data has LEVEL_SIGNAL_FAULT at priority 1 too - correctly, losing the
level signal on a well that can spill is a priority 1 condition. That comment is
the line an engineer reads when checking a priority_1_count, so it disagreeing
with the data matters more than its length suggests.
eval cases H24 and H25 record the two failures, added before the fix.
Verified against hand-written SQL on the same pinned windows: p95_value
61.104999999999976 and time_weighted_avg 42.45934027777778 both match to the
floating point tail, as do sample_count, avg_value, max_value, min_value,
seconds_above_high_level_alarm (7680 = 128 samples x 60) and every measure in
alarms and operations.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 5 deployed to lin001 for the first time, and the cube service as
committed could not answer a single query. Three separate faults, none of them
visible without deploying it.
- CUBEJS_EXT_DB_TYPE: postgres is not supported in Cube v1. Cube Store is the
only external pre-aggregation store, and it is also the default cache and
queue driver, so naming Postgres failed EVERY query - not just
pre-aggregated ones - with "It`s not possible to use Cube Store as
queue/cache driver without using it as external". So cubestore is now a
service: pinned in lockstep with cube, ai-internal only, no ports, data on
/datadisk because it grows and / is 62 GB.
CUBEJS_CACHE_AND_QUEUE_DRIVER: memory is NOT a way out. It does not fall
back - it hangs /readyz and every query indefinitely, logging nothing at
level warn. That cost longer to diagnose than the original error.
This is a deviation from the build spec, which says pre-aggregations
materialise into pg-ai schema cube_preagg. They cannot, on this version.
cube_preagg and its grants in 003_roles.sql stay, unused, so that nothing
else has to change if a later Cube restores Postgres as an external store.
- CUBEJS_REFRESH_WORKER was never set, so nothing built the pre-aggregations.
A query matching a rollup does not fall back to the source: it fails with
"No pre-aggregation partitions were built yet". max_value, min_value and
sample_count were dead on arrival while avg_value worked, which reads as a
per-measure bug and is not one.
- The healthcheck ran wget, which is not in the image (nor is curl). A
perfectly healthy cube reported unhealthy on every deploy, training the
reader to ignore the one signal that would show a real fault. It uses node,
which the image does have.
Verified on lin001: cube healthy, no published host ports, ai-internal and
proxy only, and a query matching alarms_by_hour now returns external: true -
served from Cube Store rather than scanning the source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ingest_file() inserted every chunk with superseded = FALSE, so replacing a
document's chunks reset its withdrawal. One `ai-ingest --all` made every
superseded revision citable again - including the old revision of a procedure -
silently, and only after a bulk run, so a supersede survived exactly until the
next full ingest.
- ingest_file() reads the existing state with superseded_state() before the
delete and carries it through the insert. A withdrawn document is
re-ingested as withdrawn, and logs that it did so.
- --all skips withdrawn documents, so a bulk run does not spend an embeddings
call on a document that will not be cited either way. --include-superseded
overrides it; the chunks still come back withdrawn.
- --restore DOC_NUMBER REVISION is the counterpart to --supersede, refusing
while another revision of the same document is live. Without it the
conservative default would be a dead end - mark_superseded() only ever sets
TRUE, so there was no way back.
- Rule 5 in the module docstring, beside the other four.
Corrects a claim in the Phase 9 design: 16.10 said moving the withdrawn file
out of /datadisk/ai-docs "is not tidying" because otherwise --all resurrects it.
That was true when written and is not now. The guard belongs in the ingest code,
because a rule that depends on somebody remembering to move a file is not a
rule. The move stays as archival housekeeping and is documented as such in
16.10, db/005_doc_actions.sql and README.md.
Phase 3's gate gains the proof: supersede a revision, run --all again, ask the
question that used to cite it.
Not executed anywhere - no Postgres or Docker on this machine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ai-ingest built its DSN from PGUSER/PGPASSWORD and takes its environment from
~/ai/api.env, where PGUSER=agent_ro - SELECT and nothing else, deliberately,
because it is what the answer path runs as. So
docker compose -f ~/ai-compose.yml run --rm ai-ingest --all
connected as a role that cannot INSERT INTO doc_chunks, and Phase 3 was
unrunnable exactly as the README documents it. Nothing had reached Phase 3 yet,
so nobody had hit it.
The failure would also have landed at the worst possible moment: at the final
INSERT, after the Docling parse, after a person had typed the header
confirmations for every file, and after a billed embeddings call - with a
permission error naming no cause.
- ingest_rw moves to 003_roles.sql, at Phase 1 with the other roles. It is
not a Phase 9 concept; ingestion has needed a writing role since Phase 3
and never had one. 004 keeps only its grants on the upload queue, and its
idempotent role creation so it still applies to an older database.
- ingest.py connects through INGEST_DB_USER / INGEST_DB_PASSWORD, falling
back to PGUSER only for a local shell where one pair is set.
- require_write_access() checks INSERT, UPDATE and DELETE on doc_chunks
before anything is parsed or embedded, and fails with the fix in the
message. Falling back to PGUSER cannot smuggle agent_ro past it.
- Keyword connection parameters rather than a URL: a generated password
containing @ or / breaks a DSN string silently.
- A missing doc_chunks now says "apply 001_schema.sql" instead of raising
UndefinedTable.
Phase 1's gate gains the check that would have caught this: ingest_rw must be
able to write doc_chunks. An ingestion role that cannot write is the same class
of failure as an API role that can - it just surfaces two phases later.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The design commit added ai-docs-worker with no profile and an inbox mount on
ai-api. Both take effect the moment anyone deploys, and neither has the code or
the directories behind it yet:
- ai-compose.yml's own header says to run a bare `up -d`. That would start
ai-docs-worker with entrypoint `python worker.py` - a file that does not
exist - under restart: unless-stopped, so it crash-loops on a live shared
host that also runs the demo PLC.
- Docker creates a missing bind source as a ROOT-OWNED directory. ai-api does
not run as root, so deploying the inbox mount before the directory exists
gives the API an inbox it cannot write to, on the growing disk.
So: profiles: [worker] on the worker, matching the ai-ingest precedent, and the
ai-api volume block commented out with the install command beside it - the same
"add each piece at the phase that needs it" convention as caddy/ai-routes.caddy.
A bare `up -d` from main now starts exactly what it started before: pg-ai, cube,
ai-api, ai-web.
Both guards come off in the commit that adds worker.py.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A single-page explainer for people who will not read the build spec: how a
question becomes an answer, the four lanes and why they are separate, the four
tools the assistant may reach for, where the knowledge comes from, and what is
live against what is only built. Status snapshot as observed on the host,
21 August 2026.
Kept at the repository root beside the other narrative documents rather than in
docs/, which is gitignored and holds controlled plant documents. Listed in the
layout sections of README.md and the build spec so it is findable.
Note: it predates Phase 9 and so does not mention operator document upload,
withdrawal or pool curation. Its step table stops at 8.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operators cannot add a document today: ingestion is CLI-only, needs a host
login and a TTY for confirm_header(), /datadisk/ai-docs is mounted read-only,
ai-api has no identity, and nothing in the stack has a role that can write
doc_chunks. This designs the way in, the way out, and control over what is in
the retrieval pool. Design and schema only - no router, worker or UI code yet.
Documents in (16.1-16.9, db/004):
upload -> pre-scan -> review -> approve -> published, with the header
confirmation moved from a terminal prompt to a review screen and recorded
rather than discarded. A CHECK constraint refuses an approved row without a
confirmed number, revision and effective date, so an API bug cannot skip it.
Three roles: agent_ro unchanged, uploads_rw writes the queue only, ingest_rw
writes doc_chunks and has no HTTP surface.
Documents out (16.10-16.11, db/005):
--supersede needs a revision to keep, so a cancelled procedure cannot be
withdrawn at all. Adds withdraw (immediate, reversible, audited), restore
(refused while another revision is live) and purge (off by default). A
column grant plus a trigger let the web-facing role make a document less
citable and never more.
The pool (16.13-16.15, db/006):
pool_enabled, orthogonal to superseded: one is a claim about the document,
the other about the corpus. Retrieval requires both, so re-enabling a
withdrawn document does not make it citable. Named profiles and a
per-request override let a demo trim the corpus without mutating state on a
shared live host, and every reduced-pool answer carries a banner with the
document count, following the used_fixture_data precedent.
Two existing defects found and documented while designing this:
- ai-ingest takes PGUSER=agent_ro from api.env, a SELECT-only role, so the
Phase 3 command in the README cannot write doc_chunks (16.1).
- ingest_file() always inserts superseded = FALSE, so `--all` re-ingests a
superseded revision as live. --supersede survives only until the next bulk
run (16.10).
One commit rather than three: the upload, withdrawal and pool designs
interleave in the same spec, README and compose files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bind address, and the reason this matters most. Langfuse is a Next.js
standalone server: it binds to whatever $HOSTNAME resolves to, not to all
interfaces. The container is on two networks and its hostname resolved to the
proxy address, so it listened on proxy only - HTTP 200 from proxy, HTTP 000
from ai-internal. Caddy is on proxy, so the public URL worked perfectly and
nothing looked wrong, but ai-api runs on ai-internal and every trace it sent
would have failed silently in Phase 6. HOSTNAME=0.0.0.0 fixes it; both
networks now return 200.
External network name. ai-internal does not exist - Compose created it as
azureuser_ai-internal under the host's shared project name. Verified both ways
on the host: with name: a container joins and reaches pg-ai, without it up
fails with "network ai-internal declared as external, but could not be found".
Healthcheck. Was probing localhost, which failed for the bind reason above and
marked the container unhealthy while it served fine. Restored to 127.0.0.1,
which now works because the bind is genuinely fixed rather than worked around.
Signup policy. AUTH_DISABLE_SIGNUP false. Langfuse keeps its own user table
behind Authelia; closed signup with no seeded user means nobody can log in at
all. Accepted for the PoC because Authelia is the real gate - revisit before
handover.
Phase 2 gate passed on the host with these applied: 302 to the auth portal,
certificate issued, AD + Duo login confirmed, and a test trace verified in the
database rather than trusted from the API response.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
agent_ro could still create temporary tables: Postgres grants TEMP on every
database to PUBLIC, and revoking it from the role alone does not remove what
PUBLIC grants. Revoke it from PUBLIC too. postgres is a superuser and keeps
its temp tables, so the seed load is unaffected.
Nothing but the superuser could read schema fixture: 003_roles.sql grants only
on schema public, and the fixture schema is created afterwards. Grant read to
agent_ro and cube_rw. The grants live in 002_fixtures.sql because that file
opens with DROP SCHEMA fixture CASCADE, which would destroy grants held
anywhere else on every fixture reload.
Both re-verified on pg-ai: 11/11 privilege tests and 10/10 fixture access
tests pass, reads work for both roles, writes are rejected for both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Verified on lin001 over SSH on 2026-08-20, read-only:
- Port 502 is bound to 10.0.0.17, not 0.0.0.0, so unauthenticated Modbus is
not internet-reachable at the Docker level. This was an open risk in §3 and
an unchecked item in §15; it is now a confirmation. openplc-runtime also
publishes 8443 (the Runtime web UI) on the same private address, which the
spec did not mention.
- openplc-runtime is not the only published port on the host: caddy, wireguard,
mosquitto and chirpstack-gateway-bridge all publish on 0.0.0.0. The
no-published-ports rule still applies in full to what we build, but the
"one deliberate exception" framing was wrong and invited over-reading.
- Port 22 is open to the internet. Added to §15 as an open item for the same
NSG review.
- 21 containers running, not 22; no stopped containers.
- /datadisk is 46% used with InfluxDB at 55 GB, up from 43%/52 GB. Recorded
the growth rate so it can be budgeted for.
CLAUDE.md restates two of these rules and is updated to match.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Authored on Windows, deployed to Ubuntu: a CRLF shell script fails on the
host with an unhelpful error.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Build spec and host brief carried in from C:\Claude and WRPS/02-env; the
plant model (equipment, tags, alarm bitmask, enums, unit conversions) is
derived from WRPS/04-plc/register-map.csv, WRPS/05-scada/modbus/scada-points.csv
and WRPS-CTL-003.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
