Most of this page was read off the running host on 28 August 2026 — containers listed, endpoints probed, the database queried, and two questions put through the live assistant end to end. Nothing here is taken from the plan. Where this page and the build documents disagree, this page is the later reading.
Updated 31 August. The three open findings are closed. They turned out to be one defect wearing three faces, and it was settled not by choosing between the two disagreeing sides but by going to the SCADA configuration and finding that both were describing something the historian does not use. The section that used to list them now records what was actually wrong.
Three servers and one cloud service. Two of the three servers exist; the third is the whole of what this project is still waiting for.
Nine phases, each ending in a gate that must pass before the next begins — so that when an answer comes out wrong there is one place to look, not four. Two gates have never been run, and one phase was built out of order.
cicore1, the SCADA machine, end
to end. Both halves of the access rule are confirmed: the deny half refuses everybody it
should, and the admit half has now been witnessed from the console itself.
ai-web’s access log records the real client address, and it shows
10.0.0.21 loading the page on 28 August, the day the rule was applied, and
three times since. Questions followed on 31 August: three POST /ask calls returned
200, the first 21 seconds after the console loaded the page. An operator opened the screen,
asked, and got an answer, with no sign-in.Every row read from the host today. “Up” is not proof — each public address was probed and each claim checked against something other than the container list.
| Component | Host & network | Reachable at | What proves it works |
|---|---|---|---|
| pg-ai | lin001 · ai-internal only /datadisk/pg-ai |
nothing — no host port | Healthy 8 days. 92 document chunks queried by hand; the read-only role proven unable to write. |
| cube | lin001 · ai-internal + proxy | cube.yokogawa.tech → 302 | Healthy 7 h and answering the data lane — but off stand-in tables, not the historian. |
| cubestore | lin001 · ai-internal /datadisk/cubestore |
nothing — no host port | Up 7 days. A hard dependency of cube, not an optimisation — cube will not start without it. Absent from the original architecture deck. |
| ai-api | lin001 · ai-internal + proxy 0600 env file |
api.yokogawa.tech → 302 and /ask under ai.… |
Healthy 2 h. A procedural question answered end to end in 3.96 s: correct document identity, prerequisites quoted, steps withheld, scope banner present. |
| ai-web | lin001 · proxy · static build | ai.yokogawa.tech → 403 | Up 26 h. The 403 is the deny arm working correctly. The allow arm is now proven from the console itself — the access log shows 10.0.0.21 since 28 August. |
| langfuse + lf-db | lin001 · ai-internal + proxy /datadisk/langfuse |
lf.yokogawa.tech → 302 | Both healthy 7 days. Every question traced — anonymously, since 28 August. |
| ai-ingest | lin001 · ai-internal profiles: [ingest] |
nothing — batch job | Not running, by design. The terminal ingest path, started by hand when documents change. |
| ai-docs-worker | lin001 · ai-internal profiles: [worker] |
nothing | Not running and not deployed. The upload path was built inside ai-api instead, so conversion and embedding happen in the request. |
| openplc-runtime | lin001 · host ports 502, 8443 | Modbus TCP, bound 10.0.0.17 | Up 8 days. Live control for this demo. The one deliberate exception to the no-published-ports rule, contained by that binding plus firewall scope. Never restarted as a side effect of AI work. |
| svc_agent_ro | yau-sls-poc-imh · TDS 1433 | not reachable | Does not exist. The single thing this project is still waiting on from anybody. |
ai.yokogawa.tech has an internal record pointing at the host and resolves.
api, cube, lf and — the one that bit —
auth do not. All four answer correctly when reached; they simply cannot be
reached from the plant floor.
Why it went unnoticed: the equipment that talks to this host writes straight to the database over a path that skips sign-in entirely. No browser had ever opened the sign-in page from inside the network. It no longer affects the operator, who does not sign in — but it will catch the next protected service anyone tries to open from the plant floor.
Because api.yokogawa.tech cannot be resolved from a control-room PC, the
operator’s page would have loaded perfectly and then failed on every single question. So
the question route is served under the same address as the page itself, and the browser never
names the assistant.
Only the question route is shared. Widening it would put the document
library on an address that carries no identity at all — which is exactly what the
access change of 28 August made ai.yokogawa.tech.
They were not three problems. They were one substitution, showing through in three places: the stand-in historian was keyed on names the historian does not use. The earlier plan was to reconcile the tag set against the register map. That would not have found this, because the register map is not where the answer lives either.
Four different names describe the wet well level, and only the fourth is what the historian stores. The instrument on the drawing is LIT-101. The PLC publishes it at %QW0. SCADA polls that register and calls the point WRPS_PLC:STN_LEVEL — the Modbus station name, then the point name. And then CI Server files the history under an item name, AID.WRPS.STN.LEVEL — which appeared nowhere in this project’s repository, because nothing had ever needed it before. Worse, the point list handed over at the time carried invented names that CI Server never used; a corrected one arrived on 1 September and the whole repository is keyed on the item now.
Modbus carries register numbers, not names. That is why the point layer and the item layer are free to drift apart, and why reconciling against the register map would have proved only that the first three agreed with each other. The stand-in was built on the third name. The real historian answers to the fourth.
Confirmed and fixed. The level history is now keyed on the item the historian actually uses, and LIT-101 is marked as what it is — a field input to the PLC that never reaches SCADA at all. It was the only row in the whole tag list carrying two addresses, which is the instrument and the published value quietly merged into one.
Why it will not come back: the mapping from item to tag is generated from the SCADA files, not typed. An item that resolves to neither a tag nor a written reason for having none now fails the build, fails the deploy and fails the verify script. The old failure mode — a join matching nothing and reporting “no records found” — cannot be reached silently any more.
eval case H26 rewritten · H30 added
Fixed inside the translator, where it had to be. The conversion now happens in the measure itself, and it happens in the right order — the earliest instant is found first and converted afterwards. Doing it the other way round takes the earliest clock reading, which picks the wrong record on the night the clocks go back and one local hour happens twice.
The answer came from a file, not a meeting. This was waiting on confirmation of whether the historian stores UTC or local time. Every one of the 49 SCADA points is configured Date+time GMT, and every history group has daylight correction off. The time is also no longer handed out as a bare timestamp: it comes with the timezone beside it, because a clock time with no zone is exactly what let this hide for a week.
eval case H27 rewritten
Both sides were right, which is why neither could win. The tag list said the station, because that is the section CI Server files the item under. The stand-in said the wet well, because that is what the alarm is about. The defect was never which one was correct — it was that the same fact was being asserted twice.
Fixed by removing the second assertion. The history now carries no equipment column at all, which also happens to be faithful: CI Server’s equipment tree stops at the station and the three pumps, and has no wet well to put there. Equipment is stated once, in the tag list, and reached from an alarm through the alarm bit that raised it. The verify script fails if an equipment column ever reappears in the history.
eval case H31 added, expected count pinned at 14
The historian keeps seven days, not thirty. The stand-in had been generating a month. Every question about last month worked here and would have failed the moment it was pointed at the real thing. The stand-in now keeps seven days too, so that failure happens where it can be seen. The system distinguishes “the historian does not go back that far” from “nothing happened” — different answers, and only one of them true. Extending retention is now a written request, because an assistant that cannot answer “last month” is of limited use.
The level is sampled every five seconds, not every minute. Two measures turned a count of samples into a duration by multiplying by sixty. Against a five-second signal that overstates by twelve times — and it would have read as an entirely plausible number. They now read the interval from the item rather than assuming one.
The warning about compressed data was wrong. The translator carried a prominent note saying real history would arrive irregularly spaced, so ordinary averages could not be trusted. The SCADA configuration has compression switched off on every group. The note was steering people away from the correct measure, and it has been corrected.
The stand-in now checks itself: it asserts its own alarm counts as it loads, and proves them twice over by deriving the same figure from two independent signals and refusing to load if they disagree. That is a test of the pipeline, not a fact about the plant. The number this system currently gives for high level alarms is a fact about generated data and nothing else.
Two things still need a person. An engineer has to confirm the first real figures by hand once the historian is connected. And the SCADA repository and the live server disagree about sample rates — the checked-in file says one second, the running system says five. The running system was taken as correct here, but that means the repository does not currently describe the machine, and somebody should decide which of the two is wrong.
Twenty-eight containers on one shared host. Seven are ours; twenty-one were already here and serve other demos. This is the list the architecture deck never carried.
The store. Holds the searchable text of every controlled document, the equipment and tag list that turns “Pump 02” into a real tag, the upload queue, the permanent audit trail, and — until the historian arrives — the stand-in plant history.
Five roles, deliberately split: the answer path can only read, only the ingestion role can write documents, and a database trigger stops the web role un-withdrawing anything.
The data translator. Turns “high level alarms last week” into an exact query over recorded history. The definitions that make an answer right live in files an engineer can read and check, rather than being invented per question.
The assistant never writes database code itself — it fills in a request form and Cube does the rest. Today it reads stand-in tables, so no number it produces means anything about the plant, and every answer built on one says so on screen.
Cube’s own storage engine. Holds the queue and the ready-made summaries that keep “count last week” fast without repeatedly scanning the historian.
Not an optimisation and not optional — Cube will not start without it. It is missing from the original architecture deck and from the build documents’ container table, which is why it is called out here.
The assistant itself, and the busiest thing here. It sorts each question into one of four kinds, resolves the plant names in it, sends it down the matching lane, gathers the evidence, has the model word an answer, and then checks that answer against a contract written in code before letting it out.
Since 28 August it also serves the document library — upload, review, approve, withdraw and restore — because the operator’s screen no longer carries any identity.
The operator’s screen. A question box, the answer, and a “show working” panel giving the question’s kind, the query that ran, the row count, and every citation with its revision and effective date.
It carries two banners an operator cannot switch off: one saying the figures are stand-ins, and one on procedural and advisory answers saying what the assistant deliberately did not do.
The logbook. Records every question: how it was sorted and with what confidence, which tools ran, which passages were retrieved, how many tokens it cost, how long it took, and whether the contract passed or failed. Every rejection is logged with the offending output.
Deployed early on purpose, so no experiment since has been untraceable. Self-hosted and MIT-licensed — no licence cost, and no question text leaves the host to reach it.
The terminal path for loading documents, started by hand when documents change. Reads a file, splits it on section boundaries so a numbered step sequence is never cut in half, asks a person at the keyboard to confirm the document number, revision and effective date, and only then stores it.
It cannot see anything loaded through the upload screen — published files stay in the upload area rather than moving into the document tree. The two paths must not be used on the same document.
Designed as a background worker that would pre-scan uploads, ingest approved ones and complete withdrawals, kept off the network entirely so that the web-facing part of the system could never write documents directly.
Not built. That work happens inside ai-api instead, so the same process holds both permissions. The database trigger still stops the web role un-withdrawing anything, so the boundary holds — but it is now a code boundary rather than a deployment one.
This host has served demos for months. Restarting the first two logs out every active user, including whoever is mid-demo.
The only front door. Every screen on this host is reached through it, and it obtains and renews the certificates automatically. Nothing this project built publishes a port of its own — they all sit behind it on a shared internal network.
Sign-in for the whole host: Active Directory plus a phone prompt. Omitting it from a site block silently makes that service public, which is why every block here carries it — with the one deliberate exception of the operator’s screen.
The PLC for this demo plant. It runs the actual control logic for the three pumps and the wet well; the SCADA server polls it over Modbus and historises the result. Live control, not a simulation of one.
The host’s main time-series store, used by the other demos on this box. Nothing this project built reads or writes it. It is listed here because it is effectively the only consumer of the shared data disk, growing roughly a gigabyte a week — so it, not us, is what will eventually fill it.
The internal Git server, and the master copy of this project’s repository. It sits outside the shared sign-in because forwarding authentication breaks Git clients — a reason that applies to nothing else here.
Device data arrives over MQTT into mosquitto, is processed by nodered, and is stored in InfluxDB; telegraf adds host and container metrics; and grafana draws the dashboards over the top. None of it touches the assistant.
portainer is graphical Docker management and is root-equivalent; dozzle is live searchable container logs and the best first stop when something here misbehaves; watchtower auto-updates a safe subset of images on Sunday mornings.
wireguard is the VPN through which engineers reach this host at all; showroom is a static demo site; eqp-licence issues licences with a signing key mounted read-only. None is part of the assistant.
A LoRaWAN network server and its supporting containers. Running but not configured — a future capability for this host, unrelated to the assistant. Safe to ignore.
Documented rather than hidden, and none of them to be shipped. Production closes them in this order: network separation, then secrets, then database guardrails, then document control integration, then resilience.
What the assistant will and will not say must be reviewed with an OT or safety representative. That review is still outstanding, and it now has to cover the unauthenticated console as well. Reviewing it was only meaningful once the model was in place — which it now is, so nothing stands in the way of booking it.