Most of this page was read off the running host on 28 August 2026 + — containers listed, endpoints probed, the database queried, and two questions put through the + live assistant end to end. Nothing here is taken from the plan. Where this page and the build + documents disagree, this page is the later reading.
+Updated 31 August. The three open findings are + closed. They turned out to be one defect wearing three faces, and it was settled not by choosing + between the two disagreeing sides but by going to the SCADA configuration and finding that + both were describing something the historian does not use. The section that used to list + them now records what was actually wrong.
+ +Three servers and one cloud service. Two of the three servers exist; the third + is the whole of what this project is still waiting for.
+ +Nine phases, each ending in a gate that must pass before the next begins — + so that when an answer comes out wrong there is one place to look, not four. Two gates have never + been run, and one phase was built out of order.
+ +Every row read from the host today. “Up” is not proof — + each public address was probed and each claim checked against something other than the container + list.
+ +| Component | Host & network | Reachable at | What proves it works | +
|---|---|---|---|
| pg-ai | +lin001 · ai-internal only /datadisk/pg-ai |
+ nothing — no host port | +Healthy 8 days. 92 document chunks queried by hand; the read-only role proven unable to write. |
| cube | +lin001 · ai-internal + proxy | +cube.yokogawa.tech → 302 | +Healthy 7 h and answering the data lane — but off stand-in tables, not the historian. |
| cubestore | +lin001 · ai-internal /datadisk/cubestore |
+ nothing — no host port | +Up 7 days. A hard dependency of cube, not an optimisation — cube will not start without it. Absent from the original architecture deck. |
| ai-api | +lin001 · ai-internal + proxy 0600 env file |
+ api.yokogawa.tech → 302 and /ask under ai.… |
+ Healthy 2 h. A procedural question answered end to end in 3.96 s: correct document identity, prerequisites quoted, steps withheld, scope banner present. |
| ai-web | +lin001 · proxy · static build | +ai.yokogawa.tech → 403 | +Up 26 h. The 403 is the deny arm working correctly. The allow arm is unproven and cannot be proven from here. |
| langfuse + lf-db | +lin001 · ai-internal + proxy /datadisk/langfuse |
+ lf.yokogawa.tech → 302 | +Both healthy 7 days. Every question traced — anonymously, since 28 August. |
| ai-ingest | +lin001 · ai-internal profiles: [ingest] |
+ nothing — batch job | +Not running, by design. The terminal ingest path, started by hand when documents change. |
| ai-docs-worker | +lin001 · ai-internal profiles: [worker] |
+ nothing | +Not running and not deployed. The upload path was built inside ai-api instead, so conversion and embedding happen in the request. |
| openplc-runtime | +lin001 · host ports 502, 8443 | +Modbus TCP, bound 10.0.0.17 | +Up 8 days. Live control for this demo. The one deliberate exception to the no-published-ports rule, contained by that binding plus firewall scope. Never restarted as a side effect of AI work. |
| svc_agent_ro | +yau-sls-poc-imh · TDS 1433 | +not reachable | +Does not exist. The single thing this project is still waiting on from anybody. |
ai.yokogawa.tech has an internal record pointing at the host and resolves.
+ api, cube, lf and — the one that bit —
+ auth do not. All four answer correctly when reached; they simply cannot be
+ reached from the plant floor.
Why it went unnoticed: the equipment that talks to this host writes + straight to the database over a path that skips sign-in entirely. No browser had ever opened + the sign-in page from inside the network. It no longer affects the operator, who does not sign + in — but it will catch the next protected service anyone tries to open from the plant floor.
+Because api.yokogawa.tech cannot be resolved from a control-room PC, the
+ operator’s page would have loaded perfectly and then failed on every single question. So
+ the question route is served under the same address as the page itself, and the browser never
+ names the assistant.
Only the question route is shared. Widening it would put the document
+ library on an address that carries no identity at all — which is exactly what the
+ access change of 28 August made ai.yokogawa.tech.
They were not three problems. They were one substitution, showing through in + three places: the stand-in historian was keyed on names the historian does not use. The + earlier plan was to reconcile the tag set against the register map. That would not have found + this, because the register map is not where the answer lives either.
+ +Four different names describe the wet well level, and only the fourth is what the + historian stores. The instrument on the drawing is LIT-101. The + PLC publishes it at %QW0. SCADA polls that register and calls the + point PS_STN_WET_WELL_LEVEL. And then CI Server files the history + under an item name, AID.WRPS.STN.LEVEL — which appears + nowhere in this project’s repository, because nothing had ever needed it before.
+Modbus carries register numbers, not names. That is why the point layer and the item + layer are free to drift apart, and why reconciling against the register map would have proved + only that the first three agreed with each other. The stand-in was built on the third name. The + real historian answers to the fourth.
+Confirmed and fixed. The level history is now keyed on the item the historian actually + uses, and LIT-101 is marked as what it is — a field input to + the PLC that never reaches SCADA at all. It was the only row in the whole tag list carrying + two addresses, which is the instrument and the published value quietly merged into one.
+Why it will not come back: the mapping from item to tag is generated + from the SCADA files, not typed. An item that resolves to neither a tag nor a written reason + for having none now fails the build, fails the deploy and fails the verify script. The old + failure mode — a join matching nothing and reporting “no records found” + — cannot be reached silently any more.
+eval case H26 rewritten · H30 added
+Fixed inside the translator, where it had to be. The conversion now happens in the + measure itself, and it happens in the right order — the earliest instant is found + first and converted afterwards. Doing it the other way round takes the earliest + clock reading, which picks the wrong record on the night the clocks go back and one + local hour happens twice.
+The answer came from a file, not a meeting. This was waiting on + confirmation of whether the historian stores UTC or local time. Every one of the 49 SCADA + points is configured Date+time GMT, and every history group has + daylight correction off. The time is also no longer handed out as a bare timestamp: it comes + with the timezone beside it, because a clock time with no zone is exactly what let this hide + for a week.
+eval case H27 rewritten
+Both sides were right, which is why neither could win. The tag list said the + station, because that is the section CI Server files the item under. The stand-in said the + wet well, because that is what the alarm is about. The defect was never which one was + correct — it was that the same fact was being asserted twice.
+Fixed by removing the second assertion. The history now carries no + equipment column at all, which also happens to be faithful: CI Server’s equipment tree + stops at the station and the three pumps, and has no wet well to put there. Equipment is + stated once, in the tag list, and reached from an alarm through the alarm bit that raised it. + The verify script fails if an equipment column ever reappears in the history.
+eval case H31 added, expected count pinned at 14
+The historian keeps seven days, not thirty. The stand-in had been generating a month. + Every question about last month worked here and would have failed the moment it was pointed at + the real thing. The stand-in now keeps seven days too, so that failure happens where it can be + seen. The system distinguishes “the historian does not go back that far” + from “nothing happened” — different answers, and only one of them + true. Extending retention is now a written request, because an assistant that cannot + answer “last month” is of limited use.
+The level is sampled every five seconds, not every minute. Two measures turned a count + of samples into a duration by multiplying by sixty. Against a five-second signal that overstates + by twelve times — and it would have read as an entirely plausible number. They now + read the interval from the item rather than assuming one.
+The warning about compressed data was wrong. The translator carried a prominent note + saying real history would arrive irregularly spaced, so ordinary averages could not be trusted. + The SCADA configuration has compression switched off on every group. The note was steering + people away from the correct measure, and it has been corrected.
+The stand-in now checks itself: it asserts its own alarm counts as it loads, and proves them + twice over by deriving the same figure from two independent signals and refusing to load if they + disagree. That is a test of the pipeline, not a fact about the plant. The number this + system currently gives for high level alarms is a fact about generated data and nothing else.
+Two things still need a person. An engineer has to confirm the first real figures by + hand once the historian is connected. And the SCADA repository and the live server + disagree about sample rates — the checked-in file says one second, the running system + says five. The running system was taken as correct here, but that means the repository does not + currently describe the machine, and somebody should decide which of the two is wrong.
+Twenty-eight containers on one shared host. Seven are ours; twenty-one were + already here and serve other demos. This is the list the architecture deck never carried.
+ +The store. Holds the searchable text of every controlled document, the equipment + and tag list that turns “Pump 02” into a real tag, the upload queue, the permanent + audit trail, and — until the historian arrives — the stand-in plant history.
+Five roles, deliberately split: the answer path can only read, only the ingestion role can + write documents, and a database trigger stops the web role un-withdrawing anything.
+The data translator. Turns “high level alarms last week” into an + exact query over recorded history. The definitions that make an answer right live in files an + engineer can read and check, rather than being invented per question.
+The assistant never writes database code itself — it fills in a request form and Cube + does the rest. Today it reads stand-in tables, so no number it produces means anything about + the plant, and every answer built on one says so on screen.
+Cube’s own storage engine. Holds the queue and the ready-made summaries + that keep “count last week” fast without repeatedly scanning the historian.
+Not an optimisation and not optional — Cube will not start without it. It is missing + from the original architecture deck and from the build documents’ container table, which + is why it is called out here.
+The assistant itself, and the busiest thing here. It sorts each question into + one of four kinds, resolves the plant names in it, sends it down the matching lane, gathers the + evidence, has the model word an answer, and then checks that answer against a contract written + in code before letting it out.
+Since 28 August it also serves the document library — upload, review, approve, + withdraw and restore — because the operator’s screen no longer carries any identity.
+The operator’s screen. A question box, the answer, and a + “show working” panel giving the question’s kind, the query that ran, the row + count, and every citation with its revision and effective date.
+It carries two banners an operator cannot switch off: one saying the figures are stand-ins, + and one on procedural and advisory answers saying what the assistant deliberately did + not do.
+The logbook. Records every question: how it was sorted and with what confidence, + which tools ran, which passages were retrieved, how many tokens it cost, how long it took, and + whether the contract passed or failed. Every rejection is logged with the offending output.
+Deployed early on purpose, so no experiment since has been untraceable. Self-hosted and + MIT-licensed — no licence cost, and no question text leaves the host to reach it.
+The terminal path for loading documents, started by hand when documents change. + Reads a file, splits it on section boundaries so a numbered step sequence is never cut in half, + asks a person at the keyboard to confirm the document number, revision and effective date, and + only then stores it.
+It cannot see anything loaded through the upload screen — published files stay + in the upload area rather than moving into the document tree. The two paths must not be used on + the same document.
+Designed as a background worker that would pre-scan uploads, ingest approved + ones and complete withdrawals, kept off the network entirely so that the web-facing part of the + system could never write documents directly.
+Not built. That work happens inside ai-api instead, so the same process holds both + permissions. The database trigger still stops the web role un-withdrawing anything, so the + boundary holds — but it is now a code boundary rather than a deployment one.
+This host has served demos for months. Restarting the first two logs out every + active user, including whoever is mid-demo.
+The only front door. Every screen on this host is reached through it, and it + obtains and renews the certificates automatically. Nothing this project built publishes a port + of its own — they all sit behind it on a shared internal network.
+Sign-in for the whole host: Active Directory plus a phone prompt. Omitting it + from a site block silently makes that service public, which is why every block here carries it + — with the one deliberate exception of the operator’s screen.
+The PLC for this demo plant. It runs the actual control logic for the + three pumps and the wet well; the SCADA server polls it over Modbus and historises the result. + Live control, not a simulation of one.
+The host’s main time-series store, used by the other demos on this box. + Nothing this project built reads or writes it. It is listed here because it is effectively the + only consumer of the shared data disk, growing roughly a gigabyte a week — so it, not us, + is what will eventually fill it.
+The internal Git server, and the master copy of this project’s repository. + It sits outside the shared sign-in because forwarding authentication breaks Git clients — + a reason that applies to nothing else here.
+Device data arrives over MQTT into mosquitto, is processed by + nodered, and is stored in InfluxDB; telegraf adds host and container metrics; and + grafana draws the dashboards over the top. None of it touches the assistant.
+portainer is graphical Docker management and is root-equivalent; + dozzle is live searchable container logs and the best first stop when something here + misbehaves; watchtower auto-updates a safe subset of images on Sunday mornings.
+wireguard is the VPN through which engineers reach this host at all; + showroom is a static demo site; eqp-licence issues licences with a signing key + mounted read-only. None is part of the assistant.
+A LoRaWAN network server and its supporting containers. Running but + not configured — a future capability for this host, unrelated to the assistant. + Safe to ignore.
+Documented rather than hidden, and none of them to be shipped. Production closes + them in this order: network separation, then secrets, then database guardrails, then document + control integration, then resilience.
+ +What the assistant will and will not say must be reviewed with an OT or safety representative. + That review is still outstanding, and it now has to cover the unauthenticated console as well. + Reviewing it was only meaningful once the model was in place — which it now is, so nothing + stands in the way of booking it.
+