WRPS Plant Operations Assistant

A proof-of-concept assistant that lets an operator at the Waterloo Road Pump Station ask a question in plain English and get an answer grounded in plant data and controlled documents.

Example question	Class
"How many times did the wet well high level alarm come up last week?"	Historical
"What does the level signal fault alarm on the wet well mean?"	Reference
"How do I lift the interlock on Pump 02?"	Procedural
"What discharge rate should we run to avoid spilling?"	Advisory

Those four need different retrieval paths, different answer contracts and different safety rules. One generic pipeline covering all four is the main way this project fails.

Success is a correct, citable, appropriately-scoped answer. Fluency is not success.

---

Start here

New to this repository? In this order:

1. This file — what the system is, what it refuses to do, and how it is put together. Fifteen minutes.
2. status/workflow-map.html — the same thing with no software vocabulary, written for a plant engineer. Open it in a browser.
3. status/current-state.html — what is actually running right now, phase by phase, and what is still outstanding.
4. CLAUDE.md — the rules. Short, and binding: it is loaded automatically by Claude Code at the start of every session, so it is the one file that changes what an AI agent will do in this repo. Read it before you change it.

Then, depending on what you came to do:

I want to...	Go to
Deploy it, or rebuild it from nothing	spec/REBUILD.md
Understand why it is built this way	spec/BUILD-AI-CONTAINERS.md
Understand the host it runs on	spec/YAU_Linux_Host_Onboarding.md
Know what is still blocked, and on whom	status/REQUESTS.md
Know what is broken or unfinished, and ours to fix	status/OPEN-ISSUES.md
Change code	pytest api/tests first — it needs no network, no API key and no database

You will need, and should confirm before starting: SSH to lin001, an AD account in the Authelia groups, and — only for the answer path — an Azure OpenAI key. None of them are in this repository; they come from Dan.

---

Read this before writing any code

This is an information retrieval and analysis assistant. It is not a control system, not an advisory controller, and not a substitute for a competent person.

It does not issue instructions for safety-critical actions. For "how do I lift the interlock on Pump 02" it locates and cites the controlled procedure. An interlock exists because somebody assessed a hazard; a bypass procedure reassembled from retrieved fragments is a safety document nobody approved.

It does not recommend setpoints or operating parameters. For "what discharge rate" it gives evidence — rates used, outcomes, when alarms occurred, documented capacity — and then defers. A number presented as an answer gets typed into a control system by someone who trusts it.

It does not answer outside its evidence. Zero rows means "no records found", never an invented figure.

These are code paths, not prompt instructions: api/contracts.py holds one Pydantic contract per class, validated after generation and before returning. A response that fails its contract is regenerated once, then errors. It is never returned. pytest api/tests exercises every rule above without an API key or a database, because that is the point of putting them in Python.

Full detail: BUILD-AI-CONTAINERS.md §2.

---

The plant

Waterloo Road Pump Station is a three-pump wastewater station.

• Wet well WW-101, 0–7000 mm, 120 m³ per metre of level
• Pumps PU-301/302/303, duty/assist/assist, ~120 L/s each against 22 m static lift, on a common VSD speed reference clamped 38–50 Hz
• Spill weir crest at 6000 mm, LSHH-102 at 5500 mm, high level alarm at 5200 mm, stop-all at 1000 mm
• Control logic runs on openplc-runtime; Yokogawa CI Server on cicore1 polls it over Modbus TCP and historises the result

The unit trap that will catch you: the PLC works in millimetres and litres per second; the historian stores percent of the weir crest (raw mm ÷ 60) and m³/h. Every conversion is recorded per tag in db/seed/tags.csv, which also records — in capitals, at the start of each description — whether a tag is historised at all. Field inputs to the PLC (%IW/%IX: vibration, thermal, discharge pressure) are not published to SCADA and have no history. An answer that trends PU-301 vibration is fabricating data.

Source of truth for the plant: WRPS/01-design-doc/, WRPS/04-plc/register-map.csv and WRPS/05-scada/modbus/scada-points.csv in the WRPS repository.

---

Architecture

Everything arrives at the same Caddy, which terminates TLS for all four hostnames. What differs is the gate immediately behind it. Three hostnames keep import authelia — the shared AD + Duo gate this host has always used. The operator's hostname does not: since 28 August it is matched on source IP instead. The operator is inside the Caddy/Authelia environment; they simply take a branch through it that Authelia does not sit on.

                            ┌──────────────────────────────────────────┐
 operator at cicore1 ──────►│                                          │
 (SCADA machine, 10.0.0.21) │                  Caddy                   │
                            │           (one front door, TLS)          │
 engineer / anyone else ───►│                                          │
                            └────────┬────────────────────────┬────────┘
                                     │                        │
                   ai.yokogawa.tech  │                        │  api. / cube. / lf.
                   NO import authelia│                        │  import authelia
                                     ▼                        ▼
                          @scada remote_ip 10.0.0.21    Authelia (AD + Duo)
                                     │                        │
                        ┌────────────┴───────────┐            │
                     match                    no match        │
                        │                         │           │
              /ask* ──► ai-api                   403          │
              else  ──► ai-web ──► /ask ─────────────────► ai-api
                                                             │
                                              cube ◄─────────┤
                                          langfuse ◄─────────┘

Read the left branch before relying on it. ai.yokogawa.tech has no Authelia in front of it and carries no identity at all — it is an IP allowlist on a flat network. api.yokogawa.tech is unchanged and still fully gated, which is why the document library lives there. Detail and rollback: caddy/ai-routes.caddy and and step 6 of spec/REBUILD.md.

Once a question reaches ai-api, the answer path is the same whichever door it came through:

ai-api ──► classifier ──► one branch per class ──► contract validation ──► answer
             (CHEAP_       │                              │
              DEPLOYMENT)  ├──► Cube ──► imh (SQL Server, read-only, TDS/1433)
                           │              ── PENDING ──
                           └──► pgvector (pg-ai)
                                          │
                                          ▼
                                      Langfuse (tracing)

Everything runs on yau-sls-poc-lin001 (10.0.0.17), a shared, live Docker host that already runs 21 containers including openplc-runtime — the PLC for this demo. See YAU_Linux_Host_Onboarding.md.

There is no replication job and no mirror table. imh is already an isolated copy of the raw SCADA historian, so Cube queries it directly with a read-only login. pg-ai holds pgvector chunks, Cube pre-aggregations, and the equipment/tag reference data.

Container	Stack	Networks	Public URL
pg-ai	pgvector/pgvector:pg16	ai-internal only	none
cube	cubejs/cube (pinned)	ai-internal + proxy	cube.yokogawa.tech
cubestore	cubejs/cubestore (pinned)	ai-internal only	none
ai-api	Python 3.12 + FastAPI	ai-internal + proxy	api.yokogawa.tech
ai-web	Vite build → nginx:alpine	proxy	ai.yokogawa.tech
ai-ingest	Python 3.12, on demand	ai-internal	none
ai-docs-worker	not deployed — --profile worker, see below	ai-internal	none
langfuse + lf-db	official images	ai-internal + proxy	lf.yokogawa.tech

Seven of those are running. ai-ingest is run on demand with run --rm and is not meant to be up. ai-docs-worker was never deployed: Phase 9 shipped with the document screens served by ai-api instead, and the worker sits behind a --profile worker flag — see the note at compose/ai-compose.yml:216. Counted on the host 2026-09-01: 28 containers, 21 of them pre-existing.

---

Working on it

pytest api/tests          # contracts, classifier rules, SQL allow-list. No network.

• The host is live and shared. Prefer additive changes. Snapshot config before editing. Announce anything that restarts Caddy or Authelia — it logs out every active user, including whoever is mid-demo.
• Never restart, update or reconfigure openplc-runtime as a side effect of AI work. It is the PLC for the demo plant. Its published port 502 is the one deliberate exception to the no-published-ports rule on this host, and it does not generalise to anything we build.
• Verify, don't assume. docker ps showing "Up" is not proof.
• Test every layer without the LLM first. Prove Cube returns the right number by hand. Prove retrieval finds the right procedure by hand. Then wire up the agent — otherwise a wrong answer has four possible causes.
• Do not invent schema. Inspect, or ask.
• Fix eval failures in the classifier, Cube and ingestion — not by adding instructions to the prompt. When something fails, add the failing case to eval/testset.jsonl before fixing it.

---

Repository layout

CLAUDE.md                    short rules — what Claude Code keeps front of mind

spec/                        what we agreed to build — normative, durable
  BUILD-AI-CONTAINERS.md     the build spec — the design and the reasoning
  YAU_Linux_Host_Onboarding.md  the host brief (reference; wins on conflict)
  REBUILD.md                 the deployment runbook — nine phases, in order

status/                      where it got to — snapshots, go stale, edited often
  workflow-map.html          the non-technical explainer — how a question becomes
                             an answer, and what is built. Open it in a browser
  current-state.html         what is actually running right now. Open it too
  REQUESTS.md                what is still needed from other people
  OPEN-ISSUES.md             what is broken or unfinished, and ours to fix

compose/                     deployed to ~/ai-compose.yml and ~/langfuse-compose.yml
caddy/ai-routes.caddy        blocks to paste into ~/Caddyfile
authelia/access-rules.md     the rule additions as text — never the real config
db/                          schema, roles, fixtures, and the alias seed CSVs
cube/model/                  alarms, process values, operations, equipment
api/                         FastAPI, classifier, agent, contracts, guardrails
ingest/                      Docling → chunk → embed → pg-ai, plus the
                             Phase 9 upload worker
web/                         React + Vite operator UI
eval/                        the eval set and the scorecard runner
scripts/                     deploy.sh, verify.sh

---

Known shortcuts

Deliberate, documented, and not to be shipped. Full list in BUILD-AI-CONTAINERS.md §14. The ones that matter most:

• Secrets in 0600 env files, not a vault
• No OT/IT firewall boundary — one flat 10.0.0.0/24 PoC network
• Modbus TCP on port 502 with no authentication or encryption, contained by NSG/VPN scope only — confirm the NSG does not expose it to the internet
• Single host, no HA: lin001 is a single point of failure for both the demo estate and the simulated plant's PLC
• No automated backup — pg-ai needs adding to whatever backup exists
• Document revision metadata entered semi-manually, not integrated with document control — Phase 9 records who asserted a revision, which is not the same as knowing what the current one is
• Uploaded documents are not malware-scanned; type and size checks only

Section 2 of the build spec must be reviewed with an OT/safety representative before any operator sees a demo.