7d6d90f486
status/OPEN-ISSUES.md is a fourth register, so it opens by saying what does NOT belong in it. Work blocked on other people stays in REQUESTS.md; shortcuts we consciously accepted stay in BUILD-AI-CONTAINERS.md 14 and are closed by decision, not to be re-raised here; what is running stays in current-state.html. What had no home until now is a defect we own and have not fixed - those were living in commit messages. Closed issues move to the bottom rather than being deleted: this repo is an as-built record, and an issue with no trace of how it closed is worth less than one that was never raised. OI-01: the OpenPLC Editor is not installed anywhere. It is a desktop tool published by OpenPLC, and it is what authors and compiles the IEC 61131-3 program openplc-runtime executes. Without it there is no reviewable source for the control logic in or beside this repository - the program exists only inside the container. Raising it exposed two wrong statements in the build spec, both calling port 8443 the "OpenPLC Runtime web UI". Probed read-only on the host: Server: Werkzeug/3.1.8 Python/3.11.2 / /login /index.html /programs /status /runtime -> 404 /api/v1 -> 401 Unauthorized It is an authenticated REST API with no browser interface at all. Section 4 now says so with the evidence, and 14's entry is restated: 8443 is contained by the same 10.0.0.17 binding as 502, but unlike 502 it is not anonymous, which is a better position than that section recorded. The binding is still the control that matters. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
266 lines
14 KiB
Markdown
266 lines
14 KiB
Markdown
# WRPS Plant Operations Assistant
|
||
|
||
A proof-of-concept assistant that lets an operator at the **Waterloo Road Pump
|
||
Station** ask a question in plain English and get an answer grounded in plant
|
||
data and controlled documents.
|
||
|
||
| Example question | Class |
|
||
|---|---|
|
||
| "How many times did the wet well high level alarm come up last week?" | **Historical** |
|
||
| "What does the level signal fault alarm on the wet well mean?" | **Reference** |
|
||
| "How do I lift the interlock on Pump 02?" | **Procedural** |
|
||
| "What discharge rate should we run to avoid spilling?" | **Advisory** |
|
||
|
||
Those four need different retrieval paths, different answer contracts and
|
||
different safety rules. One generic pipeline covering all four is the main way
|
||
this project fails.
|
||
|
||
**Success is a correct, citable, appropriately-scoped answer. Fluency is not
|
||
success.**
|
||
|
||
---
|
||
|
||
## Start here
|
||
|
||
New to this repository? In this order:
|
||
|
||
1. **This file** — what the system is, what it refuses to do, and how it is put
|
||
together. Fifteen minutes.
|
||
2. **[`status/workflow-map.html`](status/workflow-map.html)** — the same thing
|
||
with no software vocabulary, written for a plant engineer. Open it in a
|
||
browser.
|
||
3. **[`status/current-state.html`](status/current-state.html)** — what is
|
||
actually running right now, phase by phase, and what is still outstanding.
|
||
4. **[`CLAUDE.md`](CLAUDE.md)** — the rules. Short, and **binding**: it is
|
||
loaded automatically by Claude Code at the start of every session, so it is
|
||
the one file that changes what an AI agent will do in this repo. Read it
|
||
before you change it.
|
||
|
||
Then, depending on what you came to do:
|
||
|
||
| I want to... | Go to |
|
||
|---|---|
|
||
| Deploy it, or rebuild it from nothing | [`spec/REBUILD.md`](spec/REBUILD.md) |
|
||
| Understand *why* it is built this way | [`spec/BUILD-AI-CONTAINERS.md`](spec/BUILD-AI-CONTAINERS.md) |
|
||
| Understand the host it runs on | [`spec/YAU_Linux_Host_Onboarding.md`](spec/YAU_Linux_Host_Onboarding.md) |
|
||
| Know what is still blocked, and on whom | [`status/REQUESTS.md`](status/REQUESTS.md) |
|
||
| Know what is broken or unfinished, and ours to fix | [`status/OPEN-ISSUES.md`](status/OPEN-ISSUES.md) |
|
||
| Change code | `pytest api/tests` first — it needs no network, no API key and no database |
|
||
|
||
**You will need**, and should confirm before starting: SSH to `lin001`, an AD
|
||
account in the Authelia groups, and — only for the answer path — an Azure OpenAI
|
||
key. None of them are in this repository; they come from Dan.
|
||
|
||
---
|
||
|
||
## Read this before writing any code
|
||
|
||
This is an **information retrieval and analysis assistant**. It is not a
|
||
control system, not an advisory controller, and not a substitute for a
|
||
competent person.
|
||
|
||
**It does not issue instructions for safety-critical actions.** For *"how do I
|
||
lift the interlock on Pump 02"* it locates and cites the controlled procedure.
|
||
An interlock exists because somebody assessed a hazard; a bypass procedure
|
||
reassembled from retrieved fragments is a safety document nobody approved.
|
||
|
||
**It does not recommend setpoints or operating parameters.** For *"what
|
||
discharge rate"* it gives evidence — rates used, outcomes, when alarms
|
||
occurred, documented capacity — and then defers. A number presented as an
|
||
answer gets typed into a control system by someone who trusts it.
|
||
|
||
**It does not answer outside its evidence.** Zero rows means "no records
|
||
found", never an invented figure.
|
||
|
||
These are **code paths, not prompt instructions**: [`api/contracts.py`](api/contracts.py)
|
||
holds one Pydantic contract per class, validated after generation and before
|
||
returning. A response that fails its contract is regenerated once, then errors.
|
||
It is never returned. `pytest api/tests` exercises every rule above without an
|
||
API key or a database, because that is the point of putting them in Python.
|
||
|
||
Full detail: [`BUILD-AI-CONTAINERS.md`](spec/BUILD-AI-CONTAINERS.md) §2.
|
||
|
||
---
|
||
|
||
## The plant
|
||
|
||
Waterloo Road Pump Station is a three-pump wastewater station.
|
||
|
||
- Wet well `WW-101`, 0–7000 mm, **120 m³ per metre** of level
|
||
- Pumps `PU-301/302/303`, duty/assist/assist, ~120 L/s each against 22 m static
|
||
lift, on a **common VSD speed reference** clamped 38–50 Hz
|
||
- Spill weir crest at **6000 mm**, `LSHH-102` at 5500 mm, high level alarm at
|
||
5200 mm, stop-all at 1000 mm
|
||
- Control logic runs on `openplc-runtime`; Yokogawa CI Server on `cicore1`
|
||
polls it over Modbus TCP and historises the result
|
||
|
||
**The unit trap that will catch you:** the PLC works in millimetres and litres
|
||
per second; the historian stores **percent of the weir crest** (raw mm ÷ 60) and
|
||
**m³/h**. Every conversion is recorded per tag in
|
||
[`db/seed/tags.csv`](db/seed/tags.csv), which also records — in capitals, at the
|
||
start of each description — whether a tag is **historised at all**. Field inputs
|
||
to the PLC (`%IW`/`%IX`: vibration, thermal, discharge pressure) are not
|
||
published to SCADA and have no history. An answer that trends PU-301 vibration
|
||
is fabricating data.
|
||
|
||
Source of truth for the plant: `WRPS/01-design-doc/`, `WRPS/04-plc/register-map.csv`
|
||
and `WRPS/05-scada/modbus/scada-points.csv` in the WRPS repository.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
Everything arrives at the **same Caddy**, which terminates TLS for all four
|
||
hostnames. What differs is the gate immediately behind it. Three hostnames keep
|
||
`import authelia` — the shared AD + Duo gate this host has always used. The
|
||
operator's hostname does not: since 28 August it is matched on source IP
|
||
instead. The operator is inside the Caddy/Authelia environment; they simply take
|
||
a branch through it that Authelia does not sit on.
|
||
|
||
```
|
||
┌──────────────────────────────────────────┐
|
||
operator at cicore1 ──────►│ │
|
||
(SCADA machine, 10.0.0.21) │ Caddy │
|
||
│ (one front door, TLS) │
|
||
engineer / anyone else ───►│ │
|
||
└────────┬────────────────────────┬────────┘
|
||
│ │
|
||
ai.yokogawa.tech │ │ api. / cube. / lf.
|
||
NO import authelia│ │ import authelia
|
||
▼ ▼
|
||
@scada remote_ip 10.0.0.21 Authelia (AD + Duo)
|
||
│ │
|
||
┌────────────┴───────────┐ │
|
||
match no match │
|
||
│ │ │
|
||
/ask* ──► ai-api 403 │
|
||
else ──► ai-web ──► /ask ─────────────────► ai-api
|
||
│
|
||
cube ◄─────────┤
|
||
langfuse ◄─────────┘
|
||
```
|
||
|
||
**Read the left branch before relying on it.** `ai.yokogawa.tech` has no
|
||
Authelia in front of it and carries **no identity at all** — it is an IP
|
||
allowlist on a flat network. `api.yokogawa.tech` is unchanged and still fully
|
||
gated, which is why the document library lives there. Detail and rollback:
|
||
[`caddy/ai-routes.caddy`](caddy/ai-routes.caddy) and and step 6 of
|
||
[`spec/REBUILD.md`](spec/REBUILD.md).
|
||
|
||
Once a question reaches `ai-api`, the answer path is the same whichever door it
|
||
came through:
|
||
|
||
```
|
||
ai-api ──► classifier ──► one branch per class ──► contract validation ──► answer
|
||
(CHEAP_ │ │
|
||
DEPLOYMENT) ├──► Cube ──► imh (SQL Server, read-only, TDS/1433)
|
||
│ ── PENDING ──
|
||
└──► pgvector (pg-ai)
|
||
│
|
||
▼
|
||
Langfuse (tracing)
|
||
```
|
||
|
||
Everything runs on `yau-sls-poc-lin001` (`10.0.0.17`), a **shared, live** Docker
|
||
host that already runs 21 containers including `openplc-runtime` — the PLC for
|
||
this demo. See [`YAU_Linux_Host_Onboarding.md`](spec/YAU_Linux_Host_Onboarding.md).
|
||
|
||
**There is no replication job and no mirror table.** `imh` is already an
|
||
isolated copy of the raw SCADA historian, so Cube queries it directly with a
|
||
read-only login. `pg-ai` holds pgvector chunks, Cube pre-aggregations, and the
|
||
equipment/tag reference data.
|
||
|
||
| Container | Stack | Networks | Public URL |
|
||
|---|---|---|---|
|
||
| `pg-ai` | `pgvector/pgvector:pg16` | `ai-internal` only | none |
|
||
| `cube` | `cubejs/cube` (pinned) | `ai-internal` + `proxy` | `cube.yokogawa.tech` |
|
||
| `cubestore` | `cubejs/cubestore` (pinned) | `ai-internal` only | none |
|
||
| `ai-api` | Python 3.12 + FastAPI | `ai-internal` + `proxy` | `api.yokogawa.tech` |
|
||
| `ai-web` | Vite build → `nginx:alpine` | `proxy` | `ai.yokogawa.tech` |
|
||
| `ai-ingest` | Python 3.12, on demand | `ai-internal` | none |
|
||
| `ai-docs-worker` | **not deployed** — `--profile worker`, see below | `ai-internal` | none |
|
||
| `langfuse` + `lf-db` | official images | `ai-internal` + `proxy` | `lf.yokogawa.tech` |
|
||
|
||
**Seven of those are running.** `ai-ingest` is run on demand with `run --rm` and is not
|
||
meant to be up. `ai-docs-worker` was **never deployed**: Phase 9 shipped with the
|
||
document screens served by `ai-api` instead, and the worker sits behind a
|
||
`--profile worker` flag — see the note at `compose/ai-compose.yml:216`. Counted on the
|
||
host 2026-09-01: 28 containers, 21 of them pre-existing.
|
||
|
||
---
|
||
|
||
## Working on it
|
||
|
||
```bash
|
||
pytest api/tests # contracts, classifier rules, SQL allow-list. No network.
|
||
```
|
||
|
||
- **The host is live and shared.** Prefer additive changes. Snapshot config
|
||
before editing. **Announce anything that restarts Caddy or Authelia** — it
|
||
logs out every active user, including whoever is mid-demo.
|
||
- **Never restart, update or reconfigure `openplc-runtime`** as a side effect
|
||
of AI work. It is the PLC for the demo plant. Its published port 502 is the
|
||
one deliberate exception to the no-published-ports rule on this host, and it
|
||
does not generalise to anything we build.
|
||
- **Verify, don't assume.** `docker ps` showing "Up" is not proof.
|
||
- **Test every layer without the LLM first.** Prove Cube returns the right
|
||
number by hand. Prove retrieval finds the right procedure by hand. Then wire
|
||
up the agent — otherwise a wrong answer has four possible causes.
|
||
- **Do not invent schema.** Inspect, or ask.
|
||
- Fix eval failures in the classifier, Cube and ingestion — **not by adding
|
||
instructions to the prompt**. When something fails, add the failing case to
|
||
`eval/testset.jsonl` *before* fixing it.
|
||
|
||
---
|
||
|
||
## Repository layout
|
||
|
||
```
|
||
CLAUDE.md short rules — what Claude Code keeps front of mind
|
||
|
||
spec/ what we agreed to build — normative, durable
|
||
BUILD-AI-CONTAINERS.md the build spec — the design and the reasoning
|
||
YAU_Linux_Host_Onboarding.md the host brief (reference; wins on conflict)
|
||
REBUILD.md the deployment runbook — nine phases, in order
|
||
|
||
status/ where it got to — snapshots, go stale, edited often
|
||
workflow-map.html the non-technical explainer — how a question becomes
|
||
an answer, and what is built. Open it in a browser
|
||
current-state.html what is actually running right now. Open it too
|
||
REQUESTS.md what is still needed from other people
|
||
OPEN-ISSUES.md what is broken or unfinished, and ours to fix
|
||
|
||
compose/ deployed to ~/ai-compose.yml and ~/langfuse-compose.yml
|
||
caddy/ai-routes.caddy blocks to paste into ~/Caddyfile
|
||
authelia/access-rules.md the rule additions as text — never the real config
|
||
db/ schema, roles, fixtures, and the alias seed CSVs
|
||
cube/model/ alarms, process values, operations, equipment
|
||
api/ FastAPI, classifier, agent, contracts, guardrails
|
||
ingest/ Docling → chunk → embed → pg-ai, plus the
|
||
Phase 9 upload worker
|
||
web/ React + Vite operator UI
|
||
eval/ 78-case test set and the scorecard runner
|
||
scripts/ deploy.sh, verify.sh
|
||
```
|
||
|
||
---
|
||
|
||
## Known shortcuts
|
||
|
||
Deliberate, documented, and not to be shipped. Full list in
|
||
[`BUILD-AI-CONTAINERS.md`](spec/BUILD-AI-CONTAINERS.md) §14. The ones that matter most:
|
||
|
||
- Secrets in `0600` env files, not a vault
|
||
- No OT/IT firewall boundary — one flat `10.0.0.0/24` PoC network
|
||
- Modbus TCP on port 502 with no authentication or encryption, contained by
|
||
NSG/VPN scope only — **confirm the NSG does not expose it to the internet**
|
||
- Single host, no HA: `lin001` is a single point of failure for both the demo
|
||
estate and the simulated plant's PLC
|
||
- No automated backup — `pg-ai` needs adding to whatever backup exists
|
||
- Document revision metadata entered semi-manually, not integrated with
|
||
document control — Phase 9 records *who* asserted a revision, which is not the
|
||
same as knowing what the current one is
|
||
- Uploaded documents are not malware-scanned; type and size checks only
|
||
|
||
**Section 2 of the build spec must be reviewed with an OT/safety representative
|
||
before any operator sees a demo.**
|