0b8ff4b0c3
Counted on lin001: 28 running, none stopped. Seven are ours (ai-api, ai-web, pg-ai, cube, cubestore, langfuse, lf-db), so 21 were already there. Four documents gave four different numbers because they were counting different things at different times. BUILD-AI-CONTAINERS.md (21 pre-existing) and current-state.html (28 total) were already right and are untouched. README.md said 22 where it meant 21. The host brief said 20 in three places while naming only 19, and it is the one with a real gap behind it: openplc-runtime appears NOWHERE in the 351-line host brief. Not in the stack table, not in the data flow, not in the rules - while CLAUDE.md calls it live control that must never be restarted, and the build spec names it as the reason port 502 is the one exception to no-published- ports on this host. The document a new engineer reads first to learn what runs here omitted the only container doing live control. wireguard was missing from the same table, though mentioned in three other sections. Both are now in §3. §6's published-ports list was missing 502/tcp and 8443/tcp for the same reason; verified against docker ps, they bind to 10.0.0.17 rather than 0.0.0.0, and that binding is what keeps unauthenticated Modbus off the internet, so it is recorded as a control and not as a detail. §12's "confirm 20 containers up" no longer names a number - that is how it went stale. The brief's stamp moves to 2026-09-01 with a note saying what was amended and what was left alone, so Dan's authorship of the rest stays clear. Separately, README.md listed ai-docs-worker among the containers as though it were running. It was never deployed: Phase 9 shipped with the document screens served by ai-api, and the worker sits behind a --profile worker flag. compose and current-state.html both say so already; the README table now does too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
264 lines
13 KiB
Markdown
264 lines
13 KiB
Markdown
# WRPS Plant Operations Assistant
|
||
|
||
A proof-of-concept assistant that lets an operator at the **Waterloo Road Pump
|
||
Station** ask a question in plain English and get an answer grounded in plant
|
||
data and controlled documents.
|
||
|
||
| Example question | Class |
|
||
|---|---|
|
||
| "How many times did the wet well high level alarm come up last week?" | **Historical** |
|
||
| "What does the level signal fault alarm on the wet well mean?" | **Reference** |
|
||
| "How do I lift the interlock on Pump 02?" | **Procedural** |
|
||
| "What discharge rate should we run to avoid spilling?" | **Advisory** |
|
||
|
||
Those four need different retrieval paths, different answer contracts and
|
||
different safety rules. One generic pipeline covering all four is the main way
|
||
this project fails.
|
||
|
||
**Success is a correct, citable, appropriately-scoped answer. Fluency is not
|
||
success.**
|
||
|
||
---
|
||
|
||
## Start here
|
||
|
||
New to this repository? In this order:
|
||
|
||
1. **This file** — what the system is, what it refuses to do, and how it is put
|
||
together. Fifteen minutes.
|
||
2. **[`status/workflow-map.html`](status/workflow-map.html)** — the same thing
|
||
with no software vocabulary, written for a plant engineer. Open it in a
|
||
browser.
|
||
3. **[`status/current-state.html`](status/current-state.html)** — what is
|
||
actually running right now, phase by phase, and what is still outstanding.
|
||
4. **[`CLAUDE.md`](CLAUDE.md)** — the rules. Short, and **binding**: it is
|
||
loaded automatically by Claude Code at the start of every session, so it is
|
||
the one file that changes what an AI agent will do in this repo. Read it
|
||
before you change it.
|
||
|
||
Then, depending on what you came to do:
|
||
|
||
| I want to... | Go to |
|
||
|---|---|
|
||
| Deploy it, or rebuild it from nothing | [`spec/REBUILD.md`](spec/REBUILD.md) |
|
||
| Understand *why* it is built this way | [`spec/BUILD-AI-CONTAINERS.md`](spec/BUILD-AI-CONTAINERS.md) |
|
||
| Understand the host it runs on | [`spec/YAU_Linux_Host_Onboarding.md`](spec/YAU_Linux_Host_Onboarding.md) |
|
||
| Know what is still blocked, and on whom | [`status/REQUESTS.md`](status/REQUESTS.md) |
|
||
| Change code | `pytest api/tests` first — it needs no network, no API key and no database |
|
||
|
||
**You will need**, and should confirm before starting: SSH to `lin001`, an AD
|
||
account in the Authelia groups, and — only for the answer path — an Azure OpenAI
|
||
key. None of them are in this repository; they come from Dan.
|
||
|
||
---
|
||
|
||
## Read this before writing any code
|
||
|
||
This is an **information retrieval and analysis assistant**. It is not a
|
||
control system, not an advisory controller, and not a substitute for a
|
||
competent person.
|
||
|
||
**It does not issue instructions for safety-critical actions.** For *"how do I
|
||
lift the interlock on Pump 02"* it locates and cites the controlled procedure.
|
||
An interlock exists because somebody assessed a hazard; a bypass procedure
|
||
reassembled from retrieved fragments is a safety document nobody approved.
|
||
|
||
**It does not recommend setpoints or operating parameters.** For *"what
|
||
discharge rate"* it gives evidence — rates used, outcomes, when alarms
|
||
occurred, documented capacity — and then defers. A number presented as an
|
||
answer gets typed into a control system by someone who trusts it.
|
||
|
||
**It does not answer outside its evidence.** Zero rows means "no records
|
||
found", never an invented figure.
|
||
|
||
These are **code paths, not prompt instructions**: [`api/contracts.py`](api/contracts.py)
|
||
holds one Pydantic contract per class, validated after generation and before
|
||
returning. A response that fails its contract is regenerated once, then errors.
|
||
It is never returned. `pytest api/tests` exercises every rule above without an
|
||
API key or a database, because that is the point of putting them in Python.
|
||
|
||
Full detail: [`BUILD-AI-CONTAINERS.md`](spec/BUILD-AI-CONTAINERS.md) §2.
|
||
|
||
---
|
||
|
||
## The plant
|
||
|
||
Waterloo Road Pump Station is a three-pump wastewater station.
|
||
|
||
- Wet well `WW-101`, 0–7000 mm, **120 m³ per metre** of level
|
||
- Pumps `PU-301/302/303`, duty/assist/assist, ~120 L/s each against 22 m static
|
||
lift, on a **common VSD speed reference** clamped 38–50 Hz
|
||
- Spill weir crest at **6000 mm**, `LSHH-102` at 5500 mm, high level alarm at
|
||
5200 mm, stop-all at 1000 mm
|
||
- Control logic runs on `openplc-runtime`; Yokogawa CI Server on `cicore1`
|
||
polls it over Modbus TCP and historises the result
|
||
|
||
**The unit trap that will catch you:** the PLC works in millimetres and litres
|
||
per second; the historian stores **percent of the weir crest** (raw mm ÷ 60) and
|
||
**m³/h**. Every conversion is recorded per tag in
|
||
[`db/seed/tags.csv`](db/seed/tags.csv), which also records — in capitals, at the
|
||
start of each description — whether a tag is **historised at all**. Field inputs
|
||
to the PLC (`%IW`/`%IX`: vibration, thermal, discharge pressure) are not
|
||
published to SCADA and have no history. An answer that trends PU-301 vibration
|
||
is fabricating data.
|
||
|
||
Source of truth for the plant: `WRPS/01-design-doc/`, `WRPS/04-plc/register-map.csv`
|
||
and `WRPS/05-scada/modbus/scada-points.csv` in the WRPS repository.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
Everything arrives at the **same Caddy**, which terminates TLS for all four
|
||
hostnames. What differs is the gate immediately behind it. Three hostnames keep
|
||
`import authelia` — the shared AD + Duo gate this host has always used. The
|
||
operator's hostname does not: since 28 August it is matched on source IP
|
||
instead. The operator is inside the Caddy/Authelia environment; they simply take
|
||
a branch through it that Authelia does not sit on.
|
||
|
||
```
|
||
┌──────────────────────────────────────────┐
|
||
operator at cicore1 ──────►│ │
|
||
(SCADA machine, 10.0.0.21) │ Caddy │
|
||
│ (one front door, TLS) │
|
||
engineer / anyone else ───►│ │
|
||
└────────┬────────────────────────┬────────┘
|
||
│ │
|
||
ai.yokogawa.tech │ │ api. / cube. / lf.
|
||
NO import authelia│ │ import authelia
|
||
▼ ▼
|
||
@scada remote_ip 10.0.0.21 Authelia (AD + Duo)
|
||
│ │
|
||
┌────────────┴───────────┐ │
|
||
match no match │
|
||
│ │ │
|
||
/ask* ──► ai-api 403 │
|
||
else ──► ai-web ──► /ask ─────────────────► ai-api
|
||
│
|
||
cube ◄─────────┤
|
||
langfuse ◄─────────┘
|
||
```
|
||
|
||
**Read the left branch before relying on it.** `ai.yokogawa.tech` has no
|
||
Authelia in front of it and carries **no identity at all** — it is an IP
|
||
allowlist on a flat network. `api.yokogawa.tech` is unchanged and still fully
|
||
gated, which is why the document library lives there. Detail and rollback:
|
||
[`caddy/ai-routes.caddy`](caddy/ai-routes.caddy) and and step 6 of
|
||
[`spec/REBUILD.md`](spec/REBUILD.md).
|
||
|
||
Once a question reaches `ai-api`, the answer path is the same whichever door it
|
||
came through:
|
||
|
||
```
|
||
ai-api ──► classifier ──► one branch per class ──► contract validation ──► answer
|
||
(CHEAP_ │ │
|
||
DEPLOYMENT) ├──► Cube ──► imh (SQL Server, read-only, TDS/1433)
|
||
│ ── PENDING ──
|
||
└──► pgvector (pg-ai)
|
||
│
|
||
▼
|
||
Langfuse (tracing)
|
||
```
|
||
|
||
Everything runs on `yau-sls-poc-lin001` (`10.0.0.17`), a **shared, live** Docker
|
||
host that already runs 21 containers including `openplc-runtime` — the PLC for
|
||
this demo. See [`YAU_Linux_Host_Onboarding.md`](spec/YAU_Linux_Host_Onboarding.md).
|
||
|
||
**There is no replication job and no mirror table.** `imh` is already an
|
||
isolated copy of the raw SCADA historian, so Cube queries it directly with a
|
||
read-only login. `pg-ai` holds pgvector chunks, Cube pre-aggregations, and the
|
||
equipment/tag reference data.
|
||
|
||
| Container | Stack | Networks | Public URL |
|
||
|---|---|---|---|
|
||
| `pg-ai` | `pgvector/pgvector:pg16` | `ai-internal` only | none |
|
||
| `cube` | `cubejs/cube` (pinned) | `ai-internal` + `proxy` | `cube.yokogawa.tech` |
|
||
| `cubestore` | `cubejs/cubestore` (pinned) | `ai-internal` only | none |
|
||
| `ai-api` | Python 3.12 + FastAPI | `ai-internal` + `proxy` | `api.yokogawa.tech` |
|
||
| `ai-web` | Vite build → `nginx:alpine` | `proxy` | `ai.yokogawa.tech` |
|
||
| `ai-ingest` | Python 3.12, on demand | `ai-internal` | none |
|
||
| `ai-docs-worker` | **not deployed** — `--profile worker`, see below | `ai-internal` | none |
|
||
| `langfuse` + `lf-db` | official images | `ai-internal` + `proxy` | `lf.yokogawa.tech` |
|
||
|
||
**Seven of those are running.** `ai-ingest` is run on demand with `run --rm` and is not
|
||
meant to be up. `ai-docs-worker` was **never deployed**: Phase 9 shipped with the
|
||
document screens served by `ai-api` instead, and the worker sits behind a
|
||
`--profile worker` flag — see the note at `compose/ai-compose.yml:216`. Counted on the
|
||
host 2026-09-01: 28 containers, 21 of them pre-existing.
|
||
|
||
---
|
||
|
||
## Working on it
|
||
|
||
```bash
|
||
pytest api/tests # contracts, classifier rules, SQL allow-list. No network.
|
||
```
|
||
|
||
- **The host is live and shared.** Prefer additive changes. Snapshot config
|
||
before editing. **Announce anything that restarts Caddy or Authelia** — it
|
||
logs out every active user, including whoever is mid-demo.
|
||
- **Never restart, update or reconfigure `openplc-runtime`** as a side effect
|
||
of AI work. It is the PLC for the demo plant. Its published port 502 is the
|
||
one deliberate exception to the no-published-ports rule on this host, and it
|
||
does not generalise to anything we build.
|
||
- **Verify, don't assume.** `docker ps` showing "Up" is not proof.
|
||
- **Test every layer without the LLM first.** Prove Cube returns the right
|
||
number by hand. Prove retrieval finds the right procedure by hand. Then wire
|
||
up the agent — otherwise a wrong answer has four possible causes.
|
||
- **Do not invent schema.** Inspect, or ask.
|
||
- Fix eval failures in the classifier, Cube and ingestion — **not by adding
|
||
instructions to the prompt**. When something fails, add the failing case to
|
||
`eval/testset.jsonl` *before* fixing it.
|
||
|
||
---
|
||
|
||
## Repository layout
|
||
|
||
```
|
||
CLAUDE.md short rules — what Claude Code keeps front of mind
|
||
|
||
spec/ what we agreed to build — normative, durable
|
||
BUILD-AI-CONTAINERS.md the build spec — the design and the reasoning
|
||
YAU_Linux_Host_Onboarding.md the host brief (reference; wins on conflict)
|
||
REBUILD.md the deployment runbook — nine phases, in order
|
||
|
||
status/ where it got to — snapshots, go stale, edited often
|
||
workflow-map.html the non-technical explainer — how a question becomes
|
||
an answer, and what is built. Open it in a browser
|
||
current-state.html what is actually running right now. Open it too
|
||
REQUESTS.md what is still needed from other people
|
||
|
||
compose/ deployed to ~/ai-compose.yml and ~/langfuse-compose.yml
|
||
caddy/ai-routes.caddy blocks to paste into ~/Caddyfile
|
||
authelia/access-rules.md the rule additions as text — never the real config
|
||
db/ schema, roles, fixtures, and the alias seed CSVs
|
||
cube/model/ alarms, process values, operations, equipment
|
||
api/ FastAPI, classifier, agent, contracts, guardrails
|
||
ingest/ Docling → chunk → embed → pg-ai, plus the
|
||
Phase 9 upload worker
|
||
web/ React + Vite operator UI
|
||
eval/ 78-case test set and the scorecard runner
|
||
scripts/ deploy.sh, verify.sh
|
||
```
|
||
|
||
---
|
||
|
||
## Known shortcuts
|
||
|
||
Deliberate, documented, and not to be shipped. Full list in
|
||
[`BUILD-AI-CONTAINERS.md`](spec/BUILD-AI-CONTAINERS.md) §14. The ones that matter most:
|
||
|
||
- Secrets in `0600` env files, not a vault
|
||
- No OT/IT firewall boundary — one flat `10.0.0.0/24` PoC network
|
||
- Modbus TCP on port 502 with no authentication or encryption, contained by
|
||
NSG/VPN scope only — **confirm the NSG does not expose it to the internet**
|
||
- Single host, no HA: `lin001` is a single point of failure for both the demo
|
||
estate and the simulated plant's PLC
|
||
- No automated backup — `pg-ai` needs adding to whatever backup exists
|
||
- Document revision metadata entered semi-manually, not integrated with
|
||
document control — Phase 9 records *who* asserted a revision, which is not the
|
||
same as knowing what the current one is
|
||
- Uploaded documents are not malware-scanned; type and size checks only
|
||
|
||
**Section 2 of the build spec must be reviewed with an OT/safety representative
|
||
before any operator sees a demo.**
|