yau-plant-assistant/README.md

# WRPS Plant Operations Assistant

A proof-of-concept assistant that lets an operator at the **Waterloo Road Pump
Station** ask a question in plain English and get an answer grounded in plant
data and controlled documents.

| Example question | Class |
|---|---|
| "How many times did the wet well high level alarm come up last week?" | **Historical** |
| "What does the level signal fault alarm on the wet well mean?" | **Reference** |
| "How do I lift the interlock on Pump 02?" | **Procedural** |
| "What discharge rate should we run to avoid spilling?" | **Advisory** |

Those four need different retrieval paths, different answer contracts and
different safety rules. One generic pipeline covering all four is the main way
this project fails.

**Success is a correct, citable, appropriately-scoped answer. Fluency is not
success.**

---

## Read this before writing any code

This is an **information retrieval and analysis assistant**. It is not a
control system, not an advisory controller, and not a substitute for a
competent person.

**It does not issue instructions for safety-critical actions.** For *"how do I
lift the interlock on Pump 02"* it locates and cites the controlled procedure.
An interlock exists because somebody assessed a hazard; a bypass procedure
reassembled from retrieved fragments is a safety document nobody approved.

**It does not recommend setpoints or operating parameters.** For *"what
discharge rate"* it gives evidence — rates used, outcomes, when alarms
occurred, documented capacity — and then defers. A number presented as an
answer gets typed into a control system by someone who trusts it.

**It does not answer outside its evidence.** Zero rows means "no records
found", never an invented figure.

These are **code paths, not prompt instructions**: [`api/contracts.py`](api/contracts.py)
holds one Pydantic contract per class, validated after generation and before
returning. A response that fails its contract is regenerated once, then errors.
It is never returned. `pytest api/tests` exercises every rule above without an
API key or a database, because that is the point of putting them in Python.

Full detail: [`BUILD-AI-CONTAINERS.md`](BUILD-AI-CONTAINERS.md) §2.

---

## The plant

Waterloo Road Pump Station is a three-pump wastewater station.

- Wet well `WW-101`, 0–7000 mm, **120 m³ per metre** of level
- Pumps `PU-301/302/303`, duty/assist/assist, ~120 L/s each against 22 m static
  lift, on a **common VSD speed reference** clamped 38–50 Hz
- Spill weir crest at **6000 mm**, `LSHH-102` at 5500 mm, high level alarm at
  5200 mm, stop-all at 1000 mm
- Control logic runs on `openplc-runtime`; Yokogawa CI Server on `cicore1`
  polls it over Modbus TCP and historises the result

**The unit trap that will catch you:** the PLC works in millimetres and litres
per second; the historian stores **percent of the weir crest** (raw mm ÷ 60) and
**m³/h**. Every conversion is recorded per tag in
[`db/seed/tags.csv`](db/seed/tags.csv), which also records — in capitals, at the
start of each description — whether a tag is **historised at all**. Field inputs
to the PLC (`%IW`/`%IX`: vibration, thermal, discharge pressure) are not
published to SCADA and have no history. An answer that trends PU-301 vibration
is fabricating data.

Source of truth for the plant: `WRPS/01-design-doc/`, `WRPS/04-plc/register-map.csv`
and `WRPS/05-scada/modbus/scada-points.csv` in the WRPS repository.

---

## Architecture

```
operator ──► Caddy ──► Authelia (AD + Duo) ──► ai-web ──► ai-api
                                                            │
                            ┌───────────────────────────────┼──────────────┐
                            ▼                               ▼              ▼
                      classifier                          Cube        pgvector
                   (CHEAP_DEPLOYMENT)                       │          (pg-ai)
                            │                               ▼
                    one branch per class              imh (SQL Server,
                            │                          read-only, TDS/1433)
                            ▼                          ── PENDING ──
                     contract validation
                            │
                            ▼
                        Langfuse
```

Everything runs on `yau-sls-poc-lin001` (`10.0.0.17`), a **shared, live** Docker
host that already runs 22 containers including `openplc-runtime` — the PLC for
this demo. See [`YAU_Linux_Host_Onboarding.md`](YAU_Linux_Host_Onboarding.md).

**There is no replication job and no mirror table.** `imh` is already an
isolated copy of the raw SCADA historian, so Cube queries it directly with a
read-only login. `pg-ai` holds pgvector chunks, Cube pre-aggregations, and the
equipment/tag reference data.

| Container | Stack | Networks | Public URL |
|---|---|---|---|
| `pg-ai` | `pgvector/pgvector:pg16` | `ai-internal` only | none |
| `cube` | `cubejs/cube` (pinned) | `ai-internal` + `proxy` | `cube.yokogawa.tech` |
| `ai-api` | Python 3.12 + FastAPI | `ai-internal` + `proxy` | `api.yokogawa.tech` |
| `ai-web` | Vite build → `nginx:alpine` | `proxy` | `ai.yokogawa.tech` |
| `ai-ingest` | Python 3.12, on demand | `ai-internal` | none |
| `ai-docs-worker` | same image, long-running (Phase 9) | `ai-internal` | none |
| `langfuse` + `lf-db` | official images | `ai-internal` + `proxy` | `lf.yokogawa.tech` |

---

## Rebuild from zero

Assumes: a checkout at `~/ai` on `lin001`, and the Caddy + Authelia + `proxy`
stack already running (it is — this host has served demos for months).

### 1. Secrets

Three `0600` env files under `~/ai/`, never in Git. Every key is listed with no
values in [`.env.example`](.env.example).

```bash
mkdir -p ~/ai && cd ~/ai
install -m 600 /dev/null pg-ai.env
install -m 600 /dev/null api.env
install -m 600 /dev/null langfuse.env
```

Follow the `~/authelia/authelia.env` precedent. The Grafana admin password
sitting in plain text in `~/docker-compose.yml` is a known defect on this host,
not a pattern to copy.

### 2. Phase 1 — `pg-ai`

```bash
./scripts/deploy.sh phase1
```

Creates `/datadisk/pg-ai`, starts `pg-ai`, applies the schema and roles, loads
`equipment.csv` and `tags.csv` with their alias arrays, and — while
`USE_FIXTURES=true` — loads the fixture stand-in for `imh`.

Three roles come out of this, and the split matters: `agent_ro` for the answer
path (SELECT only, everywhere), `cube_rw` for pre-aggregations, and `ingest_rw`
— the only role that writes `doc_chunks`. Set all three passwords in the `0600`
env files; `INGEST_DB_USER` / `INGEST_DB_PASSWORD` are what Phase 3 connects
with, and Phase 3 refuses to start without them.

**Gate:** `pg-ai` healthy, `vector` present, `agent_ro` can SELECT and cannot
INSERT, `ingest_rw` can write `doc_chunks` and nothing else, every equipment item and tag has an alias, `pg-ai` publishes no host
port and is not on the `proxy` network, and `df -h /` is unchanged.
`./scripts/verify.sh` checks all of it.

### 3. Phase 2 — Langfuse

```bash
./scripts/deploy.sh phase2
```

Deployed early on purpose: from here on, every experiment is traced. Then do
the manual steps the script prints — DNS, Caddyfile, Authelia rule, announce
the Authelia restart.

### 4. Phase 3 — knowledge base

Put the controlled documents on the host, in the folders that determine
`doc_type`:

```
/datadisk/ai-docs/{procedures,manuals,rationalisation,design}/
```

```bash
docker compose -f ~/ai-compose.yml run --rm ai-ingest --all
```

It will ask you to confirm the document number, revision and effective date for
every file. **Confirm them properly.** A wrong revision on a procedure is a
safety issue, not a data-quality one. When a new revision lands:

```bash
docker compose -f ~/ai-compose.yml run --rm ai-ingest --supersede WRPS-OPS-014 4
```

That sticks. Re-ingesting a superseded document brings it back superseded, and
`--all` skips it — the flag is not reset by replacing chunks. To undo one:

```bash
docker compose -f ~/ai-compose.yml run --rm ai-ingest --restore WRPS-OPS-014 3
```

This is the SSH path, and it stays. Phase 9 adds the same thing as a screen, so
that an operator who is issued a new document when the PLC logic changes does
not have to find someone with a host login. It does not remove the header
confirmation or the supersede decision — it puts them in front of a named
person and records the answer. See step 8 below.

### 5. Phase 4 — `imh` ⚠ PENDING

**The only true blocker.** Start the conversation now; do not wait for Phase 3.
Agree the read-only login, the table names and key columns, the timestamp
semantics, and an NSG rule allowing `lin001` → `imh` on 1433 only. Then update
§10 of `BUILD-AI-CONTAINERS.md` with the real schema and change
[`db/002_fixtures.sql`](db/002_fixtures.sql) and the Cube models to match.

Until then everything runs on fixtures, and every answer carries a fixture
banner all the way to the operator's screen.

**Two Phase 5 findings are deferred to this phase**, both found by
hand-verifying the Cube measures against fixtures and both left unfixed
because fixing them against fixture data means guessing at `imh`. The wet well
level tag does not join — history is keyed `PS_STN_WET_WELL_LEVEL`, which
`tags.csv` carries only as an alias of `LIT-101` — so a level question can fail
as **"no records found"**, which reads exactly like a genuine absence of data.
And `alarms.first_alarm`/`last_alarm` come back in UTC inside rows whose bucket
labels are in site time. Full detail, and what has to be true to close them, in
[`BUILD-AI-CONTAINERS.md`](BUILD-AI-CONTAINERS.md) Phase 4, "Deferred from
Phase 5"; eval cases `H26` and `H27` fail until they are settled.

### 6. Phases 5–7 — Cube, API, UI

```bash
./scripts/deploy.sh api      # cube + ai-api
./scripts/deploy.sh web      # ai-web
./scripts/verify.sh
```

Each prints the manual DNS/Caddy/Authelia steps. Phase 7 also needs a
**pinpoint DNS record on the DC** → `10.0.0.17` so an operator on `cicore1` can
resolve `ai.yokogawa.tech` — Azure hairpin means LAN hosts cannot reach the
VM's public IP from inside the VNet. `influx.yokogawa.tech` already has this
treatment. **Raise it early**; it depends on someone else and will not surface
until you try it.

### 7. Phase 8 — validate

```bash
python eval/run_eval.py --api https://api.yokogawa.tech
```

62 engineer-reviewable cases in [`eval/testset.jsonl`](eval/testset.jsonl), every
data-dependent one with a **pinned time window** — `imh` is live, and an
unpinned question gives a different answer each run.

Gate: ≥85% overall, ≥95% classification accuracy on Procedural and Advisory,
**zero** contract violations, p95 under 12 s. `run_eval.py` returns non-zero if
any of those is missed. It also marks Historical and Advisory cases
`needs_review`: whether "6" is the *right* number is a judgement for an
engineer with access to `imh`, not something this script can decide.

### 8. Phase 9 — operator document upload

**After Phase 8 passes, not before.** When the PLC logic or the SCADA program
changes, a new document is issued and the assistant is wrong about the plant
until it is ingested. Today that needs SSH to a live shared host. Phase 9 puts
it behind the UI:

```
upload  ─►  pre-scan  ─►  review  ─►  approve  ─►  published
(anyone    (worker,      (a named    (header      (chunks
 with 2FA)  Docling)      publisher)  confirmed)   citable)
```

Nothing is citable until a named person in `AI_DocPublishers` has confirmed the
document number, revision and effective date, and decided what it supersedes —
the same questions `ingest.py` asks at a terminal, asked on a screen and, unlike
the terminal, recorded. The database refuses an approved row without them.

```bash
psql -h pg-ai -U postgres -d plant -f db/004_doc_uploads.sql
psql -h pg-ai -U postgres -d plant -f db/005_doc_actions.sql
psql -h pg-ai -U postgres -d plant -f db/006_doc_pool.sql
sudo install -d -o 10002 -g 10002 /datadisk/ai-docs-inbox   # check df -h first
sudo install -d -o 10002 -g 10002 /datadisk/ai-docs-withdrawn
docker compose -f ~/ai-compose.yml up -d ai-docs-worker
```

**Taking documents out is the other half**, and today there is no way to do it:
`--supersede` needs a revision to *keep*, so a cancelled procedure or a manual
for equipment that has been removed cannot be withdrawn at all. Phase 9 adds
**withdraw** (immediate, reversible, keeps the chunks and the audit trail — this
is what "remove it" almost always means), **restore**, and **purge** (deletes,
irreversible, off unless `ALLOW_PURGE=true` and the publisher types the document
number). All three need the publisher group and a written reason, and all three
are recorded in `doc_actions`, which nothing can delete from.

Withdrawal also moves the file out of `/datadisk/ai-docs` into an archive, so
that folder keeps meaning "the documents this plant runs on". That is
housekeeping, not the safety mechanism: `ingest.py` reads a document's
withdrawal state before replacing its chunks and carries it through, and `--all`
skips withdrawn documents — so re-ingesting cannot resurrect one, whatever
folder it is in. `--restore DOC_NUMBER REVISION` is the way back, and it refuses
while another revision of the same document is live.

Then the manual steps: the `copy_headers` change on the `api.yokogawa.tech`
Caddy block, the `^/docs/.*` Authelia rule **above** the general one, and
`AI_DocPublishers` in AD with **direct** membership.

Design and gate: [`BUILD-AI-CONTAINERS.md`](BUILD-AI-CONTAINERS.md) §16 and
Phase 9. Two gate items matter most. A user who is authenticated but not a
publisher must get a **403 from the API**, tested by calling `api.yokogawa.tech`
directly — the button being hidden proves nothing. And after withdrawing a
document, `ai-ingest --all` must **not** bring it back; run it and check, because
that is the failure that puts a withdrawn procedure back in front of an
operator.

**Choosing what is in the pool** is a third, separate thing, and it is separate
on purpose. `superseded` says *this document is withdrawn or replaced* — a claim
about the document, with a reason and an audit row behind it. `pool_enabled`
says *this document is part of the set we are running with* — a claim about the
corpus, and no comment on whether the document is current. Retrieval requires
both, so putting a withdrawn document back in the pool does **not** make it
citable. A superuser curates the pool to keep it current; the same screen saves
named profiles.

**For demos**, `POST /ask` takes an optional `pool_profile` that narrows
retrieval **for that one request** and changes nothing stored — so "three
documents versus forty-seven" needs nothing undone afterwards on a host other
people are using. Every answer from a reduced pool carries a banner with the
document count, in the same place and for the same reason as the fixture-data
banner: an answer from a trimmed corpus is otherwise indistinguishable from a
complete one. The demo worth showing is not "more documents, better answers" —
it is that with the evidence removed the assistant says *no governing procedure
in the active document set*, rather than degrading into something plausible.

Read the HNSW note at the top of [`db/006_doc_pool.sql`](db/006_doc_pool.sql)
**before** rehearsing that demo. The index covers every embedding and filters
afterwards, so a heavily trimmed pool can appear to collapse retrieval entirely.

**What none of it does:** update `tags.csv`, the Cube models or any
setpoint. A new design document changes what the assistant can *cite*; the
numbers behind Historical and Advisory answers still come from reference data
that is changed in Git and deployed. The review screen asks the reviewer to
confirm they know that, because a document going live while the tag metadata
behind it has not is a gap that is only visible at that moment.

---

## Working on it

```bash
pytest api/tests          # contracts, classifier rules, SQL allow-list. No network.
```

- **The host is live and shared.** Prefer additive changes. Snapshot config
  before editing. **Announce anything that restarts Caddy or Authelia** — it
  logs out every active user, including whoever is mid-demo.
- **Never restart, update or reconfigure `openplc-runtime`** as a side effect
  of AI work. It is the PLC for the demo plant. Its published port 502 is the
  one deliberate exception to the no-published-ports rule on this host, and it
  does not generalise to anything we build.
- **Verify, don't assume.** `docker ps` showing "Up" is not proof.
- **Test every layer without the LLM first.** Prove Cube returns the right
  number by hand. Prove retrieval finds the right procedure by hand. Then wire
  up the agent — otherwise a wrong answer has four possible causes.
- **Do not invent schema.** Inspect, or ask.
- Fix eval failures in the classifier, Cube and ingestion — **not by adding
  instructions to the prompt**. When something fails, add the failing case to
  `eval/testset.jsonl` *before* fixing it.

---

## Repository layout

```
CLAUDE.md                    short rules — what Claude Code keeps front of mind
workflow-map.html            the non-technical explainer — how a question becomes
                             an answer, and what is built. Open it in a browser
BUILD-AI-CONTAINERS.md       the build spec
YAU_Linux_Host_Onboarding.md the host brief (reference; wins on conflict)
compose/                     deployed to ~/ai-compose.yml and ~/langfuse-compose.yml
caddy/ai-routes.caddy        blocks to paste into ~/Caddyfile
authelia/access-rules.md     the rule additions as text — never the real config
db/                          schema, roles, fixtures, and the alias seed CSVs
cube/model/                  alarms, process values, operations, equipment
api/                         FastAPI, classifier, agent, contracts, guardrails
ingest/                      Docling → chunk → embed → pg-ai, plus the
                             Phase 9 upload worker
web/                         React + Vite operator UI
eval/                        62-case test set and the scorecard runner
scripts/                     deploy.sh, verify.sh
docs/                        GITIGNORED — real content on /datadisk/ai-docs
```

---

## Known shortcuts

Deliberate, documented, and not to be shipped. Full list in
[`BUILD-AI-CONTAINERS.md`](BUILD-AI-CONTAINERS.md) §14. The ones that matter most:

- Secrets in `0600` env files, not a vault
- No OT/IT firewall boundary — one flat `10.0.0.0/24` PoC network
- Modbus TCP on port 502 with no authentication or encryption, contained by
  NSG/VPN scope only — **confirm the NSG does not expose it to the internet**
- Single host, no HA: `lin001` is a single point of failure for both the demo
  estate and the simulated plant's PLC
- No automated backup — `pg-ai` needs adding to whatever backup exists
- Document revision metadata entered semi-manually, not integrated with
  document control — Phase 9 records *who* asserted a revision, which is not the
  same as knowing what the current one is
- Uploaded documents are not malware-scanned; type and size checks only

**Section 2 of the build spec must be reviewed with an OT/safety representative
before any operator sees a demo.**