yau-plant-assistant/spec/YAU_Linux_Host_Onboarding.md

# YAU PoC Linux Host — Environment Brief & AI Agent Guide

> **Host:** `yau-sls-poc-lin001` · Azure Ubuntu 22.04 LTS · Public IP `20.211.144.151` · LAN `10.0.0.17`
> **Owner:** Daniel Watson (daniel.watson@yokogawa.com) · **Brief current as of:** 2026-09-01
> **Audience:** an engineer joining this environment, and the AI coding agent working alongside her.

> **Amended 2026-09-01** by the WRPS Plant Assistant project, against a `docker ps` on the
> host. Dan's brief of 2026-08-12 is unchanged apart from the container inventory, which had
> gone stale: §3 was counting 20 and naming 19, and omitted `openplc-runtime` and `wireguard`
> entirely. `openplc-runtime` is live control and appeared nowhere in this file. §3, §6
> (published ports) and §12 are corrected; nothing else was touched.

**This file is safe to share.** It contains no passwords, tokens, or keys — only their *locations*.
Everything you need to actually authenticate comes from Dan over a secure channel (see §2).

**Using this with Claude Code:** save this file as `CLAUDE.md` in your project folder. Claude Code
loads it automatically at the start of every session, so your agent starts out knowing the host,
the stack, the deployment pattern, and the rules in §10 — which exist because breaking them has
already caused one outage here.

---

## 1. What this box is

A **secure, general-purpose Docker host and network gateway** for the YAU Innovation Team. Two roles:

1. **A multi-service platform.** Many containerised services for different sales/PoC engagements,
   added and removed as needed. The container list in §3 is a snapshot, not a fixed design.
   Publishing a new service under HTTPS with SSO is a ~5-minute, well-worn pattern (§7).
2. **A secure gateway** into the `10.0.0.0/24` PoC environment, which also holds Windows hosts
   (a Domain Controller at `10.0.0.5`, a CI Server, ~13 machines total). Devices and remote users
   come in over WireGuard rather than being exposed to the internet.

**Design principle:** the only internet-facing surface is the Caddy reverse proxy (80/443) and the
WireGuard VPN (UDP 443). Databases, MQTT, and other hosts are reached *through* the box, never
directly. Keep it that way.

---

## 2. Access — what you need from Dan

Ask for these over a secure channel (not email/chat in plaintext):

| Item | What it is |
|------|-----------|
| `yau-sls-poc-lin001_key.pem` | SSH private key. Save it locally and `chmod 600` it, or SSH refuses to use it |
| AD account + `HTTPS_UserAccess` group | Your `yau.poc` domain login, added to this group — required for **every** web UI |
| Duo enrolment | Second factor (push notification) for all web UIs |
| `Linux Machine Config.txt` | The credentials file — service admin passwords and API tokens |

```bash
ssh -i yau-sls-poc-lin001_key.pem azureuser@20.211.144.151
```

You log in as **`azureuser`** — it has `sudo` and is in the `docker` group. There are no per-person
Linux accounts; everyone shares `azureuser`, so **announce disruptive work** before you do it.

Optional but recommended: a WireGuard VPN peer, so you can reach LAN hosts and internal ports
directly. Ask Dan to add one (§6).

---

## 3. The stack at a glance

**21 containers**, all `restart: unless-stopped`, all with log rotation (10 MB × 3).
`docker ps` shows **28** today: these 21, plus seven added by the WRPS Plant Assistant
project (`ai-api`, `ai-web`, `pg-ai`, `cube`, `cubestore`, `langfuse`, `lf-db`) — those
are described in that project's own repository, not here. Counted 2026-09-01.

| Service | URL | Auth | Notes |
|---------|-----|------|-------|
| **Caddy** | — (the front door) | — | Reverse proxy, automatic Let's Encrypt certs for `*.yokogawa.tech` |
| **Authelia** | `auth.yokogawa.tech` | — (is the portal) | AD first factor + Duo push second factor; gates everything below |
| **Grafana** | `grafana.yokogawa.tech` | MFA + AD SSO | Dashboards. Auto-logs in as your AD user; new users get org **Admin** |
| **InfluxDB 2.7** | `influx.yokogawa.tech` | MFA (UI); API bypassed | Historisation — the primary data store. **52 GB and growing** |
| **Node-RED** | `nodered.yokogawa.tech` | MFA + own `yauadmin` login | Flow-based processing |
| **Mosquitto** | — (host port 1883) | ⚠️ anonymous | General MQTT broker |
| **Forgejo** | `git.yokogawa.tech` | AD (its own, **not** Authelia) | Internal Git, branded "Yokogawa Git". Not behind Authelia because that breaks git clients |
| **Portainer** | `portainer.yokogawa.tech` | MFA + own admin login | Graphical Docker management — **root-equivalent** |
| **Dozzle** | `logs.yokogawa.tech` | MFA | Live searchable container logs — your best first debugging stop |
| **Showroom** | `showroom.yokogawa.tech` | **1FA only** (no Duo) | Static demo site, gated to AD group `Showroom_Access` |
| **EQP Licence** | `licence.yokogawa.tech` | MFA | Licence issuer; signing key mounted read-only, never baked into the image |
| **Telegraf** | — | — | Host + container metrics → Influx `telemetry` bucket (30-day retention) |
| **Watchtower** | — | — | Auto-updates a **safe subset only**, Sundays 04:00 AEST |
| **ChirpStack** ⚠️ | `chirpstack.yokogawa.tech` | MFA | LoRaWAN (AU915) — **future capability, running but NOT configured**. Safe to ignore or stop |
| **WireGuard** | — (host port 443/udp) | peer keys | The VPN into `10.0.0.0/24`. Adding peers: §6 |
| **OpenPLC Runtime** ⚠️⚠️ | — (host ports **502/tcp, 8443/tcp**, bound to `10.0.0.17`) | ⚠️ **none** | **LIVE CONTROL.** The soft PLC for the Waterloo Road Pump Station demo, polled over Modbus TCP by CI Server on `cicore1`. **Never restart, update or reconfigure it as a side effect of other work**, and never change the `10.0.0.17` binding to `0.0.0.0` — that binding is the only thing keeping unauthenticated Modbus off the internet. Not in Watchtower's update list, and must not be added |

Plus `chirpstack-postgres`/`-redis`/`-mqtt`/`-gateway-bridge` (all ChirpStack support) and
`authelia-portal` (nginx that brands the login page).

### Data flow

```
Field devices / Windows hosts (10.0.0.0/24)  ──VPN/LAN──┐
                                                        ▼
   Telegraf agents ─┐                        ┌──── Linux Docker host ────┐
   MQTT (Mosquitto) ┼──► Node-RED ──────────►│  InfluxDB (historisation) │──► Grafana
   CI Server ───────┘                        │      on /datadisk         │    (dashboards)
                                             └───────────────────────────┘
   Everything web-facing is published through Caddy (HTTPS) and gated by Authelia (AD + Duo).
```

---

## 4. Where things live

All configuration is in **`/home/azureuser`** — flat, one Compose file per service group:

```
~/docker-compose.yml          caddy, grafana, influxdb, nodered, mosquitto   ← core stack
~/chirpstack-compose.yml      chirpstack + postgres/redis/mqtt/gateway-bridge
~/wg-compose.yml              wireguard
~/authelia-compose.yml        authelia          ~/authelia-portal-compose.yml  branding proxy
~/forgejo-compose.yml         forgejo           ~/eqp-compose.yml              licence issuer
~/portainer-compose.yml       portainer         ~/dozzle-compose.yml           log viewer
~/telegraf-compose.yml        telegraf          ~/watchtower-compose.yml       auto-updater
~/showroom-compose.yml        showroom

~/Caddyfile                   all reverse-proxy routes  (+ many .bak-* snapshots)
~/authelia/configuration.yml  auth rules — root-owned, edit with sudo
~/authelia/authelia.env       secrets, 0600
~/telegraf/                   telegraf.conf + Influx tokens (0600)
~/mosquitto/config/           broker config + passwordfile
~/showroom-site/              static demo content (rsync target)
```

Every `*-compose.yml` shares the default Compose project name `azureuser`. Running `docker compose`
against a single file therefore prints a **harmless "orphan containers" warning** — ignore it.

**Two disks, and the split matters:**

| Mount | Size | Used | Contents |
|-------|------|------|----------|
| `/` | 62 GB | 21% | OS, Docker images, most volumes |
| `/datadisk` | 128 GB | 43% | **InfluxDB (52 GB)**, Forgejo |

---

## 5. Authentication model

Understand this before you deploy anything.

- **Authelia** sits in front of nearly everything via Caddy's `forward_auth`. First factor is
  **Active Directory** (`ldaps://10.0.0.5:636`, base `DC=yau,DC=poc`), restricted to the AD group
  **`HTTPS_UserAccess`**. Second factor is **Duo Push** — TOTP and WebAuthn are deliberately
  disabled so no email enrolment is needed.
- **SSO:** one login covers all `*.yokogawa.tech`. Sessions are held **in memory**, so restarting
  Authelia logs everyone out. That is also the supported way to refresh someone's group membership.
- **Grafana** does true AD SSO — it trusts Authelia's `Remote-User` header via auth-proxy,
  whitelisted to the `172.18.0.0/16` Docker subnet. No second login.
- **Node-RED** and **InfluxDB OSS** keep their own logins behind the MFA gate — neither supports
  AD or proxy-header auth. Not a misconfiguration.
- **API bypass:** Influx `^/api/v2/(write|query)` and `/health` skip MFA so devices and Telegraf
  agents can write. If you add machine-to-machine endpoints, they need a similar explicit bypass.

> ### ⚠️ The AD gotcha that will bite you
> Authelia and Forgejo both resolve **DIRECT** group membership only. A user who is in
> `HTTPS_UserAccess` *via a nested group* (e.g. through `Domain Admins`) will **not** get access.
> Add people as direct members. The service account `svc-authelia` is **read-only** and cannot
> change membership — that must be done on the DC with `Add-ADGroupMember`.
>
> And if someone is added to a group *after* they logged in, they'll still be denied until the
> session refreshes: `docker compose -f ~/authelia-compose.yml restart authelia`.

---

## 6. Networking

- **Caddy** terminates HTTPS and proxies over the `proxy` Docker network (external; every
  internet-facing service joins it). DNS: all `*.yokogawa.tech` A records → `20.211.144.151`.
- **WireGuard:** endpoint `yau.poc.vpn.yokogawa.tech`, **UDP 443** (chosen to traverse restrictive
  corporate firewalls). Tunnel subnet is **`10.13.13.0/24`**, deliberately separate from the LAN.
  Peers reach `10.0.0.0/24` via `ip_forward` + container MASQUERADE, so LAN hosts see traffic as
  coming from `10.0.0.17`. Existing peers: `dan`, `laptop`, `mac`, `office`, `rut1`.
  Add one by appending to `PEERS` in `~/wg-compose.yml`, then
  `docker compose -f ~/wg-compose.yml up -d --force-recreate` (existing peers are preserved), then
  `docker exec wireguard /app/show-peer <name>` for the config/QR code.
- **Published host ports:** 80/443 tcp (Caddy), 443/udp (WireGuard), 1883/tcp (MQTT),
  1700/udp (LoRaWAN packet forwarder), and **502/tcp + 8443/tcp (OpenPLC Runtime)** —
  the last two bound to `10.0.0.17`, not `0.0.0.0`. That binding is a security control:
  Modbus has no authentication or encryption, so the bind plus the NSG is all that keeps
  it off the internet. Do not change it.
- **Firewall:** host `ufw` is **inactive** — inbound filtering is entirely the **Azure NSG**.
  Opening a port means editing the NSG in the Azure portal, not the host.
- **Azure hairpin gotcha:** LAN hosts cannot reach the VM's *public* IP from inside the VNet.
  Solved with a pinpoint DNS record on the DC: `influx.yokogawa.tech → 10.0.0.17`. If you add a
  service that LAN machines must reach by hostname, it needs the same treatment.

---

## 7. ⭐ How to deploy a new service (the pattern you'll use most)

This is the well-worn path. Follow it and your service gets HTTPS, a cert, and AD+Duo SSO for free.

**1. Write `~/<name>-compose.yml`.** Join the `proxy` network. Do **not** publish host ports —
reach it through Caddy. If it stores growing data, bind-mount under `/datadisk`, not the root disk.

```yaml
services:
  myservice:
    image: myimage:tag
    container_name: myservice
    restart: unless-stopped
    networks: [proxy]
    volumes:
      - /datadisk/myservice:/data      # only if it stores real data
networks:
  proxy:
    external: true
```

**2. Add a block to `~/Caddyfile`:**

```
myservice.yokogawa.tech {
  import authelia
  reverse_proxy myservice:8080
}
```

`import authelia` is the shared MFA gate — omit it only with a deliberate reason (Forgejo omits it
because forward-auth breaks git clients).

**3. Add the domain to the Authelia rule** in `~/authelia/configuration.yml` under the
`HTTPS_UserAccess` `two_factor` rule. The file is **root-owned — edit with `sudo`**.
There's a helper, `~/apply_rule.py`, for rewriting the trailing rule. Back the file up first;
you'll find plenty of `.bak-*` precedents.

**4. Apply and verify:**

```bash
docker compose -f ~/<name>-compose.yml up -d
docker exec caddy caddy reload --config /etc/caddy/Caddyfile
docker compose -f ~/authelia-compose.yml restart authelia   # note: logs everyone out
curl -sI https://myservice.yokogawa.tech        # expect 302 → auth portal
```

**5. Add the DNS A record** `myservice.yokogawa.tech → 20.211.144.151`. Without it Caddy cannot
get a certificate. Ask Dan — DNS is not managed on this host.

---

## 8. Operating it

```bash
# Status
docker ps
docker stats --no-stream
df -h / /datadisk                     # watch both

# Logs — or just use https://logs.yokogawa.tech (Dozzle), which is nicer
docker logs -f grafana
docker logs --tail 100 influxdb

# Apply changes / restart
docker compose -f ~/docker-compose.yml up -d
docker compose -f ~/docker-compose.yml restart grafana

# Bring everything up (also happens automatically on reboot)
cd ~ && docker compose -f docker-compose.yml -f chirpstack-compose.yml -f wg-compose.yml up -d

# Reload Caddy after editing the Caddyfile
docker exec caddy caddy reload --config /etc/caddy/Caddyfile
```

**Updates:** Watchtower auto-updates **only** `grafana nodered portainer authelia wireguard`
(Sundays 04:00 AEST). Pinned images — `influxdb:2.7`, `caddy:2`, `postgres:14`, `redis:7-alpine`,
chirpstack, mosquitto — are never touched automatically. Update those by hand:
`docker compose -f <file> pull <svc> && docker compose -f <file> up -d <svc>`.

**Monitoring:** the Grafana dashboard **"YAU Host & Containers — Health"** (`/d/yau-host-health`)
shows both disks, and an alert fires above 80%. Note the alert is **in-UI only** — no email or
Teams contact point is wired up, so nobody gets pushed a notification. Worth fixing.

---

## 9. Known issues — inherited, not yours

| Issue | Detail |
|-------|--------|
| **Mosquitto is open** | `allow_anonymous true`, no TLS, on internet-exposed port 1883. A `passwordfile` exists but isn't enforced. Restrict via NSG/VPN or enable auth before putting anything real on it |
| **Secrets in plaintext** | Admin passwords and tokens sit in `Linux Machine Config.txt` and in Compose env vars (the Grafana admin password is literally in `~/docker-compose.yml`) |
| **Docker socket exposure** | Portainer and Watchtower mount it **read-write** = root-equivalent host control; Dozzle and Telegraf mount it read-only. All are behind MFA — keep it that way |
| **No host firewall** | Entirely dependent on correct Azure NSG rules |
| **No automated backup** | Backups are manual. The most recent is `Backups/vm-config-20260812/`. **Influx's 52 GB is not in it** — that needs an Azure disk snapshot |
| **Alerts don't notify** | Disk alert shows in Grafana's UI only |
| **ChirpStack log spam** | Unconfigured gateway-bridge loops and once generated 5.9 GB of logs. Rotation caps it now; stopping the stack is the real fix if LoRaWAN isn't needed |
| **Single shared login** | Everyone is `azureuser`; no per-person audit trail on the host |

---

## 10. Rules for the AI agent

**Read these before proposing changes to this host.** Each one comes from something that already
went wrong or is a live constraint.

1. **Never put growing data on the root disk.** `/` is only 62 GB. InfluxDB data **must** stay
   bind-mounted at `/datadisk/influx`. A previous migration copied 47 GB to `/datadisk` but never
   repointed the container or deleted the original — root hit 100%, and Grafana died with
   `database or disk is full`. New services with real data go on `/datadisk`.
2. **Never move the WireGuard tunnel back onto `10.0.0.0/24`.** It used to overlap the server LAN,
   colliding with Azure-reserved `.1`–`.3` and the real host at `.5`. It lives on `10.13.13.0/24`.
3. **Don't remove or bypass Authelia** to "simplify" access. It *is* the AD login — a plain static
   site cannot AD-authenticate without it. Removing it silently makes services public.
4. **`~/authelia/configuration.yml` is root-owned.** Edit with `sudo`, back it up first, and know
   that restarting Authelia logs out every active user.
5. **AD group membership must be DIRECT** (§5). Nested membership silently fails to grant access.
6. **Don't publish host ports** for new services. Go through Caddy on the `proxy` network. Every
   published port is a new NSG dependency and a new attack surface.
7. **Watchtower's update list is deliberately short.** Don't add pinned database or proxy images
   to it — unattended major-version bumps of Influx/Postgres/Caddy are how you lose a weekend.
8. **Verify before declaring success.** `docker ps` showing "Up" is not proof; `curl -sI` the
   public URL and expect a 302 to the auth portal, and check `docker logs` for the container.
9. **This host is shared and live** — it runs customer-facing demos. Announce restarts of Caddy or
   Authelia (they interrupt everyone). Prefer additive changes; snapshot config before editing
   (the `.bak-<purpose>-<date>` convention is already established throughout `~`).
10. **Don't commit secrets.** `Linux Machine Config.txt`, `*.pem`, `*.token`, and `authelia.env`
    never go into Git, and never into a file intended for sharing.

---

## 11. Deeper reference

These live in the same project folder as this brief (ask Dan — several contain secrets):

| File | Covers |
|------|--------|
| `Host_Documentation.md` | The full ops manual — every service, network detail, and a dated change log explaining *why* things are the way they are. **Read this second.** |
| `MFA_Duo_Setup_Plan.md` | Authelia + Duo design and config templates |
| `WireGuard_RUT_Setup.md` | Onboarding RUT240/RUT950 field routers onto the VPN |
| `Windows_Telegraf_GPO_Deployment.md`, `Install-Telegraf-Windows.ps1`, `Deploy-Telegraf.ps1` | Rolling Telegraf agents out to the Windows fleet |
| `Showroom/deploy/` | Showroom demo site deploy kit and runbook |
| `Backups/vm-config-20260812/` | Full config backup + restore notes ⚠️ **contains secrets — do not share** |
| `Linux Machine Config.txt` | The credentials file ⚠️ **secrets** |

---

## 12. Suggested first day

1. Get the SSH key, AD account in `HTTPS_UserAccess`, and Duo enrolment from Dan (§2).
2. SSH in; run `docker ps` and `df -h`. Confirm every container is up and both disks are
   healthy. Don't check against a fixed number — services come and go (§3).
3. Log into `grafana.yokogawa.tech` — this exercises the whole AD + Duo + SSO path in one go.
4. Open `logs.yokogawa.tech` (Dozzle) and `portainer.yokogawa.tech` to get a feel for the stack.
5. Read `Host_Documentation.md`, especially the change log — it explains the scars.
6. Deploy something trivial (a `nginx:alpine` hello-world) end-to-end using §7. Doing the full
   compose → Caddy → Authelia → DNS loop once on a throwaway service is the fastest way to learn
   this environment, and it's safe. Tear it down afterwards.

---

### A note on adding AI features here

Nothing on this host currently calls an LLM, so you'll be first. Things worth knowing up front:

- **Outbound internet works** (Let's Encrypt and image pulls depend on it), so a container calling
  the Anthropic API will work — but **API keys must not** go into a Compose file in plaintext the
  way the Grafana password did. Use an env file at `0600` (the `~/telegraf/telegraf.env` and
  `~/authelia/authelia.env` pattern) and keep it out of Git.
- **Data is right here:** InfluxDB holds the historised time-series, Mosquitto carries live MQTT,
  and Node-RED is already wired to both — it's the path of least resistance for a prototype.
- **Follow §7 for anything with a UI** so it lands behind AD + Duo like everything else. An
  unauthenticated AI endpoint on a box this exposed is not acceptable.
- If a service needs a GPU or sustained heavy compute, this VM is not it — raise sizing with Dan
  before designing around it.