yau-plant-assistant/spec/YAU_Linux_Host_Onboarding.md

YAU PoC Linux Host — Environment Brief & AI Agent Guide

Host: yau-sls-poc-lin001 · Azure Ubuntu 22.04 LTS · Public IP 20.211.144.151 · LAN 10.0.0.17 Owner: Daniel Watson (daniel.watson@yokogawa.com) · Brief current as of: 2026-09-01 Audience: an engineer joining this environment, and the AI coding agent working alongside her.

Amended 2026-09-01 by the WRPS Plant Assistant project, against a docker ps on the host. Dan's brief of 2026-08-12 is unchanged apart from the container inventory, which had gone stale: §3 was counting 20 and naming 19, and omitted openplc-runtime and wireguard entirely. openplc-runtime is live control and appeared nowhere in this file. §3, §6 (published ports) and §12 are corrected; nothing else was touched.

This file is safe to share. It contains no passwords, tokens, or keys — only their locations. Everything you need to actually authenticate comes from Dan over a secure channel (see §2).

Using this with Claude Code: save this file as CLAUDE.md in your project folder. Claude Code loads it automatically at the start of every session, so your agent starts out knowing the host, the stack, the deployment pattern, and the rules in §10 — which exist because breaking them has already caused one outage here.

---

1. What this box is

A secure, general-purpose Docker host and network gateway for the YAU Innovation Team. Two roles:

1. A multi-service platform. Many containerised services for different sales/PoC engagements, added and removed as needed. The container list in §3 is a snapshot, not a fixed design. Publishing a new service under HTTPS with SSO is a ~5-minute, well-worn pattern (§7).
2. A secure gateway into the 10.0.0.0/24 PoC environment, which also holds Windows hosts (a Domain Controller at 10.0.0.5, a CI Server, ~13 machines total). Devices and remote users come in over WireGuard rather than being exposed to the internet.

Design principle: the only internet-facing surface is the Caddy reverse proxy (80/443) and the WireGuard VPN (UDP 443). Databases, MQTT, and other hosts are reached through the box, never directly. Keep it that way.

---

2. Access — what you need from Dan

Ask for these over a secure channel (not email/chat in plaintext):

Item	What it is
yau-sls-poc-lin001_key.pem	SSH private key. Save it locally and chmod 600 it, or SSH refuses to use it
AD account + HTTPS_UserAccess group	Your yau.poc domain login, added to this group — required for every web UI
Duo enrolment	Second factor (push notification) for all web UIs
Linux Machine Config.txt	The credentials file — service admin passwords and API tokens

ssh -i yau-sls-poc-lin001_key.pem azureuser@20.211.144.151

You log in as azureuser — it has sudo and is in the docker group. There are no per-person Linux accounts; everyone shares azureuser, so announce disruptive work before you do it.

Optional but recommended: a WireGuard VPN peer, so you can reach LAN hosts and internal ports directly. Ask Dan to add one (§6).

---

3. The stack at a glance

21 containers, all restart: unless-stopped, all with log rotation (10 MB × 3). docker ps shows 28 today: these 21, plus seven added by the WRPS Plant Assistant project (ai-api, ai-web, pg-ai, cube, cubestore, langfuse, lf-db) — those are described in that project's own repository, not here. Counted 2026-09-01.

Service	URL	Auth	Notes
Caddy	— (the front door)	—	Reverse proxy, automatic Let's Encrypt certs for *.yokogawa.tech
Authelia	auth.yokogawa.tech	— (is the portal)	AD first factor + Duo push second factor; gates everything below
Grafana	grafana.yokogawa.tech	MFA + AD SSO	Dashboards. Auto-logs in as your AD user; new users get org Admin
InfluxDB 2.7	influx.yokogawa.tech	MFA (UI); API bypassed	Historisation — the primary data store. 52 GB and growing
Node-RED	nodered.yokogawa.tech	MFA + own yauadmin login	Flow-based processing
Mosquitto	— (host port 1883)	⚠️ anonymous	General MQTT broker
Forgejo	git.yokogawa.tech	AD (its own, not Authelia)	Internal Git, branded "Yokogawa Git". Not behind Authelia because that breaks git clients
Portainer	portainer.yokogawa.tech	MFA + own admin login	Graphical Docker management — root-equivalent
Dozzle	logs.yokogawa.tech	MFA	Live searchable container logs — your best first debugging stop
Showroom	showroom.yokogawa.tech	1FA only (no Duo)	Static demo site, gated to AD group Showroom_Access
EQP Licence	licence.yokogawa.tech	MFA	Licence issuer; signing key mounted read-only, never baked into the image
Telegraf	—	—	Host + container metrics → Influx telemetry bucket (30-day retention)
Watchtower	—	—	Auto-updates a safe subset only, Sundays 04:00 AEST
ChirpStack ⚠️	chirpstack.yokogawa.tech	MFA	LoRaWAN (AU915) — future capability, running but NOT configured. Safe to ignore or stop
WireGuard	— (host port 443/udp)	peer keys	The VPN into 10.0.0.0/24. Adding peers: §6
OpenPLC Runtime ⚠️⚠️	— (host ports 502/tcp, 8443/tcp, bound to 10.0.0.17)	⚠️ none	LIVE CONTROL. The soft PLC for the Waterloo Road Pump Station demo, polled over Modbus TCP by CI Server on cicore1. Never restart, update or reconfigure it as a side effect of other work, and never change the 10.0.0.17 binding to 0.0.0.0 — that binding is the only thing keeping unauthenticated Modbus off the internet. Not in Watchtower's update list, and must not be added

Plus chirpstack-postgres/-redis/-mqtt/-gateway-bridge (all ChirpStack support) and authelia-portal (nginx that brands the login page).

Data flow

Field devices / Windows hosts (10.0.0.0/24)  ──VPN/LAN──┐
                                                        ▼
   Telegraf agents ─┐                        ┌──── Linux Docker host ────┐
   MQTT (Mosquitto) ┼──► Node-RED ──────────►│  InfluxDB (historisation) │──► Grafana
   CI Server ───────┘                        │      on /datadisk         │    (dashboards)
                                             └───────────────────────────┘
   Everything web-facing is published through Caddy (HTTPS) and gated by Authelia (AD + Duo).

---

4. Where things live

All configuration is in /home/azureuser — flat, one Compose file per service group:

~/docker-compose.yml          caddy, grafana, influxdb, nodered, mosquitto   ← core stack
~/chirpstack-compose.yml      chirpstack + postgres/redis/mqtt/gateway-bridge
~/wg-compose.yml              wireguard
~/authelia-compose.yml        authelia          ~/authelia-portal-compose.yml  branding proxy
~/forgejo-compose.yml         forgejo           ~/eqp-compose.yml              licence issuer
~/portainer-compose.yml       portainer         ~/dozzle-compose.yml           log viewer
~/telegraf-compose.yml        telegraf          ~/watchtower-compose.yml       auto-updater
~/showroom-compose.yml        showroom

~/Caddyfile                   all reverse-proxy routes  (+ many .bak-* snapshots)
~/authelia/configuration.yml  auth rules — root-owned, edit with sudo
~/authelia/authelia.env       secrets, 0600
~/telegraf/                   telegraf.conf + Influx tokens (0600)
~/mosquitto/config/           broker config + passwordfile
~/showroom-site/              static demo content (rsync target)

Every *-compose.yml shares the default Compose project name azureuser. Running docker compose against a single file therefore prints a harmless "orphan containers" warning — ignore it.

Two disks, and the split matters:

Mount	Size	Used	Contents
/	62 GB	21%	OS, Docker images, most volumes
/datadisk	128 GB	43%	InfluxDB (52 GB), Forgejo

---

5. Authentication model

Understand this before you deploy anything.

• Authelia sits in front of nearly everything via Caddy's forward_auth. First factor is Active Directory (ldaps://10.0.0.5:636, base DC=yau,DC=poc), restricted to the AD group HTTPS_UserAccess. Second factor is Duo Push — TOTP and WebAuthn are deliberately disabled so no email enrolment is needed.
• SSO: one login covers all *.yokogawa.tech. Sessions are held in memory, so restarting Authelia logs everyone out. That is also the supported way to refresh someone's group membership.
• Grafana does true AD SSO — it trusts Authelia's Remote-User header via auth-proxy, whitelisted to the 172.18.0.0/16 Docker subnet. No second login.
• Node-RED and InfluxDB OSS keep their own logins behind the MFA gate — neither supports AD or proxy-header auth. Not a misconfiguration.
• API bypass: Influx ^/api/v2/(write|query) and /health skip MFA so devices and Telegraf agents can write. If you add machine-to-machine endpoints, they need a similar explicit bypass.

⚠️ The AD gotcha that will bite you

Authelia and Forgejo both resolve DIRECT group membership only. A user who is in HTTPS_UserAccess via a nested group (e.g. through Domain Admins) will not get access. Add people as direct members. The service account svc-authelia is read-only and cannot change membership — that must be done on the DC with Add-ADGroupMember.

And if someone is added to a group after they logged in, they'll still be denied until the session refreshes: docker compose -f ~/authelia-compose.yml restart authelia.

---

6. Networking

• Caddy terminates HTTPS and proxies over the proxy Docker network (external; every internet-facing service joins it). DNS: all *.yokogawa.tech A records → 20.211.144.151.
• WireGuard: endpoint yau.poc.vpn.yokogawa.tech, UDP 443 (chosen to traverse restrictive corporate firewalls). Tunnel subnet is 10.13.13.0/24, deliberately separate from the LAN. Peers reach 10.0.0.0/24 via ip_forward + container MASQUERADE, so LAN hosts see traffic as coming from 10.0.0.17. Existing peers: dan, laptop, mac, office, rut1. Add one by appending to PEERS in ~/wg-compose.yml, then docker compose -f ~/wg-compose.yml up -d --force-recreate (existing peers are preserved), then docker exec wireguard /app/show-peer <name> for the config/QR code.
• Published host ports: 80/443 tcp (Caddy), 443/udp (WireGuard), 1883/tcp (MQTT), 1700/udp (LoRaWAN packet forwarder), and 502/tcp + 8443/tcp (OpenPLC Runtime) — the last two bound to 10.0.0.17, not 0.0.0.0. That binding is a security control: Modbus has no authentication or encryption, so the bind plus the NSG is all that keeps it off the internet. Do not change it.
• Firewall: host ufw is inactive — inbound filtering is entirely the Azure NSG. Opening a port means editing the NSG in the Azure portal, not the host.
• Azure hairpin gotcha: LAN hosts cannot reach the VM's public IP from inside the VNet. Solved with a pinpoint DNS record on the DC: influx.yokogawa.tech → 10.0.0.17. If you add a service that LAN machines must reach by hostname, it needs the same treatment.

---

7. ⭐ How to deploy a new service (the pattern you'll use most)

This is the well-worn path. Follow it and your service gets HTTPS, a cert, and AD+Duo SSO for free.

1. Write ~/<name>-compose.yml. Join the proxy network. Do not publish host ports — reach it through Caddy. If it stores growing data, bind-mount under /datadisk, not the root disk.

services:
  myservice:
    image: myimage:tag
    container_name: myservice
    restart: unless-stopped
    networks: [proxy]
    volumes:
      - /datadisk/myservice:/data      # only if it stores real data
networks:
  proxy:
    external: true

2. Add a block to ~/Caddyfile:

myservice.yokogawa.tech {
  import authelia
  reverse_proxy myservice:8080
}

import authelia is the shared MFA gate — omit it only with a deliberate reason (Forgejo omits it because forward-auth breaks git clients).

3. Add the domain to the Authelia rule in ~/authelia/configuration.yml under the HTTPS_UserAccess two_factor rule. The file is root-owned — edit with sudo. There's a helper, ~/apply_rule.py, for rewriting the trailing rule. Back the file up first; you'll find plenty of .bak-* precedents.

4. Apply and verify:

docker compose -f ~/<name>-compose.yml up -d
docker exec caddy caddy reload --config /etc/caddy/Caddyfile
docker compose -f ~/authelia-compose.yml restart authelia   # note: logs everyone out
curl -sI https://myservice.yokogawa.tech        # expect 302 → auth portal

5. Add the DNS A record myservice.yokogawa.tech → 20.211.144.151. Without it Caddy cannot get a certificate. Ask Dan — DNS is not managed on this host.

---

8. Operating it

# Status
docker ps
docker stats --no-stream
df -h / /datadisk                     # watch both

# Logs — or just use https://logs.yokogawa.tech (Dozzle), which is nicer
docker logs -f grafana
docker logs --tail 100 influxdb

# Apply changes / restart
docker compose -f ~/docker-compose.yml up -d
docker compose -f ~/docker-compose.yml restart grafana

# Bring everything up (also happens automatically on reboot)
cd ~ && docker compose -f docker-compose.yml -f chirpstack-compose.yml -f wg-compose.yml up -d

# Reload Caddy after editing the Caddyfile
docker exec caddy caddy reload --config /etc/caddy/Caddyfile

Updates: Watchtower auto-updates only grafana nodered portainer authelia wireguard (Sundays 04:00 AEST). Pinned images — influxdb:2.7, caddy:2, postgres:14, redis:7-alpine, chirpstack, mosquitto — are never touched automatically. Update those by hand: docker compose -f <file> pull <svc> && docker compose -f <file> up -d <svc>.

Monitoring: the Grafana dashboard "YAU Host & Containers — Health" (/d/yau-host-health) shows both disks, and an alert fires above 80%. Note the alert is in-UI only — no email or Teams contact point is wired up, so nobody gets pushed a notification. Worth fixing.

---

9. Known issues — inherited, not yours

Issue	Detail
Mosquitto is open	allow_anonymous true, no TLS, on internet-exposed port 1883. A passwordfile exists but isn't enforced. Restrict via NSG/VPN or enable auth before putting anything real on it
Secrets in plaintext	Admin passwords and tokens sit in Linux Machine Config.txt and in Compose env vars (the Grafana admin password is literally in ~/docker-compose.yml)
Docker socket exposure	Portainer and Watchtower mount it read-write = root-equivalent host control; Dozzle and Telegraf mount it read-only. All are behind MFA — keep it that way
No host firewall	Entirely dependent on correct Azure NSG rules
No automated backup	Backups are manual. The most recent is Backups/vm-config-20260812/. Influx's 52 GB is not in it — that needs an Azure disk snapshot
Alerts don't notify	Disk alert shows in Grafana's UI only
ChirpStack log spam	Unconfigured gateway-bridge loops and once generated 5.9 GB of logs. Rotation caps it now; stopping the stack is the real fix if LoRaWAN isn't needed
Single shared login	Everyone is azureuser; no per-person audit trail on the host

---

10. Rules for the AI agent

Read these before proposing changes to this host. Each one comes from something that already went wrong or is a live constraint.

1. Never put growing data on the root disk. / is only 62 GB. InfluxDB data must stay bind-mounted at /datadisk/influx. A previous migration copied 47 GB to /datadisk but never repointed the container or deleted the original — root hit 100%, and Grafana died with database or disk is full. New services with real data go on /datadisk.
2. Never move the WireGuard tunnel back onto 10.0.0.0/24. It used to overlap the server LAN, colliding with Azure-reserved .1–.3 and the real host at .5. It lives on 10.13.13.0/24.
3. Don't remove or bypass Authelia to "simplify" access. It is the AD login — a plain static site cannot AD-authenticate without it. Removing it silently makes services public.
4. ~/authelia/configuration.yml is root-owned. Edit with sudo, back it up first, and know that restarting Authelia logs out every active user.
5. AD group membership must be DIRECT (§5). Nested membership silently fails to grant access.
6. Don't publish host ports for new services. Go through Caddy on the proxy network. Every published port is a new NSG dependency and a new attack surface.
7. Watchtower's update list is deliberately short. Don't add pinned database or proxy images to it — unattended major-version bumps of Influx/Postgres/Caddy are how you lose a weekend.
8. Verify before declaring success. docker ps showing "Up" is not proof; curl -sI the public URL and expect a 302 to the auth portal, and check docker logs for the container.
9. This host is shared and live — it runs customer-facing demos. Announce restarts of Caddy or Authelia (they interrupt everyone). Prefer additive changes; snapshot config before editing (the .bak-<purpose>-<date> convention is already established throughout ~).
10. Don't commit secrets. Linux Machine Config.txt, *.pem, *.token, and authelia.env never go into Git, and never into a file intended for sharing.

---

11. Deeper reference

These live in the same project folder as this brief (ask Dan — several contain secrets):

File	Covers
Host_Documentation.md	The full ops manual — every service, network detail, and a dated change log explaining why things are the way they are. Read this second.
MFA_Duo_Setup_Plan.md	Authelia + Duo design and config templates
WireGuard_RUT_Setup.md	Onboarding RUT240/RUT950 field routers onto the VPN
Windows_Telegraf_GPO_Deployment.md, Install-Telegraf-Windows.ps1, Deploy-Telegraf.ps1	Rolling Telegraf agents out to the Windows fleet
Showroom/deploy/	Showroom demo site deploy kit and runbook
Backups/vm-config-20260812/	Full config backup + restore notes ⚠️ contains secrets — do not share
Linux Machine Config.txt	The credentials file ⚠️ secrets

---

12. Suggested first day

1. Get the SSH key, AD account in HTTPS_UserAccess, and Duo enrolment from Dan (§2).
2. SSH in; run docker ps and df -h. Confirm every container is up and both disks are healthy. Don't check against a fixed number — services come and go (§3).
3. Log into grafana.yokogawa.tech — this exercises the whole AD + Duo + SSO path in one go.
4. Open logs.yokogawa.tech (Dozzle) and portainer.yokogawa.tech to get a feel for the stack.
5. Read Host_Documentation.md, especially the change log — it explains the scars.
6. Deploy something trivial (a nginx:alpine hello-world) end-to-end using §7. Doing the full compose → Caddy → Authelia → DNS loop once on a throwaway service is the fastest way to learn this environment, and it's safe. Tear it down afterwards.

---

A note on adding AI features here

Nothing on this host currently calls an LLM, so you'll be first. Things worth knowing up front:

• Outbound internet works (Let's Encrypt and image pulls depend on it), so a container calling the Anthropic API will work — but API keys must not go into a Compose file in plaintext the way the Grafana password did. Use an env file at 0600 (the ~/telegraf/telegraf.env and ~/authelia/authelia.env pattern) and keep it out of Git.
• Data is right here: InfluxDB holds the historised time-series, Mosquitto carries live MQTT, and Node-RED is already wired to both — it's the path of least resistance for a prototype.
• Follow §7 for anything with a UI so it lands behind AD + Duo like everything else. An unauthenticated AI endpoint on a box this exposed is not acceptable.
• If a service needs a GPU or sustained heavy compute, this VM is not it — raise sizing with Dan before designing around it.