diff --git a/02-environment/MIGRATION.md b/02-environment/MIGRATION.md new file mode 100644 index 0000000..09dc2c2 --- /dev/null +++ b/02-environment/MIGRATION.md @@ -0,0 +1,359 @@ +# Migrating `openplc-runtime` to another Docker host (air-gapped) + +> [!NOTE] +> **COMPLETED 2026-08-19. This is the as-built record of the migration, kept as a +> runbook for repeating it.** The PLC now runs on `yau-sls-poc-lin001` +> (`10.0.0.17`) and CI Server on `yau-poc-cicore1` (`10.0.0.21`) polls it live. +> The retired source host `dev-ubuntu` is no longer part of this project. +> +> **Part E was not executed as written.** The service was deployed as its own +> compose file on its own network rather than appended to the shared stack — see +> the "What was actually done" note in Part E. The as-built configuration is +> `openplc-compose.yml` and `openplc-container.md`. + +Source: `dev-ubuntu` (SSH, now retired) · Destination: `yau-sls-poc-lin001`, +Portainer UI + SSH · both x86-64. Written 2026-08-18 against the live container. + +Why `docker commit` and not `docker pull` on the far side: the compiled PLC +program lives in the container's **writable layer** (`/workdir/build/libplc_*.so`, +`/workdir/core/generated/`), not in the volume. Only a commit carries it. + +Two artifacts move: + +- `openplc-image.tar.gz` — committed image (~400–500 MB gz, 1.08 GB virtual) +- `openplc-vol.tar.gz` — the 20 KB volume (`restapi.db` = users + program record, `.env` = JWT secret) + +--- + +## PART A — On the SOURCE host (SSH: `ssh dev-ubuntu`) — ✅ DONE 2026-08-19 + +Executed in the order A1 → A2 → A4 → A5 → A3 → A6 (volume tar and restart moved +ahead of the image save) to cut PLC downtime to ~10 s. The commit at A2 has +already frozen the layer, so the artifacts are identical either way. + +**Result — files on `dev-ubuntu:/home/dev-admin/`:** + +| File | Size | SHA-256 | +|---|---|---| +| `openplc-image.tar.gz` | 354 MB | `cbf2d2512d75acb5cce5640213d609383852f087bff6f4614397521ae7347d83` | +| `openplc-vol.tar.gz` | 746 B (mode 0600) | `a5a5011335a37f8f01fd0cae57bd8e135ec2e80fced30b723d8d08662d636628` | + +Verified: committed image contains `/workdir/build/libplc_1786668930820554523.so` +and `/workdir/core/generated/`; volume tarball contains `.env` + `restapi.db`; +source PLC restarted clean — `MODBUS_SLAVE Server listening on 0.0.0.0:502`, +69 located vars, reachable from Windows. + + + +**A1. Stop the container** — for a consistent `restapi.db` snapshot (SQLite). The PLC stops controlling; expected. + + docker stop openplc-runtime + +**A2. Commit the container to an image** — freezes the writable layer (your compiled program). + + docker commit openplc-runtime openplc-runtime-migrated:v4.1.10 + +**A3. Save the image to a file** — one portable file. Takes 1–3 min. + + cd ~ && docker save openplc-runtime-migrated:v4.1.10 | gzip > openplc-image.tar.gz + +**A4. Back up the volume.** `socket ignored` warnings are normal — runtime sockets, recreated on start. + + docker run --rm -v openplc-runtime-data:/data -v ~:/backup alpine \ + tar czf /backup/openplc-vol.tar.gz -C /data . + +**A5. Restart the source container** — leaves the source working. + + docker start openplc-runtime + +**A6. Record checksums + sizes.** Write the two hashes down; you verify them on the far side. + + cd ~ && ls -lh openplc-image.tar.gz openplc-vol.tar.gz && sha256sum openplc-*.tar.gz + +--- + +## PART B — Transfer (from your Windows PC, PowerShell) + +**B1. Pull down from source** + + scp dev-ubuntu:~/openplc-image.tar.gz dev-ubuntu:~/openplc-vol.tar.gz C:\Temp\ + +**B2. Push up to destination.** Use a dedicated subfolder — D3 bind-mounts this +directory into a container, and mounting the whole home directory would expose +`.ssh` and everything else to it. Not `/tmp`: cleared on reboot. + + ssh @ "mkdir -p ~/openplc-migration" + scp C:\Temp\openplc-image.tar.gz C:\Temp\openplc-vol.tar.gz @:~/openplc-migration/ + +Portainer's *Images → Import* upload also works for the image, but a 500 MB browser upload is failure-prone. Use scp. + +--- + +## PART C — Pre-flight on the DESTINATION (SSH) — protects the existing 20 containers + +Run all of these BEFORE changing anything. Every one must come back clean. + +**C1. Verify the files arrived intact** — must match A6. If not, re-transfer. + + cd ~/openplc-migration && sha256sum openplc-image.tar.gz openplc-vol.tar.gz + +**C2. Architecture + disk space** — `uname -m` must print `x86_64`; need ≥ 3 GB free. + + uname -m + df -h /var/lib/docker + +**C3. Name collisions — all three must return NOTHING** + + docker ps -a --format '{{.Names}}' | grep -x openplc-runtime + docker volume ls --format '{{.Name}}' | grep -x azureuser-openplc-runtime-data + docker images --format '{{.Repository}}' | grep -x openplc-runtime-migrated + +If any hits, stop and rename — see Appendix 1. + +**C4. Port 502 free — must return NOTHING** + + sudo ss -lntp | grep -w 502 + docker ps --format '{{.Names}} {{.Ports}}' | grep -w 502 + +If taken, pick another host port — see Appendix 1. + +--- + +## PART D — Load image + restore data (DESTINATION, SSH) + +**D1. Load the image.** Expect `Loaded image: openplc-runtime-migrated:v4.1.10`. The existing 25 images are untouched. + + docker load -i ~/openplc-migration/openplc-image.tar.gz + +**D2. Create the volume** + + docker volume create azureuser-openplc-runtime-data + +**D3. Restore the data into it** + + docker run --rm -v azureuser-openplc-runtime-data:/data -v ~/openplc-migration:/backup alpine \ + tar xzf /backup/openplc-vol.tar.gz -C /data + +**D4. Confirm the restore.** Expect `.env` and `restapi.db`. Sockets are absent — correct, they get recreated. + + docker run --rm -v azureuser-openplc-runtime-data:/data alpine ls -la /data + +--- + +## PART E — Deploy the service on the destination + +> [!IMPORTANT] +> **What was actually done (2026-08-19), and why it differs from E1–E6 below.** +> Rather than appending the service to the host's existing shared stack, it was +> deployed from a **separate compose file**, `~/openplc-compose.yml`, on its +> **own network** `openplc-net`. This is safer and is the recommended route: +> nothing else on the host is touched, there is no shared `volumes:`/`networks:` +> block to merge, rollback is `docker compose -f ~/openplc-compose.yml down`, and +> the PLC is isolated from the other ~20 containers. +> +> Two further differences from the draft below: ports are published on +> **`10.0.0.17:502` and `10.0.0.17:8443`** — the LAN address, not `0.0.0.0` — and +> `ufw` is **inactive** on this host (F6 does not apply; the Azure NSG governs). +> +> The live file is copied verbatim to `openplc-compose.yml`. +> E1–E6 below describe the shared-stack alternative and are retained for a +> destination host where a separate file is not an option. + +### Alternative — appending to an existing shared stack (NOT the route taken) + +The existing stack owns ~20 running containers. Updating it runs +`docker compose up -d` over the whole file. Compose only recreates services whose +config hash changed, so adding one service normally leaves the rest running — but +follow E1 and E5 exactly, they are what keeps that true. + +**E0. Confirm the stack is editable.** Portainer → **Stacks** → click the stack. +If it shows a **Git repository** section (with *Pull and redeploy*) instead of a +*Web editor*, STOP — the change must be committed to that git repo instead; a UI +edit gets reverted on the next sync. + +**E1. Back up the current stack file.** Open the stack → **Editor** tab → +select all → copy → paste into a local file (e.g. `C:\Temp\dest-stack-backup.yml`). +This is your rollback. Do not skip it. + +**E2. Note what is running now**, so you can prove nothing else restarted: + + docker ps --format '{{.Names}}\t{{.Status}}' | sort > ~/before.txt + +**E3. Append the service.** In the **Editor**, add this under the existing +top-level `services:` key, at the same indent as the sibling services: + +```yaml + openplc-runtime: + image: openplc-runtime-migrated:v4.1.10 + container_name: openplc-runtime + restart: unless-stopped + cap_add: + - SYS_NICE + - SYS_RESOURCE + ports: + - ":502:502" + - ":8443:8443" + volumes: + - azureuser-openplc-runtime-data:/var/run/runtime + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" +``` + +`cap_add` is required for the runtime's real-time scheduling. +`` — the destination host's LAN address (here `10.0.0.17`), so the +OpenPLC Editor and CI Server on **`yau-poc-cicore1` (10.0.0.21)** can reach it. +Binding the specific NIC rather than `0.0.0.0` keeps **both** 502 and 8443 off +every other interface. On a host with a public IP this is the only thing keeping +unauthenticated Modbus off the internet — never publish on `0.0.0.0`. + +**E4. Merge the volume declaration.** The file already has a top-level `volumes:` +block — add this entry INSIDE it. Do not add a second `volumes:` key; a duplicate +top-level key is a YAML error or silently discards one block. + +```yaml + azureuser-openplc-runtime-data: + external: true +``` + +`external: true` is load-bearing: without it Compose creates a NEW empty volume +named `azureuser_azureuser-openplc-runtime-data` and your restored data is ignored. + +**E5. Deploy.** Click **Update the stack**. In the confirmation dialog: + +- **Re-pull image — leave OFF.** Turning it on re-pulls every service in the + stack; anything on a moving tag gets recreated. +- **Prune services — leave OFF.** + +**E6. Confirm nothing else moved.** + + docker ps --format '{{.Names}}\t{{.Status}}' | sort > ~/after.txt + diff ~/before.txt ~/after.txt + +The only expected difference is the new `openplc-runtime` line. If other services +show fresh uptimes, they were recreated — see Rollback in Part G. + +### Note: shared stack network + +The service joins the existing stack's default network, so it can reach — and be +reached by — the other ~20 containers by service name. If you want it isolated, +give it its own network instead: + +```yaml + networks: + - openplc-net +``` + +plus a top-level `openplc-net:` entry merged into the existing `networks:` block. +Port publishing to the host works either way. + +--- + +## PART F — Verify + +**F1.** Portainer → **Containers** → `openplc-runtime` shows **running**, green. + +**F2. Logs** — Portainer container view → **Logs**, or by SSH: + + docker logs --tail 50 openplc-runtime + +Look for the runtime starting and the `modbus_slave` plugin loading. No repeating crash loop. + +**F3. Port is listening** (SSH on destination) + + sudo ss -lntp | grep -w 502 + +**F4. The PLC program came across** (SSH on destination) — a `libplc_*.so` must be present. + + docker exec openplc-runtime ls -la /workdir/build/ | grep libplc + +**F5. Modbus read from your Windows PC** — the real acceptance test. Must match the register map, same as against the old host. + + cd C:\Claude\wrps-demo-kit\05-tests + python verify_modbus.py --host --port 502 --unit 1 + +**F6. Firewall on the destination.** Check current state first. + + sudo ufw status + +On `yau-sls-poc-lin001` this returns **`Status: inactive`** — the host has no +firewall and inbound filtering is entirely the **Azure NSG**. The `ufw` commands +below were therefore **not applied** and are kept only for a destination host +that does run `ufw`. + +Modbus, for CI Server — only needed if F5 times out: + + sudo ufw allow from /24 to any port 502 proto tcp + +REST API, for the OpenPLC Editor. Restrict to the single Editor host, not the +subnet — 8443 is the control channel (upload, start/stop the program) and JWT auth +is the only thing in front of it: + + sudo ufw allow from 10.0.0.21 to any port 8443 proto tcp + +**F7. REST API reachable from the Editor host (run on 10.0.0.21).** + + curl -k https://:8443/api/login -X POST \ + -H 'Content-Type: application/json' \ + -d '{"username":"admin","password":""}' + +`-k` is required — the runtime serves a self-signed cert, so the Editor will show +a trust warning on first connect too. A JWT in the response means the path is open. + +--- + +## PART G — Afterwards + +- **CI Server:** ✅ done — `yau-poc-cicore1` (`10.0.0.21`) polls `10.0.0.17:502` + (`04-scada/` — the Modbus client config is the one place a literal IP is used). +- **Do not leave both PLCs answering on 502** with the same tags. ✅ done — the + source `dev-ubuntu` runtime was stopped and that host is retired. +- **Clean up transfer files on both hosts:** `rm -rf ~/openplc-migration`. + `openplc-vol.tar.gz` contains the JWT secret — do not leave it lying around, and + never commit either file to git. + +### Rollback + +Do NOT delete the stack — it owns the other ~20 containers. + +1. Portainer → **Stacks** → the stack → **Editor** → paste back the E1 backup + (`C:\Temp\dest-stack-backup.yml`), replacing the whole file. +2. **Update the stack**, again with *Re-pull image* and *Prune services* OFF. + This removes the `openplc-runtime` service and leaves the rest as they were. +3. Then on the destination host: + + docker rm -f openplc-runtime 2>/dev/null + docker volume rm azureuser-openplc-runtime-data + docker rmi openplc-runtime-migrated:v4.1.10 + +4. Firewall, if F6 was applied: + + sudo ufw delete allow from 10.0.0.21 to any port 8443 proto tcp + +The destination is back to its prior state. The source host was never modified +beyond a stop/start. + +--- + +## Appendix 1 — If a pre-flight check (C3/C4) found a collision + +- **Container name taken** → change `container_name:` to `wrps-openplc-runtime`. +- **Volume name taken** → use a new name in D2/D3 and in the compose file, e.g. + `wrps-openplc-data`. The container path `/var/run/runtime` must NOT change. +- **Image name taken** → retag after D1: + `docker tag openplc-runtime-migrated:v4.1.10 wrps/openplc-runtime:v4.1.10` +- **Port 502 taken** → publish elsewhere, e.g. `"5502:502"`, and point CI Server's + Modbus client at port 5502. The container-side `502` must not change. +- **Port 8443 taken** on the destination → publish elsewhere, e.g. + `":9443:8443"`, and point the OpenPLC Editor at port 9443. The + container-side `8443` must not change. Add this to the C4 pre-flight: + + sudo ss -lntp | grep -w 8443 + +## Appendix 2 — Login credentials + +The runtime's `admin` password and JWT secret travel inside `restapi.db` / `.env`, +so the same credentials work on the destination. Nothing to reconfigure. +See `02-environment/secrets.local.md`. diff --git a/02-environment/README.md b/02-environment/README.md new file mode 100644 index 0000000..d4df402 --- /dev/null +++ b/02-environment/README.md @@ -0,0 +1,144 @@ +# 02-environment — where this runs and how to reach it + +Two machines. One runs the PLC, the other runs the SCADA. Both sit on the +`10.0.0.0/24` PoC network. + +| Role | Hostname | LAN | Notes | +|---|---|---|---| +| **SCADA** — Yokogawa CI Server R1.03 | `yau-poc-cicore1` | `10.0.0.21` | Windows. Polls the PLC over Modbus TCP. | +| **PLC** — OpenPLC Runtime v4 in Docker | `yau-sls-poc-lin001` | `10.0.0.17` | Ubuntu 22.04 on Azure, public IP `20.211.144.151`. **Shared, live host** — see below. | + +Also on the network: an Active Directory domain controller at `10.0.0.5` +(`yau.poc`), and roughly a dozen other Windows hosts. + +## ⚠️ The Linux host is shared, live, and not ours + +`yau-sls-poc-lin001` is the YAU Innovation Team's general-purpose Docker host and +VPN gateway. It runs **~28 containers** for several unrelated projects — Grafana, +InfluxDB, Node-RED, Forgejo, Authelia, Portainer and more — behind a Caddy reverse +proxy with AD + Duo MFA. It serves customer-facing demos. + +**`openplc-runtime` is one container among many, and this project owns only that +one.** Rules that follow from it: + +- **Never restart Caddy or Authelia** — they interrupt every other service. +- **Never put growing data on the root disk** (`/` is 62 GB). Use `/datadisk`. +- **Never publish a container port on `0.0.0.0`.** The host has a public IP. +- Everyone shares the `azureuser` login, so there is no per-person audit trail — + **announce disruptive work.** + +`YAU_Linux_Host_Onboarding.md` is the host's own brief, written by its owner +(Daniel Watson) and current as of 2026-09-01. **Read it before changing anything +on that machine.** Its §10 is a list of rules, each of which exists because +breaking it already caused an outage. + +## Access + +### The Linux host — SSH + +```bash +ssh -i ~/.ssh/yau-sls-poc-lin001_key.pem azureuser@20.211.144.151 +``` + +An SSH config alias makes this shorter. The working copy used during the audit: + +``` +Host lin001 + HostName 20.211.144.151 + User azureuser + IdentityFile ~/.ssh/yau-sls-poc-lin001_key.pem + IdentitiesOnly yes +``` + +`10.0.0.17` is **only routable from inside the VNet or over the WireGuard VPN** — +from outside, use the public IP. + +### The CI Server host + +No remote access route is recorded. It was reached interactively during +development. If you need one, ask. + +### What you need from the host owner + +| Item | What it is | +|---|---| +| `yau-sls-poc-lin001_key.pem` | SSH private key — `chmod 600` it or SSH refuses to use it | +| AD account in `HTTPS_UserAccess` | required for every web UI on the host | +| Duo enrolment | second factor for those UIs | +| A WireGuard peer | optional, but it is what makes `10.0.0.x` reachable directly | + +## Ports + +| Service | Host | Port | Bound to | Notes | +|---|---|---|---|---| +| OpenPLC **Modbus TCP server** | `10.0.0.17` | 502 | **`10.0.0.17` only** | The PLC is the Modbus *slave*. CI Server polls it. | +| OpenPLC **REST API** (HTTPS) | `10.0.0.17` | 8443 | **`10.0.0.17` only** | Control channel — upload, start/stop. JWT auth. No browser UI. | +| CI Server Modbus **client** | `10.0.0.21` | — | outbound | | + +> [!IMPORTANT] +> **The bind address is the security control.** Modbus has no authentication or +> encryption, and 8443 is the control channel with only JWT in front of it. The +> host has a public IP, so binding these to `10.0.0.17` rather than `0.0.0.0` is +> the only thing keeping them off the internet. **Never widen it.** + +**There is no host firewall.** `ufw` is inactive on `yau-sls-poc-lin001`; inbound +filtering is entirely the **Azure Network Security Group**. Opening a port means +editing the NSG in the Azure portal, not the host. + +## Secrets + +Live in **`secrets.local.md`**, which is git-ignored and must never be committed. +`secrets.local.md.template` shows the shape — copy it, fill it in, keep it local. + +It holds the OpenPLC runtime `admin` password and CI Server credentials. The SSH +key is a file, not a line in that document, and is likewise never committed. + +Host-level credentials (service admin passwords, API tokens for Grafana, Influx, +Authelia and so on) are **not this project's** — they live in the host owner's +`Linux Machine Config.txt`. Ask; do not copy them here. + +## Files here + +| File | What | +|---|---| +| `YAU_Linux_Host_Onboarding.md` | The Linux host's own brief — the stack, the auth model, the deployment pattern, and the rules. **Written and owned by the host owner, not by this project.** Current 2026-09-01. | +| `openplc-container.md` | How the PLC container is configured, as read from the running host | +| `openplc-compose.yml` | Verbatim copy of the live `~/openplc-compose.yml` | +| `MIGRATION.md` | Moving the PLC container to another host — the as-built record of the 2026-08-19 move, kept as a repeatable runbook | +| `secrets.local.md.template` | The shape of the git-ignored secrets file | + +**Not kept here:** the host's full operations manual (`Host_Documentation.md`, +~400 lines covering ChirpStack, Forgejo, EQP licensing, the Telegraf fleet and the +rest of the platform). None of it is WRPS, it is owned elsewhere, and a stale copy +in this repo would be worse than a pointer. Ask the host owner for the current +version. + +## Checking the environment is healthy + +```bash +ssh lin001 "docker ps --filter name=openplc-runtime" +ssh lin001 "docker logs --tail 30 openplc-runtime" +ssh lin001 "df -h / /datadisk" # both disks - / is only 62 GB +``` + +From a machine that can route to `10.0.0.17` (VNet or VPN): + +```bash +python ../05-tests/verify_modbus.py --host 10.0.0.17 --port 502 --unit 1 +``` + +From anywhere else, poll from a throwaway container on the host's own network: + +```bash +ssh lin001 "docker run --rm --network openplc-net python:3.12-alpine \ + sh -c 'pip install -q pymodbus && python -c \"...\"'" +``` + +## Known issues + +| Issue | Detail | +|---|---| +| **PLC scan overruns** | The 100 ms task overran ~100 times between 2026-08-19 and 2026-08-28 — roughly one per 4–5 days. Logged as a warning; the task runs at a reduced rate and the demo continues. Not investigated; the host is shared, so contention is a candidate. | +| **Migration artefacts left on the host** | `~/openplc-migration/` still holds `openplc-image.tar.gz` (354 MB) and `openplc-vol.tar.gz`. `MIGRATION.md` Part G says to delete them, and **the volume tarball contains the runtime's JWT secret.** | +| **The runtime image cannot be re-pulled** | It was produced by `docker commit` and exists in no registry. Deleting it loses the compiled PLC program. See `03-plc/as-built/`. | +| **No deployment toolchain** | The OpenPLC Editor lived only on the retired `dev-ubuntu` host. See `03-plc/DEPLOY.md` §0. | diff --git a/02-environment/YAU_Linux_Host_Onboarding.md b/02-environment/YAU_Linux_Host_Onboarding.md new file mode 100644 index 0000000..0718f83 --- /dev/null +++ b/02-environment/YAU_Linux_Host_Onboarding.md @@ -0,0 +1,384 @@ + + +> [!NOTE] +> **This is a copy, kept here for reference. It is not maintained by this repo.** +> The document is written and owned by the Linux host's owner and describes the +> whole YAU PoC platform, of which this project uses one container. Ask the owner +> for the current version before relying on it. +> +> One instruction below does **not** apply here: §"Using this with Claude Code" +> suggests saving the file as `CLAUDE.md`. This repository has its own +> `CLAUDE.md`; do not overwrite it. Read this document as reference material. +> +> For this project's own view of the environment — hosts, ports, access, secrets +> — see `README.md` in this folder. + +--- + +# YAU PoC Linux Host — Environment Brief & AI Agent Guide + +> **Host:** `yau-sls-poc-lin001` · Azure Ubuntu 22.04 LTS · Public IP `20.211.144.151` · LAN `10.0.0.17` +> **Owner:** Daniel Watson (daniel.watson@yokogawa.com) · **Brief current as of:** 2026-09-01 +> **Audience:** an engineer joining this environment, and the AI coding agent working alongside her. + +> **Amended 2026-09-01** by the WRPS Plant Assistant project, against a `docker ps` on the +> host. Dan's brief of 2026-08-12 is unchanged apart from the container inventory, which had +> gone stale: §3 was counting 20 and naming 19, and omitted `openplc-runtime` and `wireguard` +> entirely. `openplc-runtime` is live control and appeared nowhere in this file. §3, §6 +> (published ports) and §12 are corrected; nothing else was touched. + +**This file is safe to share.** It contains no passwords, tokens, or keys — only their *locations*. +Everything you need to actually authenticate comes from Dan over a secure channel (see §2). + +**Using this with Claude Code:** save this file as `CLAUDE.md` in your project folder. Claude Code +loads it automatically at the start of every session, so your agent starts out knowing the host, +the stack, the deployment pattern, and the rules in §10 — which exist because breaking them has +already caused one outage here. + +--- + +## 1. What this box is + +A **secure, general-purpose Docker host and network gateway** for the YAU Innovation Team. Two roles: + +1. **A multi-service platform.** Many containerised services for different sales/PoC engagements, + added and removed as needed. The container list in §3 is a snapshot, not a fixed design. + Publishing a new service under HTTPS with SSO is a ~5-minute, well-worn pattern (§7). +2. **A secure gateway** into the `10.0.0.0/24` PoC environment, which also holds Windows hosts + (a Domain Controller at `10.0.0.5`, a CI Server, ~13 machines total). Devices and remote users + come in over WireGuard rather than being exposed to the internet. + +**Design principle:** the only internet-facing surface is the Caddy reverse proxy (80/443) and the +WireGuard VPN (UDP 443). Databases, MQTT, and other hosts are reached *through* the box, never +directly. Keep it that way. + +--- + +## 2. Access — what you need from Dan + +Ask for these over a secure channel (not email/chat in plaintext): + +| Item | What it is | +|------|-----------| +| `yau-sls-poc-lin001_key.pem` | SSH private key. Save it locally and `chmod 600` it, or SSH refuses to use it | +| AD account + `HTTPS_UserAccess` group | Your `yau.poc` domain login, added to this group — required for **every** web UI | +| Duo enrolment | Second factor (push notification) for all web UIs | +| `Linux Machine Config.txt` | The credentials file — service admin passwords and API tokens | + +```bash +ssh -i yau-sls-poc-lin001_key.pem azureuser@20.211.144.151 +``` + +You log in as **`azureuser`** — it has `sudo` and is in the `docker` group. There are no per-person +Linux accounts; everyone shares `azureuser`, so **announce disruptive work** before you do it. + +Optional but recommended: a WireGuard VPN peer, so you can reach LAN hosts and internal ports +directly. Ask Dan to add one (§6). + +--- + +## 3. The stack at a glance + +**21 containers**, all `restart: unless-stopped`, all with log rotation (10 MB × 3). +`docker ps` shows **28** today: these 21, plus seven added by the WRPS Plant Assistant +project (`ai-api`, `ai-web`, `pg-ai`, `cube`, `cubestore`, `langfuse`, `lf-db`) — those +are described in that project's own repository, not here. Counted 2026-09-01. + +| Service | URL | Auth | Notes | +|---------|-----|------|-------| +| **Caddy** | — (the front door) | — | Reverse proxy, automatic Let's Encrypt certs for `*.yokogawa.tech` | +| **Authelia** | `auth.yokogawa.tech` | — (is the portal) | AD first factor + Duo push second factor; gates everything below | +| **Grafana** | `grafana.yokogawa.tech` | MFA + AD SSO | Dashboards. Auto-logs in as your AD user; new users get org **Admin** | +| **InfluxDB 2.7** | `influx.yokogawa.tech` | MFA (UI); API bypassed | Historisation — the primary data store. **52 GB and growing** | +| **Node-RED** | `nodered.yokogawa.tech` | MFA + own `yauadmin` login | Flow-based processing | +| **Mosquitto** | — (host port 1883) | ⚠️ anonymous | General MQTT broker | +| **Forgejo** | `git.yokogawa.tech` | AD (its own, **not** Authelia) | Internal Git, branded "Yokogawa Git". Not behind Authelia because that breaks git clients | +| **Portainer** | `portainer.yokogawa.tech` | MFA + own admin login | Graphical Docker management — **root-equivalent** | +| **Dozzle** | `logs.yokogawa.tech` | MFA | Live searchable container logs — your best first debugging stop | +| **Showroom** | `showroom.yokogawa.tech` | **1FA only** (no Duo) | Static demo site, gated to AD group `Showroom_Access` | +| **EQP Licence** | `licence.yokogawa.tech` | MFA | Licence issuer; signing key mounted read-only, never baked into the image | +| **Telegraf** | — | — | Host + container metrics → Influx `telemetry` bucket (30-day retention) | +| **Watchtower** | — | — | Auto-updates a **safe subset only**, Sundays 04:00 AEST | +| **ChirpStack** ⚠️ | `chirpstack.yokogawa.tech` | MFA | LoRaWAN (AU915) — **future capability, running but NOT configured**. Safe to ignore or stop | +| **WireGuard** | — (host port 443/udp) | peer keys | The VPN into `10.0.0.0/24`. Adding peers: §6 | +| **OpenPLC Runtime** ⚠️⚠️ | — (host ports **502/tcp, 8443/tcp**, bound to `10.0.0.17`) | ⚠️ **none** | **LIVE CONTROL.** The soft PLC for the Waterloo Road Pump Station demo, polled over Modbus TCP by CI Server on `cicore1`. **Never restart, update or reconfigure it as a side effect of other work**, and never change the `10.0.0.17` binding to `0.0.0.0` — that binding is the only thing keeping unauthenticated Modbus off the internet. Not in Watchtower's update list, and must not be added | + +Plus `chirpstack-postgres`/`-redis`/`-mqtt`/`-gateway-bridge` (all ChirpStack support) and +`authelia-portal` (nginx that brands the login page). + +### Data flow + +``` +Field devices / Windows hosts (10.0.0.0/24) ──VPN/LAN──┐ + ▼ + Telegraf agents ─┐ ┌──── Linux Docker host ────┐ + MQTT (Mosquitto) ┼──► Node-RED ──────────►│ InfluxDB (historisation) │──► Grafana + CI Server ───────┘ │ on /datadisk │ (dashboards) + └───────────────────────────┘ + Everything web-facing is published through Caddy (HTTPS) and gated by Authelia (AD + Duo). +``` + +--- + +## 4. Where things live + +All configuration is in **`/home/azureuser`** — flat, one Compose file per service group: + +``` +~/docker-compose.yml caddy, grafana, influxdb, nodered, mosquitto ← core stack +~/chirpstack-compose.yml chirpstack + postgres/redis/mqtt/gateway-bridge +~/wg-compose.yml wireguard +~/authelia-compose.yml authelia ~/authelia-portal-compose.yml branding proxy +~/forgejo-compose.yml forgejo ~/eqp-compose.yml licence issuer +~/portainer-compose.yml portainer ~/dozzle-compose.yml log viewer +~/telegraf-compose.yml telegraf ~/watchtower-compose.yml auto-updater +~/showroom-compose.yml showroom + +~/Caddyfile all reverse-proxy routes (+ many .bak-* snapshots) +~/authelia/configuration.yml auth rules — root-owned, edit with sudo +~/authelia/authelia.env secrets, 0600 +~/telegraf/ telegraf.conf + Influx tokens (0600) +~/mosquitto/config/ broker config + passwordfile +~/showroom-site/ static demo content (rsync target) +``` + +Every `*-compose.yml` shares the default Compose project name `azureuser`. Running `docker compose` +against a single file therefore prints a **harmless "orphan containers" warning** — ignore it. + +**Two disks, and the split matters:** + +| Mount | Size | Used | Contents | +|-------|------|------|----------| +| `/` | 62 GB | 21% | OS, Docker images, most volumes | +| `/datadisk` | 128 GB | 43% | **InfluxDB (52 GB)**, Forgejo | + +--- + +## 5. Authentication model + +Understand this before you deploy anything. + +- **Authelia** sits in front of nearly everything via Caddy's `forward_auth`. First factor is + **Active Directory** (`ldaps://10.0.0.5:636`, base `DC=yau,DC=poc`), restricted to the AD group + **`HTTPS_UserAccess`**. Second factor is **Duo Push** — TOTP and WebAuthn are deliberately + disabled so no email enrolment is needed. +- **SSO:** one login covers all `*.yokogawa.tech`. Sessions are held **in memory**, so restarting + Authelia logs everyone out. That is also the supported way to refresh someone's group membership. +- **Grafana** does true AD SSO — it trusts Authelia's `Remote-User` header via auth-proxy, + whitelisted to the `172.18.0.0/16` Docker subnet. No second login. +- **Node-RED** and **InfluxDB OSS** keep their own logins behind the MFA gate — neither supports + AD or proxy-header auth. Not a misconfiguration. +- **API bypass:** Influx `^/api/v2/(write|query)` and `/health` skip MFA so devices and Telegraf + agents can write. If you add machine-to-machine endpoints, they need a similar explicit bypass. + +> ### ⚠️ The AD gotcha that will bite you +> Authelia and Forgejo both resolve **DIRECT** group membership only. A user who is in +> `HTTPS_UserAccess` *via a nested group* (e.g. through `Domain Admins`) will **not** get access. +> Add people as direct members. The service account `svc-authelia` is **read-only** and cannot +> change membership — that must be done on the DC with `Add-ADGroupMember`. +> +> And if someone is added to a group *after* they logged in, they'll still be denied until the +> session refreshes: `docker compose -f ~/authelia-compose.yml restart authelia`. + +--- + +## 6. Networking + +- **Caddy** terminates HTTPS and proxies over the `proxy` Docker network (external; every + internet-facing service joins it). DNS: all `*.yokogawa.tech` A records → `20.211.144.151`. +- **WireGuard:** endpoint `yau.poc.vpn.yokogawa.tech`, **UDP 443** (chosen to traverse restrictive + corporate firewalls). Tunnel subnet is **`10.13.13.0/24`**, deliberately separate from the LAN. + Peers reach `10.0.0.0/24` via `ip_forward` + container MASQUERADE, so LAN hosts see traffic as + coming from `10.0.0.17`. Existing peers: `dan`, `laptop`, `mac`, `office`, `rut1`. + Add one by appending to `PEERS` in `~/wg-compose.yml`, then + `docker compose -f ~/wg-compose.yml up -d --force-recreate` (existing peers are preserved), then + `docker exec wireguard /app/show-peer ` for the config/QR code. +- **Published host ports:** 80/443 tcp (Caddy), 443/udp (WireGuard), 1883/tcp (MQTT), + 1700/udp (LoRaWAN packet forwarder), and **502/tcp + 8443/tcp (OpenPLC Runtime)** — + the last two bound to `10.0.0.17`, not `0.0.0.0`. That binding is a security control: + Modbus has no authentication or encryption, so the bind plus the NSG is all that keeps + it off the internet. Do not change it. +- **Firewall:** host `ufw` is **inactive** — inbound filtering is entirely the **Azure NSG**. + Opening a port means editing the NSG in the Azure portal, not the host. +- **Azure hairpin gotcha:** LAN hosts cannot reach the VM's *public* IP from inside the VNet. + Solved with a pinpoint DNS record on the DC: `influx.yokogawa.tech → 10.0.0.17`. If you add a + service that LAN machines must reach by hostname, it needs the same treatment. + +--- + +## 7. ⭐ How to deploy a new service (the pattern you'll use most) + +This is the well-worn path. Follow it and your service gets HTTPS, a cert, and AD+Duo SSO for free. + +**1. Write `~/-compose.yml`.** Join the `proxy` network. Do **not** publish host ports — +reach it through Caddy. If it stores growing data, bind-mount under `/datadisk`, not the root disk. + +```yaml +services: + myservice: + image: myimage:tag + container_name: myservice + restart: unless-stopped + networks: [proxy] + volumes: + - /datadisk/myservice:/data # only if it stores real data +networks: + proxy: + external: true +``` + +**2. Add a block to `~/Caddyfile`:** + +``` +myservice.yokogawa.tech { + import authelia + reverse_proxy myservice:8080 +} +``` + +`import authelia` is the shared MFA gate — omit it only with a deliberate reason (Forgejo omits it +because forward-auth breaks git clients). + +**3. Add the domain to the Authelia rule** in `~/authelia/configuration.yml` under the +`HTTPS_UserAccess` `two_factor` rule. The file is **root-owned — edit with `sudo`**. +There's a helper, `~/apply_rule.py`, for rewriting the trailing rule. Back the file up first; +you'll find plenty of `.bak-*` precedents. + +**4. Apply and verify:** + +```bash +docker compose -f ~/-compose.yml up -d +docker exec caddy caddy reload --config /etc/caddy/Caddyfile +docker compose -f ~/authelia-compose.yml restart authelia # note: logs everyone out +curl -sI https://myservice.yokogawa.tech # expect 302 → auth portal +``` + +**5. Add the DNS A record** `myservice.yokogawa.tech → 20.211.144.151`. Without it Caddy cannot +get a certificate. Ask Dan — DNS is not managed on this host. + +--- + +## 8. Operating it + +```bash +# Status +docker ps +docker stats --no-stream +df -h / /datadisk # watch both + +# Logs — or just use https://logs.yokogawa.tech (Dozzle), which is nicer +docker logs -f grafana +docker logs --tail 100 influxdb + +# Apply changes / restart +docker compose -f ~/docker-compose.yml up -d +docker compose -f ~/docker-compose.yml restart grafana + +# Bring everything up (also happens automatically on reboot) +cd ~ && docker compose -f docker-compose.yml -f chirpstack-compose.yml -f wg-compose.yml up -d + +# Reload Caddy after editing the Caddyfile +docker exec caddy caddy reload --config /etc/caddy/Caddyfile +``` + +**Updates:** Watchtower auto-updates **only** `grafana nodered portainer authelia wireguard` +(Sundays 04:00 AEST). Pinned images — `influxdb:2.7`, `caddy:2`, `postgres:14`, `redis:7-alpine`, +chirpstack, mosquitto — are never touched automatically. Update those by hand: +`docker compose -f pull && docker compose -f up -d `. + +**Monitoring:** the Grafana dashboard **"YAU Host & Containers — Health"** (`/d/yau-host-health`) +shows both disks, and an alert fires above 80%. Note the alert is **in-UI only** — no email or +Teams contact point is wired up, so nobody gets pushed a notification. Worth fixing. + +--- + +## 9. Known issues — inherited, not yours + +| Issue | Detail | +|-------|--------| +| **Mosquitto is open** | `allow_anonymous true`, no TLS, on internet-exposed port 1883. A `passwordfile` exists but isn't enforced. Restrict via NSG/VPN or enable auth before putting anything real on it | +| **Secrets in plaintext** | Admin passwords and tokens sit in `Linux Machine Config.txt` and in Compose env vars (the Grafana admin password is literally in `~/docker-compose.yml`) | +| **Docker socket exposure** | Portainer and Watchtower mount it **read-write** = root-equivalent host control; Dozzle and Telegraf mount it read-only. All are behind MFA — keep it that way | +| **No host firewall** | Entirely dependent on correct Azure NSG rules | +| **No automated backup** | Backups are manual. The most recent is `Backups/vm-config-20260812/`. **Influx's 52 GB is not in it** — that needs an Azure disk snapshot | +| **Alerts don't notify** | Disk alert shows in Grafana's UI only | +| **ChirpStack log spam** | Unconfigured gateway-bridge loops and once generated 5.9 GB of logs. Rotation caps it now; stopping the stack is the real fix if LoRaWAN isn't needed | +| **Single shared login** | Everyone is `azureuser`; no per-person audit trail on the host | + +--- + +## 10. Rules for the AI agent + +**Read these before proposing changes to this host.** Each one comes from something that already +went wrong or is a live constraint. + +1. **Never put growing data on the root disk.** `/` is only 62 GB. InfluxDB data **must** stay + bind-mounted at `/datadisk/influx`. A previous migration copied 47 GB to `/datadisk` but never + repointed the container or deleted the original — root hit 100%, and Grafana died with + `database or disk is full`. New services with real data go on `/datadisk`. +2. **Never move the WireGuard tunnel back onto `10.0.0.0/24`.** It used to overlap the server LAN, + colliding with Azure-reserved `.1`–`.3` and the real host at `.5`. It lives on `10.13.13.0/24`. +3. **Don't remove or bypass Authelia** to "simplify" access. It *is* the AD login — a plain static + site cannot AD-authenticate without it. Removing it silently makes services public. +4. **`~/authelia/configuration.yml` is root-owned.** Edit with `sudo`, back it up first, and know + that restarting Authelia logs out every active user. +5. **AD group membership must be DIRECT** (§5). Nested membership silently fails to grant access. +6. **Don't publish host ports** for new services. Go through Caddy on the `proxy` network. Every + published port is a new NSG dependency and a new attack surface. +7. **Watchtower's update list is deliberately short.** Don't add pinned database or proxy images + to it — unattended major-version bumps of Influx/Postgres/Caddy are how you lose a weekend. +8. **Verify before declaring success.** `docker ps` showing "Up" is not proof; `curl -sI` the + public URL and expect a 302 to the auth portal, and check `docker logs` for the container. +9. **This host is shared and live** — it runs customer-facing demos. Announce restarts of Caddy or + Authelia (they interrupt everyone). Prefer additive changes; snapshot config before editing + (the `.bak--` convention is already established throughout `~`). +10. **Don't commit secrets.** `Linux Machine Config.txt`, `*.pem`, `*.token`, and `authelia.env` + never go into Git, and never into a file intended for sharing. + +--- + +## 11. Deeper reference + +These live in the same project folder as this brief (ask Dan — several contain secrets): + +| File | Covers | +|------|--------| +| `Host_Documentation.md` | The full ops manual — every service, network detail, and a dated change log explaining *why* things are the way they are. **Read this second.** | +| `MFA_Duo_Setup_Plan.md` | Authelia + Duo design and config templates | +| `WireGuard_RUT_Setup.md` | Onboarding RUT240/RUT950 field routers onto the VPN | +| `Windows_Telegraf_GPO_Deployment.md`, `Install-Telegraf-Windows.ps1`, `Deploy-Telegraf.ps1` | Rolling Telegraf agents out to the Windows fleet | +| `Showroom/deploy/` | Showroom demo site deploy kit and runbook | +| `Backups/vm-config-20260812/` | Full config backup + restore notes ⚠️ **contains secrets — do not share** | +| `Linux Machine Config.txt` | The credentials file ⚠️ **secrets** | + +--- + +## 12. Suggested first day + +1. Get the SSH key, AD account in `HTTPS_UserAccess`, and Duo enrolment from Dan (§2). +2. SSH in; run `docker ps` and `df -h`. Confirm every container is up and both disks are + healthy. Don't check against a fixed number — services come and go (§3). +3. Log into `grafana.yokogawa.tech` — this exercises the whole AD + Duo + SSO path in one go. +4. Open `logs.yokogawa.tech` (Dozzle) and `portainer.yokogawa.tech` to get a feel for the stack. +5. Read `Host_Documentation.md`, especially the change log — it explains the scars. +6. Deploy something trivial (a `nginx:alpine` hello-world) end-to-end using §7. Doing the full + compose → Caddy → Authelia → DNS loop once on a throwaway service is the fastest way to learn + this environment, and it's safe. Tear it down afterwards. + +--- + +### A note on adding AI features here + +Nothing on this host currently calls an LLM, so you'll be first. Things worth knowing up front: + +- **Outbound internet works** (Let's Encrypt and image pulls depend on it), so a container calling + the Anthropic API will work — but **API keys must not** go into a Compose file in plaintext the + way the Grafana password did. Use an env file at `0600` (the `~/telegraf/telegraf.env` and + `~/authelia/authelia.env` pattern) and keep it out of Git. +- **Data is right here:** InfluxDB holds the historised time-series, Mosquitto carries live MQTT, + and Node-RED is already wired to both — it's the path of least resistance for a prototype. +- **Follow §7 for anything with a UI** so it lands behind AD + Duo like everything else. An + unauthenticated AI endpoint on a box this exposed is not acceptable. +- If a service needs a GPU or sustained heavy compute, this VM is not it — raise sizing with Dan + before designing around it. diff --git a/02-environment/openplc-compose.yml b/02-environment/openplc-compose.yml new file mode 100644 index 0000000..b0ef0e7 --- /dev/null +++ b/02-environment/openplc-compose.yml @@ -0,0 +1,28 @@ +services: + openplc-runtime: + image: openplc-runtime-migrated:v4.1.10 + container_name: openplc-runtime + restart: unless-stopped + cap_add: + - SYS_NICE + - SYS_RESOURCE + ports: + - "10.0.0.17:502:502" + - "10.0.0.17:8443:8443" + volumes: + - azureuser-openplc-runtime-data:/var/run/runtime + networks: + - openplc + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" + +networks: + openplc: + name: openplc-net + +volumes: + azureuser-openplc-runtime-data: + external: true diff --git a/02-environment/openplc-container.md b/02-environment/openplc-container.md new file mode 100644 index 0000000..419ee0e --- /dev/null +++ b/02-environment/openplc-container.md @@ -0,0 +1,71 @@ +# The OpenPLC Runtime container + +The PLC for this demo runs as a **Docker Compose service** on the shared host +`yau-sls-poc-lin001` (`10.0.0.17`). It is deployed from its **own** compose file, +`~/openplc-compose.yml` on that host, not from the host's main stack. + +`openplc-compose.yml` in this folder is a **verbatim copy of the live file**, +read from the host on 2026-09-02. The host is canonical for what is running; this +copy exists so the deployment is reviewable and re-creatable from the repo. + +> [!WARNING] +> **This container is live control and the demo is running.** CI Server on +> `yau-poc-cicore1` (`10.0.0.21`) polls it over Modbus TCP. Do not restart, +> recreate or update it as a side effect of other work. + +## As-built — verified on the host 2026-09-02 + +| Item | Value | +|---|---| +| Container | `openplc-runtime` | +| Image | `openplc-runtime-migrated:v4.1.10` (`sha256:1e3bd0e1…a502cbc6`) | +| Compose file | `/home/azureuser/openplc-compose.yml` · project `azureuser` | +| Published ports | `10.0.0.17:502->502/tcp`, `10.0.0.17:8443->8443/tcp` | +| Network | `openplc-net` — **isolated**, not the shared stack network | +| Volume | `azureuser-openplc-runtime-data` (external) | +| Capabilities | `SYS_NICE`, `SYS_RESOURCE` | +| Restart policy | `unless-stopped` · logging `json-file` 10 MB × 3 | +| Loaded program | `libplc_1786668930820554523.so`, built 2026-08-14 | + +The image is **local to the host** — it was produced by `docker commit` during the +migration (see `MIGRATION.md`), not pulled from a registry. It cannot be re-pulled; +if it is deleted, the compiled program is lost and must be rebuilt and re-uploaded +through the OpenPLC Editor (`03-plc/DEPLOY.md`). + +## Why each setting matters + +| Setting | Reason | +|---|---| +| `cap_add: SYS_NICE, SYS_RESOURCE` | Real-time scheduling. **Omitting these breaks the runtime** — always include them in a recreate. | +| `azureuser-openplc-runtime-data:/var/run/runtime` | All runtime state: `restapi.db` (users + program record) and `.env` (JWT secret). Survives recreate; **do not delete the volume**. Declared `external: true` so Compose uses the restored volume instead of creating an empty one. | +| `10.0.0.17:502:502` | Modbus TCP for CI Server. **Bound to the LAN address, not `0.0.0.0`.** Modbus has no authentication or encryption, and this host has a public IP — the bind address is the security control. Do not widen it. | +| `10.0.0.17:8443:8443` | REST API — the control channel (upload, start/stop). Reachable from the LAN so the OpenPLC Editor can drive it; JWT auth is the only thing in front of it, so the same bind rule applies. | +| `networks: openplc-net` | Deliberate isolation from the host's other ~20 containers. The PLC has no reason to reach them, or they it. | +| image pinned by tag+digest | Stops a recreate from silently changing the runtime under a tested program. | + +## Firewall + +**UFW on this host is inactive** — inbound filtering is entirely the **Azure +Network Security Group**. There is no host firewall rule for 502 or 8443, and +adding one is not the mechanism here: 502/8443 are bound to `10.0.0.17`, which is +not reachable from outside the VNet, and the NSG governs the public interface. + +## If port 502 refuses connections + +**Two conditions must both hold**, and neither is a container setting: + +1. a program is **running** on the runtime, and +2. the Editor project defines a Modbus **Server** so the runtime's `modbus_slave` + plugin is enabled — see `03-plc/DEPLOY.md` §A3. + +Check those before touching the container. A refusal looks like a firewall drop +but is not one: Docker DNATs the packet to the container, which returns RST +because nothing is bound inside. + +## Known issue — PLC scan overruns + +The runtime logs `[task PLC_TASK] scan overrun` warnings: the 100 ms cycle body +occasionally exceeds its period and the task runs at a reduced rate. **100 +overruns between 2026-08-19 and 2026-08-28** (roughly one per 4–5 days). Other +tasks are unaffected and the demo continues to run. Not yet investigated — this +host is shared, so contention with the other containers is a candidate cause. diff --git a/02-environment/secrets.local.md.template b/02-environment/secrets.local.md.template new file mode 100644 index 0000000..0d358c3 --- /dev/null +++ b/02-environment/secrets.local.md.template @@ -0,0 +1,50 @@ +# Secrets — NOT COMMITTED + +Copy this file to `secrets.local.md` and fill it in. That name is git-ignored +(`*.local.md` in `.gitignore`); this template is not, so **never put a real +value in this file**. + +Everything here is referenced from the repo by *location*, never inlined. + +--- + +## OpenPLC Runtime v4 REST API + +Host `yau-sls-poc-lin001`, `https://10.0.0.17:8443` — self-signed cert, so +clients need `-k` / accept-on-first-connect. + +| | | +|---|---| +| Username | `admin` | +| Password | `` | + +Used by: the OpenPLC Editor when connecting to the runtime (`03-plc/DEPLOY.md` +§B3), and by any direct `POST /api/login` for a JWT. + +> The password and the JWT secret travel inside the container volume +> (`restapi.db` and `.env`). A migrated container keeps the same credentials — +> see `MIGRATION.md` Appendix 2. + +## Yokogawa CI Server + +Host `yau-poc-cicore1`, `10.0.0.21`. + +| | | +|---|---| +| Username | `` | +| Password | `` | + +## SSH to the Linux host + +The key is a **file**, not a value to paste here: + + ~/.ssh/yau-sls-poc-lin001_key.pem chmod 600, or SSH refuses it + +Obtain it from the host owner over a secure channel. Never commit it — the +`.gitignore` blocks `*.pem`, but do not rely on that alone. + +## Not this project's secrets + +Host-level credentials — Grafana, InfluxDB, Authelia, Portainer, Forgejo admin +logins and API tokens — belong to the host owner and live in their +`Linux Machine Config.txt`. Ask for them; do not copy them into this repo.