diff --git a/BUILD-AI-CONTAINERS.md b/BUILD-AI-CONTAINERS.md index b7213b0..89d72fc 100644 --- a/BUILD-AI-CONTAINERS.md +++ b/BUILD-AI-CONTAINERS.md @@ -693,8 +693,8 @@ it is the same derivation that will run against `imh`. **Gate** - [ ] `curl -sI https://ai.yokogawa.tech` **from lin001** → 403. That is the deny arm: this host, the VPN and the internet are all shut out. A 302 means the SCADA-only block was reverted; a 200 means the matcher is restricting nothing. - [ ] `curl -s -o /dev/null -w '%{http_code}' -X POST https://ai.yokogawa.tech/ask` from lin001 → 403, **not 404**. A 404 is ai-web answering, which means the `/ask` route is missing and every question will fail once you are on cicore1. -- [ ] **On `cicore1`**: `https://ai.yokogawa.tech` loads the UI with no login. Nothing on lin001 can prove this — a typo in the matcher 403s cicore1 too and looks identical from the host. -- [ ] An operator on `cicore1` reaches the UI **by hostname** and gets an answer end to end — this is the check that catches a name resolvable from outside and not from inside. Doing it from an engineer's laptop proves nothing about it. +- [x] **On `cicore1`**: `https://ai.yokogawa.tech` loads the UI with no login. ~~Nothing on lin001 can prove this~~ — a curl from lin001 cannot, but `ai-web`'s nginx log records the real client address Caddy forwards as its last field, so the host can prove it after all. **Confirmed 2026-08-28 06:49 UTC**, the same day the rule was applied: `10.0.0.21` loading `/`, the JS bundle and the CSS, all `200`. `verify.sh` now reads that log rather than asking someone to go and look. +- [x] An operator on `cicore1` reaches the UI **by hostname** and gets an answer end to end. **Confirmed 2026-08-31**: page load from `10.0.0.21` at 07:25:08 UTC, then `POST /ask` → `200` at 07:25:29, 07:38:11 and 07:38:34 UTC. The requests reach `ai-api` from `172.18.0.6` (Caddy on the proxy network), which is why the API log alone cannot attribute them — the origin has to be read from `ai-web`. The `Referer` is `https://ai.yokogawa.tech/`, so the page was reached by hostname and `/ask` was same-origin: the pinpoint record and the single-origin routing both work. - [ ] Citations show document number, revision and effective date - [ ] The scope banner appears on every Procedural and Advisory answer @@ -799,7 +799,7 @@ it is the same derivation that will run against `imh`. - Secrets in `0600` env files, not a vault - No OT/IT firewall boundary — one flat `10.0.0.0/24` PoC network -- **`ai.yokogawa.tech` is unauthenticated from `cicore1`.** Applied 2026-08-28 at the customer's direction: an on-site operator should not complete a Duo push to ask a question, and nobody outside the plant should reach the assistant at all. The Caddy block admits `remote_ip 10.0.0.21` only and returns 403 to everything else, `import authelia` removed from that branch. This is an IP allowlist on a flat network with no OT/IT boundary — anything that can take `10.0.0.21`, or ARP-spoof it, inherits unauthenticated access to every answer the assistant can give. It is a demo affordance, **not a security control**, and it is the first thing network segmentation closes. Two consequences worth stating separately: Langfuse traces are now anonymous, so there is no record of who asked what; and the assistant is unreachable by browser from the VPN, so engineers need an SSH tunnel. `api.yokogawa.tech` is unchanged and still fully gated — Phase 9 document publishing depends on that and must stay there +- **`ai.yokogawa.tech` is unauthenticated from `cicore1`.** Applied 2026-08-28 at the customer's direction: an on-site operator should not complete a Duo push to ask a question, and nobody outside the plant should reach the assistant at all. The Caddy block admits `remote_ip 10.0.0.21` only and returns 403 to everything else, `import authelia` removed from that branch. This is an IP allowlist on a flat network with no OT/IT boundary — anything that can take `10.0.0.21`, or ARP-spoof it, inherits unauthenticated access to every answer the assistant can give. It is a demo affordance, **not a security control**, and it is the first thing network segmentation closes. Two consequences worth stating separately: Langfuse traces are now anonymous, so there is no record of who asked what; and the assistant is unreachable by browser from the VPN, so engineers need an SSH tunnel. **A third, found 2026-08-31: there is no usage record at the edge either.** The Caddy block has no `log` directive — no site on this host does — so the front door logs nothing, and the only evidence that the console has ever been used is `ai-web`'s container log, which `docker compose up -d --force-recreate` wipes. That is how a working operator path sat unnoticed for three days while the repo recorded it as unproven. **Access logging was considered and declined on 2026-09-01.** The ephemeral evidence is accepted: usage is confirmed as at 31 August and recorded in Phase 7's gate, and if the container is recreated that confirmation stands on the record here rather than in a log. Do not re-raise this as a task. `api.yokogawa.tech` is unchanged and still fully gated — Phase 9 document publishing depends on that and must stay there - Public egress to Azure OpenAI, no private endpoint - Chromium running on the SCADA VM itself - Modbus TCP on port 502 with no authentication or encryption — inherent to the protocol; contained by its bind to `10.0.0.17` plus NSG/VPN scope. The OpenPLC Runtime web UI on 8443 is contained the same way and nothing else diff --git a/REQUESTS.md b/REQUESTS.md index 5e422d3..8020b20 100644 --- a/REQUESTS.md +++ b/REQUESTS.md @@ -121,10 +121,12 @@ Dan for this item. Two things are worth recording: - **The names resolving is not the same as the services answering.** The Caddyfile blocks and the Authelia rules for `ai`, `api` and `cube` were applied on the host the same day; - all four AI hostnames now return 302 to the auth portal. What is still unproven is the - part no check here can reach: an operator on `cicore1` — since 28 August, with no sign-in - at all — getting an - answer end to end. + all four AI hostnames now return 302 to the auth portal. The part no check here could + reach — an operator on `cicore1`, since 28 August with no sign-in at all, getting an + answer end to end — **is now proven**. `ai-web`'s access log carries the real client + address Caddy forwards: `10.0.0.21` loaded the page on 28 August, the day the rule was + applied, and asked three questions on 31 August, each answered `200`. The operator path is + therefore used and confirmed, not merely built. The original request follows, unchanged. diff --git a/caddy/ai-routes.caddy b/caddy/ai-routes.caddy index b89d1a8..84ebb7c 100644 --- a/caddy/ai-routes.caddy +++ b/caddy/ai-routes.caddy @@ -150,6 +150,17 @@ api.yokogawa.tech { # 10.0.0.21 (yau-poc-cicore1, static) -> straight through, no auth # everything else -> 403 # +# BOTH ARMS PROVEN - the console has been used from cicore1, end to end. The +# deny arm from lin001 (403). The allow arm from ai-web's own access log, which +# carries the real client address Caddy forwards as the last field: 10.0.0.21 +# loaded the page on 2026-08-28, the day this went in, and asked three questions +# on 2026-08-31, each answered 200. +# +# This block has NO `log` directive and is not getting one - access logging was +# declined 2026-09-01. The evidence above is ephemeral (a recreate of ai-web +# wipes it); that is accepted, and the confirmation lives in the Phase 7 gate in +# BUILD-AI-CONTAINERS.md instead. Do not add logging here without asking. +# # Why an IP matcher works here, verified on the host before applying: # - Caddy sees REAL client addresses. Docker's DNAT preserves the source; the # access log carries public IPs and 10.0.0.5 with no 172.18.x rewriting. If diff --git a/current-state.html b/current-state.html index fb9bad3..08104f5 100644 --- a/current-state.html +++ b/current-state.html @@ -318,7 +318,7 @@ footer { margin-top: 72px; padding-top: 22px; border-top: 1px solid var(--hair);
28containers on lin001
7 of them ours
-
6phases done
of nine
+
7phases done
of nine
1request still outstanding
the historian login
0open findings
all three closed 31 Aug
78exam questions
never yet run as a gate
@@ -581,16 +581,19 @@ footer { margin-top: 72px; padding-top: 22px; border-top: 1px solid var(--hair);
-
+
07

The operator’s screen

- Built — last gate unproven -
The screen is live and the deny half of the new access rule was checked from - the host: it correctly refuses everybody it should. The half that matters cannot be checked - from here — that the one machine it is meant to admit is actually admitted. A typo in - the rule would refuse the control room too and look identical from this side. Until somebody - opens the page at the console, this phase is not finished.
- + Used from the SCADA machine +
The screen has been used from cicore1, the SCADA machine, end + to end. Both halves of the access rule are confirmed: the deny half refuses everybody it + should, and the admit half has now been witnessed from the console itself. + ai-web’s access log records the real client address, and it shows + 10.0.0.21 loading the page on 28 August, the day the rule was applied, and + three times since. Questions followed on 31 August: three POST /ask calls returned + 200, the first 21 seconds after the console loaded the page. An operator opened the screen, + asked, and got an answer, with no sign-in.
+
@@ -652,7 +655,7 @@ footer { margin-top: 72px; padding-top: 22px; border-top: 1px solid var(--hair); ai-web lin001 · proxy · static build ai.yokogawa.tech → 403 - Up 26 h. The 403 is the deny arm working correctly. The allow arm is unproven and cannot be proven from here. + Up 26 h. The 403 is the deny arm working correctly. The allow arm is now proven from the console itself — the access log shows 10.0.0.21 since 28 August. langfuse + lf-db lin001 · ai-internal + proxy
/datadisk/langfuse lf.yokogawa.tech → 302 diff --git a/scripts/deploy.sh b/scripts/deploy.sh index b41318c..c038b96 100644 --- a/scripts/deploy.sh +++ b/scripts/deploy.sh @@ -203,7 +203,10 @@ deploy_web() { warn "Leave the ai.yokogawa.tech entry in the Authelia rule in place anyway;" warn "it is inert, and it keeps rollback to a Caddy reload instead of an" warn "Authelia restart. Verifying from this host gives 403, which is the PASS." - warn "The allow arm can only be proved from cicore1." + warn "The allow arm cannot be proved by curl from here, but it IS proved:" + warn "ai-web logs the real client address Caddy forwards, and 10.0.0.21" + warn "appears there from 28 August (page) and 31 August (questions, 200)." + warn "verify.sh checks that log rather than asking someone to go and look." # A bundle built with VITE_API_BASE set to a public hostname works from # outside and fails on every control-room PC. Catch it here, not on the day. if docker exec ai-web sh -c 'grep -rqs "api\.yokogawa\.tech" /usr/share/nginx/html' 2>/dev/null; then diff --git a/scripts/verify.sh b/scripts/verify.sh index 2fa6d46..5422760 100644 --- a/scripts/verify.sh +++ b/scripts/verify.sh @@ -80,10 +80,15 @@ head_ "ai.yokogawa.tech is closed to everything except the SCADA console" # 403 is the PASS - it proves the deny arm works and that this host, the VPN and # the internet are all shut out. # -# THIS SCRIPT CANNOT PROVE THE ALLOW ARM. A typo in the matcher gives 403 to -# cicore1 too and looks identical from here. Somebody must open -# https://ai.yokogawa.tech on cicore1 and get the UI with no login. There is no -# way around that, and it is the same gap as the Phase 7 end-to-end check. +# A CURL FROM HERE CANNOT PROVE THE ALLOW ARM: a typo in the matcher gives 403 +# to cicore1 too and looks identical from this side. But the evidence does exist +# on this host - ai-web's nginx log records the REAL client address forwarded by +# Caddy, so a page load from 10.0.0.21 proves the matcher admits the console. +# That check is below, and it replaces the old "somebody go and look" step. +# +# PROVEN 2026-08-28: 10.0.0.21 loaded the page the same day the rule was applied, +# and asked three questions on 31 August, each answered 200. It sat unnoticed for +# three days because Caddy keeps no access log of its own. for u in "https://ai.yokogawa.tech" "https://ai.yokogawa.tech/ask"; do code=$(curl -s -o /dev/null -w '%{http_code}' -X POST "$u" \ -H 'Content-Type: application/json' -d '{}' --max-time 10 || echo "000") @@ -96,7 +101,32 @@ for u in "https://ai.yokogawa.tech" "https://ai.yokogawa.tech/ask"; do *) bad "$u -> $code" ;; esac done -printf ' [33m>>[0m ALLOW ARM UNPROVEN: open https://ai.yokogawa.tech on cicore1 (10.0.0.21) - expect the UI, no login\n' +# The allow arm, proved from evidence rather than asserted. Caddy forwards the +# real client address and ai-web logs it as the last field; 10.0.0.21 there means +# the console was admitted. NOTE docker logs are EPHEMERAL - recreating ai-web +# destroys this evidence, so a clean miss is a warning, not a failure. +scada_hits=$(docker logs ai-web 2>&1 | grep -c '"10\.0\.0\.21"' || true) +if [ "${scada_hits:-0}" -gt 0 ]; then + ok "allow arm PROVEN: $scada_hits requests from cicore1 (10.0.0.21) in the ai-web log" + last_hit=$(docker logs ai-web 2>&1 | grep '"10\.0\.0\.21"' | tail -1 | grep -oE '\[[^]]+\]' || true) + ok "most recent console request: ${last_hit:-unknown}" +else + printf ' \033[33m>>\033[0m allow arm NOT evidenced in the current ai-web log. If the container was\n' + printf ' recreated this proves nothing - the log starts empty. Otherwise open\n' + printf ' https://ai.yokogawa.tech on cicore1 (10.0.0.21) - expect the UI, no login.\n' +fi + +# The deny arm from the other direction: anything in the log that is NOT the +# console got past a matcher that should have refused it. Entries dated before +# 2026-08-28 are legitimate - the hostname was Authelia-gated and open then. +foreign=$(docker logs ai-web 2>&1 | grep -oE '"[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+"$' | grep -v '"10\.0\.0\.21"' | sort -u || true) +if [ -n "$foreign" ]; then + printf ' \033[33m>>\033[0m non-console clients in the ai-web log. CHECK THE DATES - anything before\n' + printf ' 2026-08-28 predates the SCADA-only rule and was correctly gated by Authelia:\n' + printf ' %s\n' $foreign +else + ok "no non-console client has ever reached ai-web" +fi # api.yokogawa.tech has no pinpoint DNS record, so it is unresolvable from # inside the VNet. A bundle that hard-codes it loads fine here and fails on a # control-room PC. Check what was actually built into the image. diff --git a/workflow-map.html b/workflow-map.html index 673c639..d4ff079 100644 --- a/workflow-map.html +++ b/workflow-map.html @@ -506,9 +506,10 @@ outside this project’s control. Nobody outside this project is holding up there. It went unnoticed for weeks because the equipment that talks to this host never opens a browser. It no longer affects the operator, who does not sign in at all, but it will catch the next protected service anyone tries to open from the plant floor.

-

What is still unproven is the part no check from here can reach: an operator sitting - at a control-room PC and getting an answer end to end. Names resolving is not the same as a - person succeeding, and a mistake in the new address rule would look identical from here.

+

The part no check from here could reach has now happened: an operator sat at a + control-room PC and got an answer end to end. The screen’s own access log records the + control-room machine opening the page on 28 August and asking questions on 31 August. Names + resolving is not the same as a person succeeding — and this time the person succeeded.

Have: ai, api and cube under yokogawa.tech, plus the internal record for ai.
@@ -640,14 +641,14 @@ through. Every one was invisible while the stand-in was in place. The exam grew 7 The operator’s screen — the question box, the answer, the working -Live, never yet used from the control room -The screen is live, and since 28 August it admits the SCADA machine by its network address with -no sign-in at all — see section 3b for what that costs. The refusing half of that rule -was checked from the host and works: this host, the remote connection and the internet are all shut -out. The half that matters cannot be checked from here — that the one machine it is -meant to admit is actually admitted. A typo in the rule would refuse the control room too and look -exactly the same from this side. Until somebody opens the page at the console, this step is not -finished, and no check written here can finish it. +Used from the control room, end to end +The screen has been used from the SCADA machine, end to end. It is live, and since 28 +August it admits that machine by its network address with no sign-in at all — see +section 3b for what that costs. The refusing half of that rule was checked from the host and works: +this host, the remote connection and the internet are all shut out. The admitting half has now +been confirmed the only way it could be — from the console. The screen’s own record +shows the control-room machine opening the page on 28 August, the day the rule went in, and asking +three questions on 31 August, each answered. 8 The exam — 75 engineer-checked questions run end to end and scored