An operator standing at the SCADA console should not complete a Duo push to ask a question, and nobody outside the plant should reach the assistant at all. The ai.yokogawa.tech Caddy block now admits remote_ip 10.0.0.21 (yau-poc-cicore1, static) and returns 403 to everything else. Applied on lin001 2026-08-28; snapshot at ~/Caddyfile.bak-ai-scadaonly-20260828. This also settles why the console could not reach the assistant at all: auth.yokogawa.tech has no pinpoint record on the DC, so a LAN browser got a correct 302 to the portal and then died on DNS. It went unnoticed because the device agents write to Influx over the /api/v2/write MFA bypass and never touch the portal - no browser had ever hit Authelia from inside the VNet. Verified before applying that an IP matcher can work here: Caddy sees real client addresses, and WireGuard peers arrive masqueraded as 172.19.0.6 so they do not match and are refused along with the internet. What this costs, recorded in section 14 as a shortcut and not as a security control: it is an IP allowlist on a flat network with no OT/IT boundary, so anything that can take 10.0.0.21 inherits unauthenticated access; Langfuse traces are now anonymous, so there is no record of who asked what; and the assistant is out of browser reach over the VPN. It is in scope for the section 2 OT/safety review, which is still outstanding. api.yokogawa.tech is unchanged and still fully gated - Phase 9 publishing depends on Remote-User/Remote-Groups and stays there. The now-inert ai.yokogawa.tech entry in the Authelia rule is deliberately left in place so restoring the gate is a Caddy reload rather than an Authelia restart that logs out every user on the host. verify.sh treats 403-from-lin001 as the pass for the deny arm and states plainly that the allow arm can only be proved from cicore1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
194 lines
9.5 KiB
Text
194 lines
9.5 KiB
Text
# =============================================================================
|
|
# ai-routes.caddy - the blocks to paste into ~/Caddyfile on lin001.
|
|
#
|
|
# This file is NOT deployed as-is. ~/Caddyfile is a single hand-maintained file
|
|
# with many .bak-* snapshots beside it. Append these blocks, then:
|
|
#
|
|
# cp ~/Caddyfile ~/Caddyfile.bak-ai-$(date +%Y%m%d)
|
|
# docker exec caddy caddy reload --config /etc/caddy/Caddyfile
|
|
#
|
|
# `import authelia` is the shared AD + Duo gate. OMITTING IT SILENTLY MAKES THE
|
|
# SERVICE PUBLIC. Every block below keeps it - there is no deliberate exception
|
|
# anywhere in this stack. (Forgejo omits it only because forward-auth breaks
|
|
# git clients; that reason does not apply to anything here.)
|
|
#
|
|
# DNS is not managed on this host. Each hostname needs an A record ->
|
|
# 20.211.144.151 before Caddy can issue a certificate. Ask Dan.
|
|
#
|
|
# DO NOT ADD A HOSTNAME THIS PROJECT IS NOT BLOCKED WITHOUT. lin001 is shared,
|
|
# already carries a dozen *.yokogawa.tech services, and other people have to
|
|
# work out which name serves what. The cost of a name is not the DNS change, it
|
|
# is the attribution problem it leaves behind. Prefer a path route under a name
|
|
# that already exists - which is what /ask under ai.yokogawa.tech is.
|
|
#
|
|
# What this project owns, and what each one is for:
|
|
#
|
|
# ai.yokogawa.tech the operator's screen, and /ask behind it. The ONLY one
|
|
# a control-room PC needs, and the only one with a DC
|
|
# pinpoint record.
|
|
# api.yokogawa.tech ai-api direct. Used by the eval runner and, at Phase 9,
|
|
# by the /docs endpoints - whose publisher rule is scoped
|
|
# to THIS hostname. Not used by the operator's browser.
|
|
# cube.yokogawa.tech Cube's playground and REST API, for an engineer
|
|
# hand-verifying a measure. ai-api reaches Cube
|
|
# container-to-container and never by hostname, so
|
|
# nothing breaks if this one is dropped.
|
|
# lf.yokogawa.tech Langfuse - traces, prompts, eval runs.
|
|
#
|
|
# DONE 2026-08-27: ai, api and cube .yokogawa.tech all resolve to
|
|
# 20.211.144.151, and the DC carries the pinpoint record
|
|
# ai.yokogawa.tech -> 10.0.0.17 for the Azure hairpin (same treatment as
|
|
# influx.yokogawa.tech). api and cube have NO pinpoint record and are
|
|
# unresolvable from inside the VNet - which is why /ask is routed under
|
|
# ai.yokogawa.tech below rather than called cross-origin.
|
|
#
|
|
# Add each block at the phase that needs it, not all at once. A hostname with a
|
|
# Caddyfile block and no Authelia rule is a hole.
|
|
# =============================================================================
|
|
|
|
# --- Phase 2 -----------------------------------------------------------------
|
|
lf.yokogawa.tech {
|
|
import authelia
|
|
reverse_proxy langfuse:3000
|
|
}
|
|
|
|
# --- Phase 5 -----------------------------------------------------------------
|
|
# NOT ESSENTIAL. Nothing depends on this hostname: ai-api reaches Cube over the
|
|
# proxy network as cube:4000 and never by name, and the operator's screen never
|
|
# touches Cube at all. It exists so an engineer can open Cube's playground to
|
|
# hand-verify a measure - an SSH tunnel does the same job. If this host is being
|
|
# tidied, or a name is wanted for something else, drop this one first. Removing
|
|
# it means deleting this block AND the domain from the Authelia rule; batch that
|
|
# with an Authelia restart that is happening anyway, rather than spending an
|
|
# all-user logout on it.
|
|
cube.yokogawa.tech {
|
|
import authelia
|
|
reverse_proxy cube:4000
|
|
}
|
|
|
|
# --- Phase 6 -----------------------------------------------------------------
|
|
api.yokogawa.tech {
|
|
import authelia
|
|
reverse_proxy ai-api:8000
|
|
}
|
|
|
|
# --- Phase 9 -----------------------------------------------------------------
|
|
# Document upload needs two changes to the block above. No new hostname: the
|
|
# upload endpoints live under /docs on the existing API.
|
|
#
|
|
# 1. ai-api must learn WHO is calling. Authelia returns Remote-User,
|
|
# Remote-Name, Remote-Email and Remote-Groups from the forward-auth
|
|
# subrequest; Caddy only passes them upstream if it is told to.
|
|
#
|
|
# CHECK ~/Caddyfile's shared `authelia` snippet FIRST. If it already sets
|
|
# copy_headers, this is done and duplicating it is a no-op at best. Do NOT
|
|
# edit the shared snippet to add it - every other service on the host
|
|
# imports it, and this is not the change to make on their behalf.
|
|
#
|
|
# CHECKED 2026-08-27: the snippet on lin001 is
|
|
# forward_auth authelia:9091 {
|
|
# uri /api/authz/forward-auth
|
|
# copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
|
|
# }
|
|
# It already copies all four headers, so Phase 9 needs NO change here -
|
|
# only request_body if 50MB uploads are wanted. Note the uri differs from
|
|
# the one written below. Re-check before Phase 9 rather than trusting this
|
|
# line; it is a file other people edit.
|
|
#
|
|
# If the snippet does not copy them, replace the Phase 6 block with:
|
|
#
|
|
# api.yokogawa.tech {
|
|
# forward_auth authelia:9091 {
|
|
# uri /api/verify?rd=https://auth.yokogawa.tech
|
|
# copy_headers Remote-User Remote-Name Remote-Email Remote-Groups
|
|
# }
|
|
# request_body {
|
|
# max_size 50MB # match MAX_UPLOAD_MB; Caddy refuses larger
|
|
# }
|
|
# reverse_proxy ai-api:8000
|
|
# }
|
|
#
|
|
# - matching the shared snippet's own forward_auth arguments, which must be
|
|
# read off the host rather than assumed from this comment.
|
|
#
|
|
# 2. ai-api trusts those headers only because nothing outside the proxy network
|
|
# can reach it. If anything here ever gains a published port, that trust is
|
|
# gone and the /docs endpoints are open to whoever can reach the port.
|
|
#
|
|
# The group restriction is NOT expressed here. It goes in the Authelia rule for
|
|
# ^/docs/.* (authelia/access-rules.md), and is re-checked in ai-api. A Caddy
|
|
# matcher would be a third place to keep in step, and the first to be forgotten.
|
|
|
|
# --- Phase 7 -----------------------------------------------------------------
|
|
# /ask is served under THIS hostname as well as under api.yokogawa.tech, so the
|
|
# operator's page is same-origin. That is not a convenience: LAN hosts resolve
|
|
# yokogawa.tech names through the DC, which holds pinpoint records only, and
|
|
# there is one for ai.yokogawa.tech and none for api.yokogawa.tech. Cross-origin
|
|
# means an operator on cicore1 loads the page and every question fails on DNS.
|
|
#
|
|
# ONLY /ask. Do NOT widen this matcher.
|
|
# - The Phase 9 publisher rule is `domain: api.yokogawa.tech` +
|
|
# `resources: ^/docs/.*` (authelia/access-rules.md). A /docs route under this
|
|
# hostname would not match it, and approving a procedure revision would stop
|
|
# being gated on AI_DocPublishers. ai-api re-checks Remote-Groups itself, so
|
|
# it would not actually fail open - but the rule would be inert and nobody
|
|
# would know.
|
|
# - `/docs*` under this hostname hits the second handle and is served by
|
|
# ai-web, which 404s it. That is the intended outcome. Leave it that way.
|
|
#
|
|
# `route` is required. `import authelia` expands to forward_auth, which sorts
|
|
# AFTER handle in Caddy's default directive order - outside a route block the
|
|
# handles would be terminal and the gate would never run, silently serving the
|
|
# UI to anyone. Inside `route`, directives run in written order.
|
|
# APPLIED 2026-08-28: THIS BLOCK NO LONGER IMPORTS AUTHELIA.
|
|
#
|
|
# At the customer's direction, the operator console is unauthenticated and the
|
|
# assistant is unreachable from anywhere else. Two reasons, both theirs: an
|
|
# operator standing at the SCADA console should not complete a Duo push to ask
|
|
# a question, and nobody outside the plant should reach the assistant at all.
|
|
#
|
|
# 10.0.0.21 (yau-poc-cicore1, static) -> straight through, no auth
|
|
# everything else -> 403
|
|
#
|
|
# Why an IP matcher works here, verified on the host before applying:
|
|
# - Caddy sees REAL client addresses. Docker's DNAT preserves the source; the
|
|
# access log carries public IPs and 10.0.0.5 with no 172.18.x rewriting. If
|
|
# that ever stops being true this block fails CLOSED (403 for everyone),
|
|
# which is the safe direction but will look like an outage.
|
|
# - WireGuard peers do NOT match. The wg container sits on a bridge at
|
|
# 172.19.0.6 and MASQUERADEs outbound, so a peer reaching 10.0.0.17:443
|
|
# arrives as 172.19.0.6. VPN users get 403 - including engineers. Use an
|
|
# SSH tunnel, or add a second matcher, deliberately.
|
|
#
|
|
# WHAT THIS COSTS, and it is not small:
|
|
# - ai-api no longer learns who is asking. Langfuse traces are anonymous and
|
|
# there is no record of who asked what.
|
|
# - It is an IP allowlist on a flat network with NO OT/IT boundary. Anything
|
|
# that can take 10.0.0.21 inherits unauthenticated access. Documented in
|
|
# BUILD-AI-CONTAINERS.md S14 as a shortcut, NOT as a security control.
|
|
# - It is in scope for the S2 OT/safety review, which is still outstanding.
|
|
#
|
|
# DO NOT put /docs here. Phase 9 publishing needs Remote-User/Remote-Groups and
|
|
# stays on api.yokogawa.tech, which is unchanged and still fully gated.
|
|
#
|
|
# TO RESTORE THE GATE: delete the @scada matcher and the two handle wrappers,
|
|
# put `import authelia` back as the first directive inside `route`. That is a
|
|
# Caddy reload only - the ai.yokogawa.tech entry in the Authelia rule was left
|
|
# in place precisely so this does not need an Authelia restart. See
|
|
# authelia/access-rules.md.
|
|
ai.yokogawa.tech {
|
|
@scada remote_ip 10.0.0.21
|
|
route {
|
|
handle @scada {
|
|
handle /ask* {
|
|
reverse_proxy ai-api:8000
|
|
}
|
|
handle {
|
|
reverse_proxy ai-web:80
|
|
}
|
|
}
|
|
handle {
|
|
respond "Not available from this network" 403
|
|
}
|
|
}
|
|
}
|