fd85e62ebf
Phase 9's operator path, built ahead of Phase 8 at the customer's direction and live at api.yokogawa.tech/documents. Upload, convert, review, approve, withdraw and restore. The pool screen is explicitly out of scope. Served by ai-api rather than ai-web, and mounted at /documents rather than /docs. ai.yokogawa.tech is SCADA-only since 2026-08-28 and passes through no Authelia, so it has no identity to record; publishers arrive on api.yokogawa.tech where the forward-auth headers still do. /docs stays with Swagger, which the customer is keeping - two things under one prefix with two different access policies is what gets misread during a later edit. Conversion is text extraction, not document parsing: pypdf, python-docx and openpyxl. Docling would be better at this and pulls torch, which lin001 has neither the memory to install nor the business running next to the demo plant's PLC. The cost is real - no layout, no table structure, and a scan cannot be read at all, so it is refused rather than stored empty. It is acceptable only because the converted text is shown to a person before the document can be cited, which is the same safety net the design already required for the header. convert.py is the one file to change if that stops being true. Chunking is mirrored from ingest.py rather than shared, because the two live in different images. They must stay identical: if they drift, the same document chunks differently depending on who loaded it, and the assistant answers or fails to answer depending on that. The step-sequence rule is locked by a test. Identity is self-asserted for the demo - the actor is typed on the form, which section 16 forbids, and the publisher list is one name with no password. Rows are written as `demo:<name>` with actor_groups = 'DEMO-UNVERIFIED' so that when real auth goes on, a name somebody typed stays tellable from a name Authelia proved. doc_actions cannot be deleted from, so an ambiguity there would be permanent. Two rules the code enforces rather than documents: uploading is open to anyone who reaches the page, because uploading changes nothing an operator can see - approving does, and that is what is gated; and an empty publisher list means nobody, not everybody. Verified on the host end to end: withdraw as a non-publisher 403s, with a short reason 400s, and as admin flips 5 chunks and writes a complete audit row; restore puts them back and keeps both rows. The corpus is unchanged afterwards. Requirements are split so the document dependencies install in their own layer - a change there costs four small wheels instead of re-resolving fastapi, langgraph and langfuse on a 2 vCPU shared host. The five divergences from section 16 are recorded in section 14. The one with teeth: files published through the UI stay in the inbox, so `ai-ingest --all` cannot see them and the two paths must not be used on the same document. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
164 lines
7.6 KiB
Text
164 lines
7.6 KiB
Text
# =============================================================================
|
|
# .env.example — every key, no values. Committed deliberately.
|
|
#
|
|
# On lin001 these live as TWO 0600 files under ~/ai/, never in Git:
|
|
# ~/ai/pg-ai.env the POSTGRES_* / AGENT_DB_* block
|
|
# ~/ai/api.env everything else
|
|
# Follow the ~/authelia/authelia.env precedent: chmod 0600, owned by azureuser.
|
|
# =============================================================================
|
|
|
|
# --- pg-ai (~/ai/pg-ai.env) --------------------------------------------------
|
|
POSTGRES_PASSWORD=
|
|
AGENT_DB_USER=agent_ro
|
|
AGENT_DB_PASSWORD=
|
|
|
|
# --- imh (PENDING — leave blank until Phase 4) -------------------------------
|
|
IMH_HOST=yau-sls-poc-imh
|
|
IMH_PORT=1433
|
|
IMH_DB=
|
|
IMH_USER=svc_agent_ro
|
|
IMH_PASSWORD=
|
|
USE_FIXTURES=true # flip to false when imh is live
|
|
|
|
# --- local Postgres ----------------------------------------------------------
|
|
PGHOST=pg-ai
|
|
PGPORT=5432
|
|
PGDATABASE=plant
|
|
PGUSER=agent_ro
|
|
PGPASSWORD=
|
|
|
|
# --- Azure OpenAI ------------------------------------------------------------
|
|
AZURE_OPENAI_ENDPOINT=
|
|
AZURE_OPENAI_API_KEY=
|
|
AZURE_OPENAI_API_VERSION=
|
|
CHAT_DEPLOYMENT= # flagship — final prose only
|
|
CHEAP_DEPLOYMENT= # nano/mini — classifier, entities, tool selection
|
|
EMBED_DEPLOYMENT= # text-embedding-3-small
|
|
|
|
# --- Documents / upload UI (Phase 9) -----------------------------------------
|
|
# Two roles, and the split IS the safety boundary. db/005 carries a trigger that
|
|
# stops uploads_rw setting superseded = FALSE, so the web path can make a
|
|
# document less visible and never more. ingest_rw is the only one that writes
|
|
# chunks or restores a document.
|
|
UPLOADS_DB_USER=uploads_rw
|
|
UPLOADS_DB_PASSWORD=
|
|
INGEST_DB_USER=ingest_rw
|
|
INGEST_DB_PASSWORD=
|
|
DOCS_INBOX=/inbox
|
|
MAX_UPLOAD_MB=25
|
|
|
|
# authelia | demo
|
|
# authelia the actor is Remote-User from the forward-auth headers. The design.
|
|
# demo the actor is TYPED ON THE FORM. Self-asserted and unverified -
|
|
# exactly what the design forbids. Rows are written as `demo:<name>`
|
|
# with actor_groups = 'DEMO-UNVERIFIED' so they can never be mistaken
|
|
# for authenticated ones, and every screen says so.
|
|
DOC_IDENTITY_MODE=authelia
|
|
# Who may approve, withdraw or restore. Comma-separated. EMPTY MEANS NOBODY and
|
|
# every mutating endpoint 403s - that is the intended failure direction. This
|
|
# stands in for the AD group AI_DocPublishers; swapping to the group later is a
|
|
# config change, not a code change.
|
|
DOC_PUBLISHERS=
|
|
|
|
# --- no-LLM stub mode --------------------------------------------------------
|
|
# OFF for anything real. With it on, no model is called: the class comes from
|
|
# the caller instead of the classifier and the prose is a fixed placeholder.
|
|
# Every answer carries stub_mode: true and a banner. See api/stub.py for what
|
|
# it proves and what it does not.
|
|
NO_LLM_STUB=false
|
|
|
|
# --- behaviour ---------------------------------------------------------------
|
|
CLASSIFIER_CONFIDENCE_THRESHOLD=0.7
|
|
# Where the CONTROLLED copy of a procedure actually lives - a site fact, the
|
|
# same for every document, so it is not in doc_chunks. Set this to the real
|
|
# DMS location. The default names who to ask, which is always true and never
|
|
# sends anybody to a place that does not exist.
|
|
CONTROLLED_COPY_LOCATION=
|
|
SITE_TIMEZONE=Australia/Sydney # storage UTC; convert once, in Cube
|
|
MAX_ROWS_RETURNED=5000
|
|
QUERY_TIMEOUT_SECONDS=30
|
|
MAX_OUTPUT_TOKENS=1200
|
|
|
|
# --- Cube --------------------------------------------------------------------
|
|
CUBEJS_API_SECRET=
|
|
CUBEJS_API_URL=http://cube:4000/cubejs-api/v1
|
|
|
|
# The upstream source Cube reads. ai-compose.yml says these come from this
|
|
# file, and this file did not list them - so the first person to deploy Cube
|
|
# had to work out from the compose comments and db/003_roles.sql what the
|
|
# service actually needed. While USE_FIXTURES=true the fixtures live in pg-ai,
|
|
# so this points at pg-ai with the cube_rw credentials from pg-ai.env.
|
|
# At Phase 4 the whole block becomes CUBEJS_DB_TYPE=mssql against imh.
|
|
CUBEJS_DB_TYPE=postgres
|
|
CUBEJS_DB_HOST=pg-ai
|
|
CUBEJS_DB_PORT=5432
|
|
CUBEJS_DB_NAME=plant
|
|
CUBEJS_DB_USER=cube_rw # CUBE_DB_USER in pg-ai.env
|
|
CUBEJS_DB_PASS= # CUBE_DB_PASSWORD in pg-ai.env
|
|
|
|
# NOT CUBEJS_EXT_DB_*. Cube v1 will not use Postgres as an external
|
|
# pre-aggregation store; cubestore does that job and needs no keys here.
|
|
# See the note above the cube service in compose/ai-compose.yml.
|
|
|
|
# --- Langfuse ----------------------------------------------------------------
|
|
# PROJECT keys, created in the Langfuse UI - not the server's own secrets
|
|
# (SALT, NEXTAUTH_SECRET), which live in langfuse.env and are a different
|
|
# thing. The two must be a MATCHED PAIR from the same key: a public key from
|
|
# one pair with a secret from another authenticates as neither.
|
|
#
|
|
# Langfuse shows the secret ONCE, at creation, and stores only a hash - the
|
|
# database keeps a masked form (sk-lf-...abcd) and nothing can recover it. If
|
|
# it is lost, generate a new pair; there is no way to read the old one back.
|
|
#
|
|
# Getting this wrong is silent in both directions. Keys absent -> _langfuse()
|
|
# returns None and every question is simply untraced. Keys present but wrong ->
|
|
# a client is built, Langfuse rejects it, and main.py swallows the exception on
|
|
# purpose, because observability must never break the answer path. Neither case
|
|
# logs anything. Confirm traces are ARRIVING; do not infer it from config.
|
|
LANGFUSE_HOST=http://langfuse:3000
|
|
LANGFUSE_PUBLIC_KEY=
|
|
LANGFUSE_SECRET_KEY=
|
|
LANGFUSE_SALT=
|
|
LANGFUSE_NEXTAUTH_SECRET=
|
|
LANGFUSE_DB_PASSWORD=
|
|
|
|
# --- ingest ------------------------------------------------------------------
|
|
AI_DOCS_ROOT=/datadisk/ai-docs
|
|
CHUNK_TOKEN_TARGET=800
|
|
|
|
# --- document upload (Phase 9) -----------------------------------------------
|
|
# Two more roles, because the component reachable from the internet must not be
|
|
# the component that can write doc_chunks. See db/004_doc_uploads.sql.
|
|
UPLOADS_DB_USER=uploads_rw # ai-api: the queue only, never doc_chunks
|
|
UPLOADS_DB_PASSWORD=
|
|
INGEST_DB_USER=ingest_rw # ai-docs-worker AND the ai-ingest CLI:
|
|
# doc_chunks + the queue. PGUSER is agent_ro
|
|
# and cannot INSERT - see BUILD-AI-CONTAINERS §16.1.
|
|
INGEST_DB_PASSWORD=
|
|
|
|
AI_DOCS_INBOX=/datadisk/ai-docs-inbox # writable staging; NOT the ingest root
|
|
AI_DOCS_WITHDRAWN=/datadisk/ai-docs-withdrawn # withdrawn files are moved, not deleted
|
|
MAX_UPLOAD_MB=50
|
|
UPLOAD_DISK_LIMIT_PCT=90 # refuse uploads above this on /datadisk
|
|
ALLOWED_UPLOAD_EXTENSIONS=.pdf,.docx,.md,.txt
|
|
|
|
# DIRECT membership only — Authelia does not resolve nested groups.
|
|
DOC_PUBLISHER_GROUP=AI_DocPublishers
|
|
# Who may curate the retrieval pool and run trimmed-pool demos. Defaults to the
|
|
# publisher group; point it at a narrower AD group if that should be separate.
|
|
DOC_ADMIN_GROUP=AI_DocPublishers
|
|
ALLOW_SELF_APPROVAL=false # uploader approving their own document
|
|
# Withdrawal (superseded = TRUE) is always available to the publisher group and
|
|
# is reversible. PURGE deletes chunks and is not. Leave it off unless there is a
|
|
# document that must not be in the database at all.
|
|
ALLOW_PURGE=false
|
|
|
|
# Retrieval pool. pool_enabled is orthogonal to superseded - see db/006_doc_pool.sql.
|
|
# Below this share of documents enabled, retrieval drops to an exact scan: the
|
|
# HNSW index is built over ALL embeddings and filters afterwards, so a heavily
|
|
# trimmed pool can return almost nothing. This bites in exactly the demo that
|
|
# trims the pool. Rehearse it.
|
|
POOL_EXACT_SCAN_BELOW_PCT=50
|
|
|
|
WORKER_POLL_SECONDS=10
|
|
WORKER_LEASE_MINUTES=30 # an `ingesting` row older than this is a dead worker
|