98083cd8d6
Operators cannot add a document today: ingestion is CLI-only, needs a host
login and a TTY for confirm_header(), /datadisk/ai-docs is mounted read-only,
ai-api has no identity, and nothing in the stack has a role that can write
doc_chunks. This designs the way in, the way out, and control over what is in
the retrieval pool. Design and schema only - no router, worker or UI code yet.
Documents in (16.1-16.9, db/004):
upload -> pre-scan -> review -> approve -> published, with the header
confirmation moved from a terminal prompt to a review screen and recorded
rather than discarded. A CHECK constraint refuses an approved row without a
confirmed number, revision and effective date, so an API bug cannot skip it.
Three roles: agent_ro unchanged, uploads_rw writes the queue only, ingest_rw
writes doc_chunks and has no HTTP surface.
Documents out (16.10-16.11, db/005):
--supersede needs a revision to keep, so a cancelled procedure cannot be
withdrawn at all. Adds withdraw (immediate, reversible, audited), restore
(refused while another revision is live) and purge (off by default). A
column grant plus a trigger let the web-facing role make a document less
citable and never more.
The pool (16.13-16.15, db/006):
pool_enabled, orthogonal to superseded: one is a claim about the document,
the other about the corpus. Retrieval requires both, so re-enabling a
withdrawn document does not make it citable. Named profiles and a
per-request override let a demo trim the corpus without mutating state on a
shared live host, and every reduced-pool answer carries a banner with the
document count, following the used_fixture_data precedent.
Two existing defects found and documented while designing this:
- ai-ingest takes PGUSER=agent_ro from api.env, a SELECT-only role, so the
Phase 3 command in the README cannot write doc_chunks (16.1).
- ingest_file() always inserts superseded = FALSE, so `--all` re-ingests a
superseded revision as live. --supersede survives only until the next bulk
run (16.10).
One commit rather than three: the upload, withdrawal and pool designs
interleave in the same spec, README and compose files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
96 lines
4 KiB
Text
96 lines
4 KiB
Text
# =============================================================================
|
|
# .env.example — every key, no values. Committed deliberately.
|
|
#
|
|
# On lin001 these live as TWO 0600 files under ~/ai/, never in Git:
|
|
# ~/ai/pg-ai.env the POSTGRES_* / AGENT_DB_* block
|
|
# ~/ai/api.env everything else
|
|
# Follow the ~/authelia/authelia.env precedent: chmod 0600, owned by azureuser.
|
|
# =============================================================================
|
|
|
|
# --- pg-ai (~/ai/pg-ai.env) --------------------------------------------------
|
|
POSTGRES_PASSWORD=
|
|
AGENT_DB_USER=agent_ro
|
|
AGENT_DB_PASSWORD=
|
|
|
|
# --- imh (PENDING — leave blank until Phase 4) -------------------------------
|
|
IMH_HOST=yau-sls-poc-imh
|
|
IMH_PORT=1433
|
|
IMH_DB=
|
|
IMH_USER=svc_agent_ro
|
|
IMH_PASSWORD=
|
|
USE_FIXTURES=true # flip to false when imh is live
|
|
|
|
# --- local Postgres ----------------------------------------------------------
|
|
PGHOST=pg-ai
|
|
PGPORT=5432
|
|
PGDATABASE=plant
|
|
PGUSER=agent_ro
|
|
PGPASSWORD=
|
|
|
|
# --- Azure OpenAI ------------------------------------------------------------
|
|
AZURE_OPENAI_ENDPOINT=
|
|
AZURE_OPENAI_API_KEY=
|
|
AZURE_OPENAI_API_VERSION=
|
|
CHAT_DEPLOYMENT= # flagship — final prose only
|
|
CHEAP_DEPLOYMENT= # nano/mini — classifier, entities, tool selection
|
|
EMBED_DEPLOYMENT= # text-embedding-3-small
|
|
|
|
# --- behaviour ---------------------------------------------------------------
|
|
CLASSIFIER_CONFIDENCE_THRESHOLD=0.7
|
|
SITE_TIMEZONE=Australia/Sydney # storage UTC; convert once, in Cube
|
|
MAX_ROWS_RETURNED=5000
|
|
QUERY_TIMEOUT_SECONDS=30
|
|
MAX_OUTPUT_TOKENS=1200
|
|
|
|
# --- Cube --------------------------------------------------------------------
|
|
CUBEJS_API_SECRET=
|
|
CUBEJS_API_URL=http://cube:4000/cubejs-api/v1
|
|
|
|
# --- Langfuse ----------------------------------------------------------------
|
|
LANGFUSE_HOST=http://langfuse:3000
|
|
LANGFUSE_PUBLIC_KEY=
|
|
LANGFUSE_SECRET_KEY=
|
|
LANGFUSE_SALT=
|
|
LANGFUSE_NEXTAUTH_SECRET=
|
|
LANGFUSE_DB_PASSWORD=
|
|
|
|
# --- ingest ------------------------------------------------------------------
|
|
AI_DOCS_ROOT=/datadisk/ai-docs
|
|
CHUNK_TOKEN_TARGET=800
|
|
|
|
# --- document upload (Phase 9) -----------------------------------------------
|
|
# Two more roles, because the component reachable from the internet must not be
|
|
# the component that can write doc_chunks. See db/004_doc_uploads.sql.
|
|
UPLOADS_DB_USER=uploads_rw # ai-api: the queue only, never doc_chunks
|
|
UPLOADS_DB_PASSWORD=
|
|
INGEST_DB_USER=ingest_rw # ai-docs-worker AND the ai-ingest CLI:
|
|
# doc_chunks + the queue. PGUSER is agent_ro
|
|
# and cannot INSERT - see BUILD-AI-CONTAINERS §16.1.
|
|
INGEST_DB_PASSWORD=
|
|
|
|
AI_DOCS_INBOX=/datadisk/ai-docs-inbox # writable staging; NOT the ingest root
|
|
AI_DOCS_WITHDRAWN=/datadisk/ai-docs-withdrawn # withdrawn files are moved, not deleted
|
|
MAX_UPLOAD_MB=50
|
|
UPLOAD_DISK_LIMIT_PCT=90 # refuse uploads above this on /datadisk
|
|
ALLOWED_UPLOAD_EXTENSIONS=.pdf,.docx,.md,.txt
|
|
|
|
# DIRECT membership only — Authelia does not resolve nested groups.
|
|
DOC_PUBLISHER_GROUP=AI_DocPublishers
|
|
# Who may curate the retrieval pool and run trimmed-pool demos. Defaults to the
|
|
# publisher group; point it at a narrower AD group if that should be separate.
|
|
DOC_ADMIN_GROUP=AI_DocPublishers
|
|
ALLOW_SELF_APPROVAL=false # uploader approving their own document
|
|
# Withdrawal (superseded = TRUE) is always available to the publisher group and
|
|
# is reversible. PURGE deletes chunks and is not. Leave it off unless there is a
|
|
# document that must not be in the database at all.
|
|
ALLOW_PURGE=false
|
|
|
|
# Retrieval pool. pool_enabled is orthogonal to superseded - see db/006_doc_pool.sql.
|
|
# Below this share of documents enabled, retrieval drops to an exact scan: the
|
|
# HNSW index is built over ALL embeddings and filters afterwards, so a heavily
|
|
# trimmed pool can return almost nothing. This bites in exactly the demo that
|
|
# trims the pool. Rehearse it.
|
|
POOL_EXACT_SCAN_BELOW_PCT=50
|
|
|
|
WORKER_POLL_SECONDS=10
|
|
WORKER_LEASE_MINUTES=30 # an `ingesting` row older than this is a dead worker
|