YAU.POC/yau-plant-assistant
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add the document library screens: upload, review, withdraw, restore

Browse source 
Phase 9's operator path, built ahead of Phase 8 at the customer's direction and
live at api.yokogawa.tech/documents. Upload, convert, review, approve, withdraw
and restore. The pool screen is explicitly out of scope.

Served by ai-api rather than ai-web, and mounted at /documents rather than
/docs. ai.yokogawa.tech is SCADA-only since 2026-08-28 and passes through no
Authelia, so it has no identity to record; publishers arrive on
api.yokogawa.tech where the forward-auth headers still do. /docs stays with
Swagger, which the customer is keeping - two things under one prefix with two
different access policies is what gets misread during a later edit.

Conversion is text extraction, not document parsing: pypdf, python-docx and
openpyxl. Docling would be better at this and pulls torch, which lin001 has
neither the memory to install nor the business running next to the demo plant's
PLC. The cost is real - no layout, no table structure, and a scan cannot be read
at all, so it is refused rather than stored empty. It is acceptable only because
the converted text is shown to a person before the document can be cited, which
is the same safety net the design already required for the header. convert.py is
the one file to change if that stops being true.

Chunking is mirrored from ingest.py rather than shared, because the two live in
different images. They must stay identical: if they drift, the same document
chunks differently depending on who loaded it, and the assistant answers or
fails to answer depending on that. The step-sequence rule is locked by a test.

Identity is self-asserted for the demo - the actor is typed on the form, which
section 16 forbids, and the publisher list is one name with no password. Rows are
written as `demo:<name>` with actor_groups = 'DEMO-UNVERIFIED' so that when real
auth goes on, a name somebody typed stays tellable from a name Authelia proved.
doc_actions cannot be deleted from, so an ambiguity there would be permanent.

Two rules the code enforces rather than documents: uploading is open to anyone
who reaches the page, because uploading changes nothing an operator can see -
approving does, and that is what is gated; and an empty publisher list means
nobody, not everybody.

Verified on the host end to end: withdraw as a non-publisher 403s, with a short
reason 400s, and as admin flips 5 chunks and writes a complete audit row;
restore puts them back and keeps both rows. The corpus is unchanged afterwards.

Requirements are split so the document dependencies install in their own layer -
a change there costs four small wheels instead of re-resolving fastapi,
langgraph and langfuse on a 2 vCPU shared host.

The five divergences from section 16 are recorded in section 14. The one with
teeth: files published through the UI stay in the inbox, so `ai-ingest --all`
cannot see them and the two paths must not be used on the same document.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude 2026-08-28 14:13:17 +10:00
parent 5e0984b2cf
commit fd85e62ebf
13 changed files with 1743 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

25
.env.example
View file

@ -35,6 +35,31 @@ CHAT_DEPLOYMENT= # flagship — final prose only
CHEAP_DEPLOYMENT= # nano/mini — classifier, entities, tool selection CHEAP_DEPLOYMENT= # nano/mini — classifier, entities, tool selection
EMBED_DEPLOYMENT= # text-embedding-3-small EMBED_DEPLOYMENT= # text-embedding-3-small
# --- Documents / upload UI (Phase 9) -----------------------------------------
# Two roles, and the split IS the safety boundary. db/005 carries a trigger that
# stops uploads_rw setting superseded = FALSE, so the web path can make a
# document less visible and never more. ingest_rw is the only one that writes
# chunks or restores a document.
UPLOADS_DB_USER=uploads_rw
UPLOADS_DB_PASSWORD=
INGEST_DB_USER=ingest_rw
INGEST_DB_PASSWORD=
DOCS_INBOX=/inbox
MAX_UPLOAD_MB=25
# authelia | demo
# authelia the actor is Remote-User from the forward-auth headers. The design.
# demo the actor is TYPED ON THE FORM. Self-asserted and unverified -
# exactly what the design forbids. Rows are written as `demo:<name>`
# with actor_groups = 'DEMO-UNVERIFIED' so they can never be mistaken
# for authenticated ones, and every screen says so.
DOC_IDENTITY_MODE=authelia
# Who may approve, withdraw or restore. Comma-separated. EMPTY MEANS NOBODY and
# every mutating endpoint 403s - that is the intended failure direction. This
# stands in for the AD group AI_DocPublishers; swapping to the group later is a
# config change, not a code change.
DOC_PUBLISHERS=
# --- no-LLM stub mode -------------------------------------------------------- # --- no-LLM stub mode --------------------------------------------------------
# OFF for anything real. With it on, no model is called: the class comes from # OFF for anything real. With it on, no model is called: the class comes from
# the caller instead of the classifier and the prose is a fixed placeholder. # the caller instead of the classifier and the prose is a fixed placeholder.

8
BUILD-AI-CONTAINERS.md
View file

@ -724,6 +724,14 @@ local time changes the answer.
- Uploaded documents are not scanned for malware — there is no ClamAV on this host. Extension, magic bytes and size only, on a host that also runs the demo PLC - Uploaded documents are not scanned for malware — there is no ClamAV on this host. Extension, magic bytes and size only, on a host that also runs the demo PLC
- `ai-api` trusts Authelia's `Remote-User` / `Remote-Groups` headers because nothing outside the `proxy` network can reach it. Any container on `proxy` could forge them; that assumption is exactly as strong as the no-published-ports rule - `ai-api` trusts Authelia's `Remote-User` / `Remote-Groups` headers because nothing outside the `proxy` network can reach it. Any container on `proxy` could forge them; that assumption is exactly as strong as the no-published-ports rule
**Phase 9 as built, 2026-08-28.** The document screens are live at `api.yokogawa.tech/documents` and diverge from §16 in five ways. All five are reversible and none needed anything from outside the project; each is a demo affordance, not a design improvement.
- **Identity is self-asserted.** `DOC_IDENTITY_MODE=demo`: the actor is typed on the form, not taken from `Remote-User`, which is exactly what §16 forbids. Rows are written as `demo:<name>` with `actor_groups = 'DEMO-UNVERIFIED'` and every screen says so, precisely so that a self-asserted row stays tellable from an authenticated one after real auth goes on — `doc_actions` is a table nothing can delete from, so an ambiguity there is permanent. **The publisher list is one name, `admin`, with no password**, standing in for `AI_DocPublishers`; anyone who reaches the page can claim it. Swap `DOC_IDENTITY_MODE=authelia` and `DOC_PUBLISHERS` for the AD group and this is closed.
- **No `ai-docs-worker`.** Conversion, chunking and embedding run inside the HTTP request, and the same process holds both `uploads_rw` and `ingest_rw`. `db/005`'s trigger still stops the web role un-withdrawing anything, so the boundary holds — but it is now a code boundary rather than a deployment one, and a large upload blocks its own request.
- **Text extraction, not document parsing.** pypdf, python-docx and openpyxl instead of Docling, because Docling pulls torch and lin001 must not build or run that. No layout, no table structure, and **scans cannot be read at all** — they are refused rather than stored empty. Tolerable only because a person reads the converted text before it can be cited. `api/convert.py` is the single file to change.
- **Chunking is duplicated** between `api/chunking.py` and `ingest/ingest.py`, because they live in different images. They must stay identical or the same document chunks differently depending on who loaded it. `api/tests/test_documents.py` locks the rule that matters; it cannot see drift in `ingest.py`.
- **Published files stay in `/datadisk/ai-docs-inbox`** and are never moved into `/datadisk/ai-docs`. `ai-api` has no write access to the document tree. Consequence: `ai-ingest --all` cannot see anything published through the UI, so **the two ingest paths must not be used on the same document**.
Production closes these in the order: network segmentation → secrets → SQL guardrails → document control integration → HA. Phase 9 makes document control integration the more urgent of those, not less: once operators can add documents, the assistant's document set drifts from the controlled set faster. Production closes these in the order: network segmentation → secrets → SQL guardrails → document control integration → HA. Phase 9 makes document control integration the more urgent of those, not less: once operators can add documents, the assistant's document set drifts from the controlled set faster.
--- ---

22
README.md
View file

@ -284,6 +284,28 @@ engineer with access to `imh`, not something this script can decide.
### 8. Phase 9 — operator document upload ### 8. Phase 9 — operator document upload
**BUILT AND LIVE 2026-08-28, ahead of Phase 8, at the customer's direction.**
The screens are at `https://api.yokogawa.tech/documents` — served by `ai-api`,
not `ai-web`, because `ai.yokogawa.tech` is now SCADA-only and carries no
identity at all. Upload → convert → review → approve, plus withdraw and
restore. The pool screen was explicitly descoped.
Files are converted to text with pypdf / python-docx / openpyxl and the
converted text is shown to the reviewer before approval — the raw file is never
what the assistant reads, and a bad conversion is meant to be caught by eye.
**Scanned documents cannot be read** and are refused rather than stored empty.
Two things to know before trusting it. **Identity is self-asserted**: the
publisher is a typed name checked against a one-entry list (`admin`) with no
password, so anyone who reaches the page can claim it. Rows are marked
`demo:<name>` / `DEMO-UNVERIFIED` so they stay distinguishable from
authenticated ones later. And **the two ingest paths must not be used on the
same document** — files published through the UI stay in the inbox and
`ai-ingest --all` cannot see them. Full list of divergences in
[`BUILD-AI-CONTAINERS.md`](BUILD-AI-CONTAINERS.md) §14.
The design below is what §16 specifies, and remains the target.
**After Phase 8 passes, not before.** When the PLC logic or the SCADA program **After Phase 8 passes, not before.** When the PLC logic or the SCADA program
changes, a new document is issued and the assistant is wrong about the plant changes, a new document is issued and the assistant is wrong about the plant
until it is ingested. Today that needs SSH to a live shared host. Phase 9 puts until it is ingested. Today that needs SSH to a live shared host. Phase 9 puts

6
api/Dockerfile
View file

@ -11,6 +11,12 @@ WORKDIR /app
COPY requirements.txt . COPY requirements.txt .
RUN pip install --no-cache-dir --requirement requirements.txt RUN pip install --no-cache-dir --requirement requirements.txt
# The document-upload dependencies go in a SECOND layer, so changing them does
# not invalidate the one above. Four pure-Python wheels, a few megabytes, no
# compiler. See requirements-docs.txt for what belongs here and what does not.
COPY requirements-docs.txt .
RUN pip install --no-cache-dir --requirement requirements-docs.txt
COPY . . COPY . .
USER appuser USER appuser

174
api/chunking.py Normal file
View file

@ -0,0 +1,174 @@
"""Sectioning, chunking and header extraction — MIRRORED FROM ingest/ingest.py.
WHY THIS FILE IS A COPY, WHICH IS NORMALLY THE WRONG ANSWER.
Documents reach `doc_chunks` by two routes now: `ai-ingest` at a terminal, and
the upload screen in this API. **They must chunk identically.** If they do not,
the same document ingested by the two paths produces different chunks,
different embeddings and different retrieval behaviour — and the difference
would show up as an assistant that answers a question correctly or not
depending on who loaded the document, which is close to undiagnosable.
The functions below are therefore mirrored from `ingest/ingest.py` with the
logic unchanged. They are not imported from it because `ingest.py` lives in a
different image, with a different and heavier dependency set.
A shared module needs both images built from a common context, which is a
deployment change to a live host and not one to make in the same stroke as a
new feature.
**If you change chunking, change it in BOTH files.** `api/tests/test_chunking.py`
locks the behaviour that matters — the refusal to split a numbered step
sequence — so a drift in that rule fails the suite. It cannot detect drift in
`ingest.py`, which is the residual risk and is recorded in
BUILD-AI-CONTAINERS.md S14.
Mirrored: MARKDOWN_HEADING, STEP_RE, the header regexes, approx_tokens,
split_sections, chunk_section, link_equipment, extract_header, parse_date.
NOT mirrored: parse_document, and everything that touches the disk.
"""
from __future__ import annotations
import os
import re
from dataclasses import dataclass
from datetime import date, datetime
CHUNK_TOKEN_TARGET = int(os.environ.get("CHUNK_TOKEN_TARGET", "800"))
MARKDOWN_HEADING = re.compile(r"^(#{1,6})\s+(.*\S)\s*$")
DOC_NUMBER_RE = re.compile(r"\b(WRPS-[A-Z]{2,4}-\d{3,4})\b")
REVISION_RE = re.compile(
r"\b(?:rev(?:ision)?|issue)[\s.:]*([A-Z0-9]{1,4})\b", re.IGNORECASE
)
DATE_RE = re.compile(
r"\b(?:effective|issued|approved)[\s\w]{0,12}?[:\s]\s*"
r"(\d{1,2}[/-]\d{1,2}[/-]\d{2,4}|\d{4}-\d{2}-\d{2}|"
r"\d{1,2}\s+\w+\s+\d{4})\b",
re.IGNORECASE,
)
TITLE_RE = re.compile(r"^\s*title[\s:]+(.+\S)\s*$", re.IGNORECASE | re.MULTILINE)
AUTHORISING_ROLE_RE = re.compile(
r"^[ \t]*authoris(?:ing|ed)[ \t]*(?:role|by)?[ \t]*:[ \t]*(.+\S)[ \t]*$",
re.IGNORECASE | re.MULTILINE,
)
# A numbered step. Used to refuse to split, not to parse the procedure.
STEP_RE = re.compile(r"^\s*(?:\d+\.|\(\d+\)|step\s+\d+)", re.IGNORECASE | re.MULTILINE)
@dataclass(frozen=True)
class Header:
doc_number: str | None
revision: str | None
effective_date: date | None
title: str | None
authorising_role: str | None
def parse_date(text: str) -> date | None:
for fmt in ("%d/%m/%Y", "%d-%m-%Y", "%Y-%m-%d", "%d %B %Y", "%d %b %Y", "%d/%m/%y"):
try:
return datetime.strptime(text.strip(), fmt).date()
except ValueError:
continue
return None
def extract_header(text: str) -> Header:
"""Pull document identity from the first page. ALWAYS confirmed by a human.
Everything here is a proposal shown on the review screen with the field
already filled in. It is not authority. A wrong revision on a procedure is
a safety issue, and a regex is not a person.
"""
head = text[:4000]
number = DOC_NUMBER_RE.search(head)
revision = REVISION_RE.search(head)
effective = DATE_RE.search(head)
title = TITLE_RE.search(head)
role = AUTHORISING_ROLE_RE.search(head)
return Header(
doc_number=number.group(1) if number else None,
revision=revision.group(1) if revision else None,
effective_date=parse_date(effective.group(1)) if effective else None,
title=title.group(1).strip() if title else None,
authorising_role=role.group(1).strip() if role else None,
)
def approx_tokens(text: str) -> int:
"""Rough, and deliberately so - it decides when to split, and the rule that
matters is the one that refuses to."""
return len(text) // 4
def split_sections(text: str) -> list[tuple[int, str, str]]:
"""Markdown -> [(page, section_title, section_text)].
Mirrors ingest.parse_markdown, which takes a Path; this takes the text,
because by the time it gets here the upload has already been converted in
memory.
Page is always 1. A converted document has no pages that survive the
conversion, so a citation to one carries a section title and no page
number. That is honest. Do not invent page numbers to make citations look
uniform.
"""
sections: list[tuple[int, str, list[str]]] = []
current_title = "(untitled)"
buffer: list[str] = []
for line in text.splitlines():
heading = MARKDOWN_HEADING.match(line)
if heading:
if buffer:
sections.append((1, current_title, buffer))
current_title = heading.group(2).strip()
buffer = []
elif line.strip():
buffer.append(line.rstrip())
if buffer:
sections.append((1, current_title, buffer))
return [(page, title, "\n".join(body)) for page, title, body in sections]
def chunk_section(text: str, doc_type: str) -> list[str]:
"""Split a section, unless splitting it would break a step sequence.
For procedures the rule is absolute: a section containing numbered steps is
emitted whole, however long it is. An oversized chunk costs tokens. Half a
procedure costs more than that.
"""
if approx_tokens(text) <= CHUNK_TOKEN_TARGET:
return [text]
if doc_type == "procedure" and STEP_RE.search(text):
return [text]
chunks: list[str] = []
buffer: list[str] = []
for paragraph in text.split("\n\n"):
candidate = "\n\n".join(buffer + [paragraph])
if buffer and approx_tokens(candidate) > CHUNK_TOKEN_TARGET:
chunks.append("\n\n".join(buffer))
buffer = [paragraph]
else:
buffer.append(paragraph)
if buffer:
chunks.append("\n\n".join(buffer))
return chunks
def link_equipment(text: str, equipment_ids: list[str]) -> str | None:
"""Tie a chunk to equipment when it is unambiguously about one thing.
Two different units mentioned means no link, not a guess - a chunk linked
to the wrong pump is worse than one linked to nothing, because retrieval
filtering will then hide it from the pump it actually describes.
"""
found = {eid for eid in equipment_ids if eid.lower() in text.lower()}
return found.pop() if len(found) == 1 else None

77
api/config.py
View file

@ -53,6 +53,52 @@ class Settings(BaseModel):
query_timeout_seconds: int = 30 query_timeout_seconds: int = 30
max_output_tokens: int = 1200 max_output_tokens: int = 1200
# --- Documents (Phase 9) -----------------------------------------------
# Two extra roles, and the split is the safety boundary, not bookkeeping.
#
# uploads_rw the queue, and withdraw. db/005 carries a trigger that
# refuses to let this role set superseded = FALSE, so it can
# make a document LESS visible and never more.
# ingest_rw writes doc_chunks: approve and restore. Anything that makes
# a document citable uses this one.
#
# In the design these are two components - ai-api and ai-docs-worker, the
# second with no HTTP surface. There is no worker here, so both connections
# live in this process. That is a real deviation and it is recorded in
# BUILD-AI-CONTAINERS.md S14: the trigger still blocks the web role, but the
# process holding uploads_rw also holds ingest_rw, so the separation is now
# a code boundary rather than a deployment one. Splitting the worker out
# later is a config change and a compose file, not a redesign.
uploads_db_user: str = "uploads_rw"
uploads_db_password: str = ""
ingest_db_user: str = "ingest_rw"
ingest_db_password: str = ""
# Where uploaded files land before review. Never inside /datadisk/ai-docs:
# that folder means "the documents this plant runs on", and an unreviewed
# upload is not one of those.
docs_inbox: str = "/inbox"
max_upload_mb: int = 25
# --- Document identity -------------------------------------------------
# "authelia" - the actor is Remote-User from the forward-auth headers, and
# a missing header is a 401. This is the design.
# "demo" - the actor is TYPED BY THE PERSON on the form. Self-asserted,
# unverified, and exactly what the design forbids ("identity
# comes from the headers, never from the request body").
#
# Demo mode is off unless asked for, the screens carry a banner saying the
# name is unverified, and every row it writes is stored as `demo:<name>`
# with actor_groups = 'DEMO-UNVERIFIED'. That marking is the point: when
# real auth goes on, a self-asserted audit row must still be tellable from
# an authenticated one. Without it they are indistinguishable forever.
doc_identity_mode: str = "authelia"
# Who may change anything. In the design this is the AD group
# AI_DocPublishers; with no AD group available it is a name list, swapped
# for the group later with one config change. Empty means nobody, and the
# API fails closed - approve, withdraw and restore all 403.
doc_publishers: tuple[str, ...] = ()
# --- Cube -------------------------------------------------------------- # --- Cube --------------------------------------------------------------
cubejs_api_url: str = "http://cube:4000/cubejs-api/v1" cubejs_api_url: str = "http://cube:4000/cubejs-api/v1"
cubejs_api_secret: str = "" cubejs_api_secret: str = ""
@ -69,10 +115,27 @@ class Settings(BaseModel):
f"@{self.pghost}:{self.pgport}/{self.pgdatabase}" f"@{self.pghost}:{self.pgport}/{self.pgdatabase}"
) )
def docs_dsn(self, role: str) -> str:
"""DSN for one of the document roles. Never log the result.
`role` is "uploads" or "ingest" - spelled out at every call site rather
than defaulted, because picking the wrong one is the difference between
a web request that can withdraw a document and one that can publish it.
"""
user, password = {
"uploads": (self.uploads_db_user, self.uploads_db_password),
"ingest": (self.ingest_db_user, self.ingest_db_password),
}[role]
return (
f"postgresql://{user}:{password}"
f"@{self.pghost}:{self.pgport}/{self.pgdatabase}"
)
def redacted(self) -> dict[str, object]: def redacted(self) -> dict[str, object]:
"""Safe to log and safe to return from /healthz.""" """Safe to log and safe to return from /healthz."""
secret = {"pgpassword", "azure_openai_api_key", "cubejs_api_secret", secret = {"pgpassword", "azure_openai_api_key", "cubejs_api_secret",
"langfuse_secret_key"} "langfuse_secret_key", "uploads_db_password",
"ingest_db_password"}
return { return {
k: ("set" if v else "unset") if k in secret else v k: ("set" if v else "unset") if k in secret else v
for k, v in self.model_dump().items() for k, v in self.model_dump().items()
@ -108,6 +171,18 @@ def settings() -> Settings:
max_rows_returned=int(env.get("MAX_ROWS_RETURNED", "5000")), max_rows_returned=int(env.get("MAX_ROWS_RETURNED", "5000")),
query_timeout_seconds=int(env.get("QUERY_TIMEOUT_SECONDS", "30")), query_timeout_seconds=int(env.get("QUERY_TIMEOUT_SECONDS", "30")),
max_output_tokens=int(env.get("MAX_OUTPUT_TOKENS", "1200")), max_output_tokens=int(env.get("MAX_OUTPUT_TOKENS", "1200")),
uploads_db_user=env.get("UPLOADS_DB_USER", "uploads_rw"),
uploads_db_password=env.get("UPLOADS_DB_PASSWORD", ""),
ingest_db_user=env.get("INGEST_DB_USER", "ingest_rw"),
ingest_db_password=env.get("INGEST_DB_PASSWORD", ""),
docs_inbox=env.get("DOCS_INBOX", "/inbox"),
max_upload_mb=int(env.get("MAX_UPLOAD_MB", "25")),
doc_identity_mode=env.get("DOC_IDENTITY_MODE", "authelia").lower(),
# Comma-separated. Blank entries dropped so a trailing comma in an env
# file cannot silently authorise "".
doc_publishers=tuple(
n.strip() for n in env.get("DOC_PUBLISHERS", "").split(",") if n.strip()
),
cubejs_api_url=env.get("CUBEJS_API_URL", "http://cube:4000/cubejs-api/v1"), cubejs_api_url=env.get("CUBEJS_API_URL", "http://cube:4000/cubejs-api/v1"),
cubejs_api_secret=env.get("CUBEJS_API_SECRET", ""), cubejs_api_secret=env.get("CUBEJS_API_SECRET", ""),
langfuse_host=env.get("LANGFUSE_HOST", "http://langfuse:3000"), langfuse_host=env.get("LANGFUSE_HOST", "http://langfuse:3000"),

217
api/convert.py Normal file
View file

@ -0,0 +1,217 @@
"""Uploaded file -> Markdown, before anything is chunked or embedded.
WHY THERE IS A CONVERSION STEP AT ALL. The raw PDF is never what the assistant
reads - it never was. Retrieval only ever sees `doc_chunks.chunk_text`. What
this module adds is that the extracted text becomes a thing a person can LOOK
AT before approving it. If a table comes out as garbage, the reviewer sees the
garbage and rejects the document, instead of an operator discovering it months
later inside a citation.
WHAT THESE CONVERTERS DO AND DO NOT DO. They extract text, not layout. A
multi-column page interleaves. A merged-cell spreadsheet flattens. A scanned
page yields nothing at all, and is refused rather than stored empty.
That limitation is acceptable HERE, and only here, because a person reads the
converted text before it can be cited - the same safety net the design already
required for the document header. It would not be acceptable in a pipeline that
published without review.
REPLACING THEM. Everything below the `convert()` boundary is swappable: return
Markdown from bytes, raise ConversionError when you cannot. Nothing outside this
module knows which library did the work, so a better converter is a change to
this file alone. If the real documents turn out to be scans, that is the
conversation to have - OCR is the missing capability, and nothing else here
changes.
"""
from __future__ import annotations
import io
import re
from dataclasses import dataclass
# A scanned page yields a handful of stray characters, not nothing - so an
# emptiness check has to have a floor above zero. Below this, across the whole
# document, we refuse rather than store a blank document that a reviewer might
# approve without noticing there is nothing in it.
_MIN_CHARS = 200
SUPPORTED = {".pdf", ".docx", ".xlsx", ".xlsm", ".md", ".txt"}
class ConversionError(Exception):
"""Conversion failed, or produced something not worth reviewing."""
@dataclass(frozen=True)
class Converted:
markdown: str
converter: str # recorded on the upload row - which code produced this
page_count: int | None
def _clean(text: str) -> str:
"""Collapse the whitespace damage that text extraction always leaves.
Not cosmetic: runs of blank lines and trailing spaces change where a chunker
splits, so the same document converted twice should look the same.
"""
text = text.replace("\r\n", "\n").replace("\r", "\n")
text = re.sub(r"[ \t]+\n", "\n", text)
text = re.sub(r"\n{3,}", "\n\n", text)
return text.strip()
def _from_pdf(data: bytes) -> Converted:
from pypdf import PdfReader
try:
reader = PdfReader(io.BytesIO(data))
except Exception as exc:
raise ConversionError(f"not a readable PDF: {exc}") from exc
if reader.is_encrypted:
# Refuse rather than guess at an empty password. A locked document that
# silently converts to nothing is the worst outcome here.
raise ConversionError(
"this PDF is encrypted - remove the protection and upload it again"
)
parts: list[str] = []
for number, page in enumerate(reader.pages, start=1):
try:
text = page.extract_text() or ""
except Exception:
# One bad page must not lose the other ninety. Mark it so the
# reviewer can see exactly what is missing.
text = ""
parts.append(f"\n\n<!-- page {number}: could not be extracted -->\n")
if text.strip():
parts.append(f"\n\n<!-- page {number} -->\n\n{text}")
return Converted(
markdown=_clean("".join(parts)),
converter="pypdf",
page_count=len(reader.pages),
)
def _from_docx(data: bytes) -> Converted:
import docx
try:
document = docx.Document(io.BytesIO(data))
except Exception as exc:
raise ConversionError(f"not a readable .docx: {exc}") from exc
lines: list[str] = []
for para in document.paragraphs:
text = para.text.strip()
if not text:
continue
# Word's built-in heading styles are the one piece of structure that
# survives reliably, and headings matter: section_title is what a
# citation falls back to.
style = (para.style.name or "").lower() if para.style else ""
if style.startswith("heading"):
level = "".join(c for c in style if c.isdigit()) or "1"
lines.append(f"{'#' * min(int(level), 6)} {text}")
elif style.startswith("title"):
lines.append(f"# {text}")
else:
lines.append(text)
for index, table in enumerate(document.tables, start=1):
lines.append(f"\n<!-- table {index} -->")
for row in table.rows:
cells = [c.text.strip().replace("|", "\\|") for c in row.cells]
lines.append("| " + " | ".join(cells) + " |")
return Converted(
markdown=_clean("\n\n".join(lines)),
converter="python-docx",
page_count=None,
)
def _from_xlsx(data: bytes) -> Converted:
from openpyxl import load_workbook
try:
# read_only keeps a large workbook from being held in memory twice, and
# data_only takes the cached VALUE of a formula rather than the formula
# text. A cell reading "=SUM(B2:B9)" is not something to embed - and if
# the workbook was never opened in Excel there is no cached value, so
# that cell comes through empty. Say so in the output rather than
# letting it look like a blank cell.
book = load_workbook(io.BytesIO(data), read_only=True, data_only=True)
except Exception as exc:
raise ConversionError(f"not a readable spreadsheet: {exc}") from exc
lines: list[str] = []
for sheet in book.worksheets:
lines.append(f"## {sheet.title}")
for row in sheet.iter_rows(values_only=True):
cells = ["" if v is None else str(v).strip().replace("|", "\\|")
for v in row]
if not any(cells):
continue
lines.append("| " + " | ".join(cells) + " |")
lines.append("")
book.close()
return Converted(
markdown=_clean("\n".join(lines)),
converter="openpyxl",
page_count=None,
)
def _from_text(data: bytes) -> Converted:
try:
text = data.decode("utf-8")
except UnicodeDecodeError:
text = data.decode("latin-1")
return Converted(markdown=_clean(text), converter="passthrough", page_count=None)
_CONVERTERS = {
".pdf": _from_pdf,
".docx": _from_docx,
".xlsx": _from_xlsx,
".xlsm": _from_xlsx,
".md": _from_text,
".txt": _from_text,
}
def convert(filename: str, data: bytes) -> Converted:
"""Convert an uploaded file to Markdown, or raise ConversionError.
Dispatch is on the extension, which is a claim the uploader made about the
file. The libraries below all fail loudly on a mismatch, so a .docx renamed
to .pdf raises rather than producing plausible rubbish.
"""
suffix = "." + filename.rsplit(".", 1)[-1].lower() if "." in filename else ""
handler = _CONVERTERS.get(suffix)
if handler is None:
raise ConversionError(
f"{suffix or 'no extension'} is not supported - "
f"expected one of {', '.join(sorted(SUPPORTED))}"
)
result = handler(data)
if len(result.markdown) < _MIN_CHARS:
# The common cause by far is a scanned document: a picture of text,
# which text extraction cannot read. Name that explicitly, because
# "conversion produced nothing" sends somebody looking for a bug in the
# upload instead of at the file.
raise ConversionError(
f"only {len(result.markdown)} characters of text came out of this "
f"file. If it is a scan or a photograph, no text can be extracted "
f"from it - it needs OCR, which this converter does not do. "
f"Nothing has been added to the library."
)
return result

789
api/documents.py Normal file
View file

@ -0,0 +1,789 @@
"""The document library screens: upload, review, approve, withdraw, restore.
Served under /documents, NOT /docs. FastAPI's Swagger UI already owns /docs and
the customer wants to keep it; two different things under one prefix with two
different access policies is the kind of thing that gets misread during a later
edit.
WHERE THIS RUNS AND WHY. These screens are served by ai-api on
api.yokogawa.tech, not by ai-web. ai-web is on ai.yokogawa.tech, which since
2026-08-28 admits only the SCADA console and passes through no Authelia at all,
so it has no identity to record. Publishers come in on api.yokogawa.tech, where
the forward-auth headers still arrive. Putting the screens where the identity
already is avoids reopening that routing question.
HOW THIS DIFFERS FROM THE DESIGN, all of it deliberate and all of it recorded
in BUILD-AI-CONTAINERS.md S14:
- No ai-docs-worker. Conversion, chunking and embedding happen inside the
request. The design puts publication behind a component with no HTTP
surface; here the same process holds both database roles, so the boundary
is enforced by db/005's trigger and by which connection each function opens,
rather than by deployment. A large upload therefore blocks its own request
rather than queueing - acceptable at this size, and the reason status
values like `scanning` and `ingesting` are passed through rather than
lingered in.
- Conversion is text extraction, not layout parsing, and cannot read a
scan. See convert.py for what that costs.
- Files are NOT moved into /datadisk/ai-docs. ai-api has no write access to
the document tree and is not getting any. The inbox is the store, and
doc_chunks.source_file points into it. The consequence to know about:
`ai-ingest --all` walks /datadisk/ai-docs and will not see anything
published this way, so the two paths must not be used on the same document.
"""
from __future__ import annotations
import hashlib
import html
import logging
import os
import re
import uuid
from datetime import date
from pathlib import Path
import psycopg
from fastapi import APIRouter, Form, HTTPException, Request, UploadFile
from fastapi.responses import HTMLResponse, RedirectResponse
import chunking
import convert
from config import settings
from identity import Actor, DEMO_GROUPS, actor, is_publisher, require_publisher
log = logging.getLogger("api.documents")
router = APIRouter(prefix="/documents", tags=["documents"])
DOC_TYPES = ("procedure", "manual", "rationalisation", "design")
# doc_actions.reason has a CHECK of length >= 10. Mirrored here so the person
# gets a sentence explaining why, instead of a database error.
MIN_REASON = 10
# Anything that is not a plain filename. The stored name is generated from the
# upload_id anyway; this only keeps the ORIGINAL name printable on screen.
_UNSAFE = re.compile(r"[^A-Za-z0-9._ -]")
# --- plumbing ---------------------------------------------------------------
def _connect(role: str) -> psycopg.Connection:
"""Open a connection as one of the two document roles.
`role` is spelled out at every call site. "uploads" may write the queue and
may set superseded = TRUE; "ingest" is the only one that may write chunks
or make a document citable again. Choosing the wrong one here is the whole
security boundary, so it is never defaulted.
"""
return psycopg.connect(settings().docs_dsn(role), connect_timeout=10)
def _who(request: Request, declared_name: str | None = None) -> Actor:
return actor(request, declared_name)
def _embed(texts: list[str]) -> list[list[float]]:
from openai import AzureOpenAI
cfg = settings()
if not (cfg.azure_openai_endpoint and cfg.azure_openai_api_key
and cfg.embed_deployment):
raise HTTPException(
status_code=503,
detail=(
"no embedding model is configured, so nothing can be published. "
"Set AZURE_OPENAI_* and EMBED_DEPLOYMENT."
),
)
client = AzureOpenAI(
azure_endpoint=cfg.azure_openai_endpoint,
api_key=cfg.azure_openai_api_key,
api_version=cfg.azure_openai_api_version,
)
vectors: list[list[float]] = []
for i in range(0, len(texts), 64):
response = client.embeddings.create(
model=cfg.embed_deployment, input=texts[i : i + 64]
)
vectors.extend(item.embedding for item in response.data)
return vectors
def _upload_dir(upload_id: str) -> Path:
return Path(settings().docs_inbox) / upload_id
# --- HTML -------------------------------------------------------------------
#
# Hand-written rather than templated. Six screens do not earn a template engine,
# a templates directory and another pinned dependency, and keeping the markup
# next to the handler means a reviewer can see what a form posts without
# opening a second file. Everything interpolated goes through e().
def e(value: object) -> str:
return html.escape("" if value is None else str(value))
def _page(title: str, body: str, who: Actor | None = None) -> HTMLResponse:
banner = ""
if who is not None and who.is_demo:
# The whole point of demo mode being visible. A person looking at this
# screen must not believe the names on it were verified.
banner = (
'<div class="warn"><b>Demo identity.</b> The name below is typed in, '
'not verified by sign-in. Every change is recorded as '
f'<code>demo:{e(who.display)}</code> so it can never be mistaken for '
'an authenticated action.</div>'
)
return HTMLResponse(f"""<!doctype html>
<html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>{e(title)} — WRPS Document Library</title>
<style>
:root {{ color-scheme: light dark; }}
body {{ font: 15px/1.5 system-ui, sans-serif; margin: 0 auto; padding: 1.5rem;
max-width: 60rem; }}
h1 {{ font-size: 1.3rem; }} h2 {{ font-size: 1.05rem; margin-top: 2rem; }}
table {{ border-collapse: collapse; width: 100%; margin: .5rem 0 1.5rem; }}
th, td {{ text-align: left; padding: .4rem .6rem; border-bottom: 1px solid #8884;
vertical-align: top; }}
th {{ font-weight: 600; font-size: .85rem; text-transform: uppercase;
letter-spacing: .03em; opacity: .7; }}
.warn {{ border-left: 4px solid #c60; background: #c6600018; padding: .6rem .9rem;
margin: 0 0 1.2rem; }}
.danger {{ border-left-color: #c00; background: #c0000018; }}
.muted {{ opacity: .65; }}
pre {{ background: #8881; padding: .8rem; overflow-x: auto; max-height: 26rem;
white-space: pre-wrap; }}
label {{ display: block; margin: .6rem 0 .15rem; font-weight: 600;
font-size: .85rem; }}
input, select, textarea {{ font: inherit; padding: .35rem; width: 100%;
max-width: 28rem; box-sizing: border-box; }}
button {{ font: inherit; padding: .4rem .9rem; margin-top: .8rem; cursor: pointer; }}
nav a {{ margin-right: 1rem; }}
code {{ font-size: .9em; }}
</style></head><body>
<nav><a href="/documents">Library</a><a href="/docs">API reference</a></nav>
<h1>{e(title)}</h1>
{banner}
{body}
</body></html>""")
def _name_field(who_default: str = "") -> str:
"""The demo identity prompt. Only rendered in demo mode."""
if settings().doc_identity_mode != "demo":
return ""
return (
'<label for="declared_name">Your name (recorded against this action)</label>'
f'<input id="declared_name" name="declared_name" required maxlength="64" '
f'value="{e(who_default)}">'
)
# --- reads ------------------------------------------------------------------
def _fetch(conn: psycopg.Connection, sql: str, args: tuple = ()) -> list[tuple]:
with conn.cursor() as cur:
cur.execute(sql, args)
return cur.fetchall()
@router.get("", response_class=HTMLResponse)
@router.get("/", response_class=HTMLResponse)
def library(request: Request) -> HTMLResponse:
"""The one screen: what is live, what is waiting, what has been withdrawn.
Readable by anyone who got through the edge. Only the actions are
restricted, and the restriction is enforced in the handlers - hiding a
button proves nothing, which is why the API is what checks.
"""
cfg = settings()
demo = cfg.doc_identity_mode == "demo"
# In demo mode there is nobody to identify until they type a name, so the
# listing must not 401. Build a placeholder purely for the banner.
who = (
Actor(stored="", display="", name=None, email=None,
groups=DEMO_GROUPS, verified=False)
if demo else _who(request)
)
with _connect("uploads") as conn:
live = _fetch(conn, """
SELECT doc_number, revision, doc_type, effective_date,
chunk_count, ingested_at
FROM live_documents ORDER BY doc_number, revision""")
queue = _fetch(conn, """
SELECT upload_id, original_filename, status, uploaded_by,
uploaded_at, error
FROM doc_uploads
WHERE status NOT IN ('published','rejected')
ORDER BY uploaded_at DESC""")
gone = _fetch(conn, """
SELECT doc_number, revision, doc_type, chunk_count,
withdrawn_at, withdrawn_by, reason
FROM withdrawn_documents ORDER BY doc_number, revision""")
rows = "".join(
f"<tr><td><code>{e(d)}</code></td><td>{e(r)}</td><td>{e(t)}</td>"
f"<td>{e(ed)}</td><td>{e(n)}</td>"
f"<td><form method=post action='/documents/withdraw'>"
f"<input type=hidden name=doc_number value='{e(d)}'>"
f"<input type=hidden name=revision value='{e(r)}'>"
f"<button>Withdraw…</button></form></td></tr>"
for d, r, t, ed, n, _ in live
) or "<tr><td colspan=6 class=muted>Nothing is published.</td></tr>"
qrows = "".join(
f"<tr><td>{e(f)}</td><td>{e(s)}</td><td>{e(u)}</td><td>{e(at)}</td>"
f"<td>{e(err) or ''}</td>"
f"<td><a href='/documents/review/{e(uid)}'>Review</a></td></tr>"
for uid, f, s, u, at, err in queue
) or "<tr><td colspan=6 class=muted>Nothing waiting.</td></tr>"
grows = "".join(
f"<tr><td><code>{e(d)}</code></td><td>{e(r)}</td><td>{e(t)}</td>"
f"<td>{e(n)}</td><td>{e(at)}</td><td>{e(by)}</td><td>{e(why)}</td>"
f"<td><form method=post action='/documents/restore'>"
f"<input type=hidden name=doc_number value='{e(d)}'>"
f"<input type=hidden name=revision value='{e(r)}'>"
f"<button>Restore…</button></form></td></tr>"
for d, r, t, n, at, by, why in gone
) or "<tr><td colspan=8 class=muted>Nothing withdrawn.</td></tr>"
return _page("Document library", f"""
<h2>Add a document</h2>
<form method=post action="/documents/upload" enctype="multipart/form-data">
<label for=file>File (PDF, Word, Excel, Markdown or text — max {cfg.max_upload_mb} MB)</label>
<input id=file type=file name=file required>
<label for=proposed_doc_type>Type</label>
<select id=proposed_doc_type name=proposed_doc_type required>
{''.join(f'<option value="{t}">{t}</option>' for t in DOC_TYPES)}
</select>
<label for=uploader_note>Note (optional) — what changed, and why you are adding it</label>
<input id=uploader_note name=uploader_note maxlength=500>
{_name_field()}
<button>Upload and convert</button>
</form>
<p class=muted>The file is converted to text and shown to you for review.
Nothing can be cited until somebody confirms its document number, revision and
effective date.</p>
<h2>Waiting for review</h2>
<table><tr><th>File<th>Status<th>Uploaded by<th>When<th>Error<th></tr>{qrows}</table>
<h2>Published — citable now</h2>
<table><tr><th>Document<th>Rev<th>Type<th>Effective<th>Chunks<th></tr>{rows}</table>
<h2>Withdrawn</h2>
<p class=muted>Not cited in any answer. The chunks and the audit trail are kept,
so "why did the assistant stop citing this?" still has an answer.</p>
<table><tr><th>Document<th>Rev<th>Type<th>Chunks<th>When<th>By<th>Reason<th></tr>{grows}</table>
""", who)
# --- upload -----------------------------------------------------------------
@router.post("/upload")
async def upload(
request: Request,
file: UploadFile,
proposed_doc_type: str = Form(...),
uploader_note: str = Form(""),
declared_name: str = Form(""),
) -> RedirectResponse:
"""Take a file, convert it, and park it for review.
Anyone who reaches this endpoint may upload - that is deliberate, and it is
the one action that is not restricted to publishers. Uploading changes
nothing an operator can see. APPROVING does, and that is what is gated.
"""
cfg = settings()
who = _who(request, declared_name)
if proposed_doc_type not in DOC_TYPES:
raise HTTPException(status_code=400, detail="unknown document type")
data = await file.read()
if not data:
raise HTTPException(status_code=400, detail="the file is empty")
if len(data) > cfg.max_upload_mb * 1024 * 1024:
raise HTTPException(
status_code=413,
detail=f"larger than the {cfg.max_upload_mb} MB limit",
)
upload_id = str(uuid.uuid4())
original = _UNSAFE.sub("_", os.path.basename(file.filename or "upload"))[:200]
folder = _upload_dir(upload_id)
folder.mkdir(parents=True, exist_ok=True)
stored = folder / original
stored.write_bytes(data)
digest = hashlib.sha256(data).hexdigest()
# Insert BEFORE converting, so a conversion that crashes still leaves a row
# naming the file and the person. A failure with no record is the one
# outcome that teaches nobody anything.
with _connect("uploads") as conn:
with conn.cursor() as cur:
cur.execute("""
INSERT INTO doc_uploads (upload_id, status, original_filename,
stored_path, content_type, size_bytes, sha256,
uploaded_by, uploaded_by_name, uploader_note,
proposed_doc_type)
VALUES (%s,'scanning',%s,%s,%s,%s,%s,%s,%s,%s,%s)""",
(upload_id, file.filename or original, str(stored),
file.content_type, len(data), digest,
who.stored, who.name, uploader_note or None, proposed_doc_type))
conn.commit()
try:
converted = convert.convert(original, data)
except convert.ConversionError as exc:
with conn.cursor() as cur:
cur.execute(
"UPDATE doc_uploads SET status='failed', error=%s "
"WHERE upload_id=%s", (str(exc), upload_id))
conn.commit()
return RedirectResponse("/documents", status_code=303)
(folder / "converted.md").write_text(converted.markdown, encoding="utf-8")
header = chunking.extract_header(converted.markdown)
with conn.cursor() as cur:
cur.execute("""
UPDATE doc_uploads
SET status='awaiting_review', page_count=%s, preview_text=%s,
detected_doc_number=%s, detected_revision=%s,
detected_effective_date=%s, error=NULL
WHERE upload_id=%s""",
(converted.page_count, converted.markdown[:20000],
header.doc_number, header.revision, header.effective_date,
upload_id))
conn.commit()
return RedirectResponse(f"/documents/review/{upload_id}", status_code=303)
# --- review -----------------------------------------------------------------
@router.get("/review/{upload_id}", response_class=HTMLResponse)
def review(request: Request, upload_id: str) -> HTMLResponse:
demo = settings().doc_identity_mode == "demo"
who = (
Actor(stored="", display="", name=None, email=None,
groups=DEMO_GROUPS, verified=False)
if demo else _who(request)
)
with _connect("uploads") as conn:
rows = _fetch(conn, """
SELECT original_filename, status, proposed_doc_type,
detected_doc_number, detected_revision,
detected_effective_date, preview_text, error, uploaded_by,
uploader_note
FROM doc_uploads WHERE upload_id = %s""", (upload_id,))
if not rows:
raise HTTPException(status_code=404, detail="no such upload")
(fname, status, dtype, dnum, drev, ddate, preview, error, by,
note) = rows[0]
live = _fetch(conn, """
SELECT revision, effective_date, chunk_count FROM live_documents
WHERE doc_number = %s ORDER BY revision""", (dnum,)) if dnum else []
if status == "published":
return _page("Already published", "<p>This upload is live. "
"<a href='/documents'>Back to the library</a>.</p>", who)
if status == "failed":
return _page("Conversion failed", f"""
<div class="warn danger"><b>{e(fname)} could not be converted.</b>
<p>{e(error)}</p></div>
<p>Nothing was added to the library. <a href="/documents">Back</a>.</p>""", who)
# What the reviewer is about to supersede, shown BEFORE they tick the box.
supersede_note = "".join(
f"<li>Revision <b>{e(r)}</b>, effective {e(d)}, {e(n)} chunks</li>"
for r, d, n in live
)
supersede_block = (
f"<div class='warn'><b>{e(dnum)} is already in the library:</b>"
f"<ul>{supersede_note}</ul>"
"Ticking supersede withdraws those revisions when this one is "
"published.</div>" if live else ""
)
return _page(f"Review — {fname}", f"""
<p class=muted>Uploaded by {e(by)}. {e(note) or ''}</p>
{supersede_block}
<h2>Converted text — this is what the assistant will read</h2>
<p class=muted>The original file is never used to answer a question. If the
text below is wrong or garbled, reject it: approving it puts this text in front
of an operator.</p>
<pre>{e(preview)}</pre>
<h2>Confirm the document</h2>
<div class="warn"><b>A wrong revision on a procedure is a safety issue, not a
data-quality one.</b> The values below were guessed by a regular expression.
Check them against the document itself.</div>
<form method=post action="/documents/review/{e(upload_id)}/approve">
<label for=confirmed_doc_type>Type</label>
<select id=confirmed_doc_type name=confirmed_doc_type required>
{''.join(f'<option value="{t}"{" selected" if t == dtype else ""}>{t}</option>'
for t in DOC_TYPES)}
</select>
<label for=confirmed_doc_number>Document number</label>
<input id=confirmed_doc_number name=confirmed_doc_number required value="{e(dnum)}">
<label for=confirmed_revision>Revision</label>
<input id=confirmed_revision name=confirmed_revision required value="{e(drev)}">
<label for=confirmed_effective_date>Effective date (YYYY-MM-DD)</label>
<input id=confirmed_effective_date name=confirmed_effective_date required
type=date value="{e(ddate)}">
<label><input type=checkbox name=supersede_previous value=yes
{"checked" if live else ""}> Withdraw the other revisions of this document</label>
<label><input type=checkbox name=reference_data_checked value=yes required>
I understand this changes only what the assistant can <b>cite</b>. Tag
metadata, alarm setpoints and the Cube models are unchanged, and every
numeric answer still comes from those.</label>
{_name_field()}
<button>Approve and publish</button>
</form>
<h2>Reject</h2>
<form method=post action="/documents/review/{e(upload_id)}/reject">
<label for=review_note>Reason (required)</label>
<textarea id=review_note name=review_note rows=2 required
minlength="{MIN_REASON}"></textarea>
{_name_field()}
<button>Reject</button>
</form>
""", who)
@router.post("/review/{upload_id}/reject")
def reject(
request: Request,
upload_id: str,
review_note: str = Form(...),
declared_name: str = Form(""),
) -> RedirectResponse:
who = require_publisher(_who(request, declared_name))
if len(review_note.strip()) < MIN_REASON:
raise HTTPException(
status_code=400,
detail=f"give a reason of at least {MIN_REASON} characters",
)
with _connect("uploads") as conn:
with conn.cursor() as cur:
cur.execute("""
UPDATE doc_uploads SET status='rejected', review_note=%s,
reviewed_by=%s, reviewed_by_name=%s, reviewed_at=now()
WHERE upload_id=%s AND status='awaiting_review'""",
(review_note.strip(), who.stored, who.name, upload_id))
if cur.rowcount == 0:
raise HTTPException(
status_code=409, detail="not awaiting review any more")
conn.commit()
return RedirectResponse("/documents", status_code=303)
@router.post("/review/{upload_id}/approve")
def approve(
request: Request,
upload_id: str,
confirmed_doc_type: str = Form(...),
confirmed_doc_number: str = Form(...),
confirmed_revision: str = Form(...),
confirmed_effective_date: date = Form(...),
reference_data_checked: str = Form(""),
supersede_previous: str = Form(""),
declared_name: str = Form(""),
) -> RedirectResponse:
"""Confirm the header, then chunk, embed and publish.
This is the only path that writes doc_chunks, and the only one that needs
the ingest role. Everything before it is reversible; this is the step that
puts text in front of an operator.
"""
who = require_publisher(_who(request, declared_name))
if confirmed_doc_type not in DOC_TYPES:
raise HTTPException(status_code=400, detail="unknown document type")
if not reference_data_checked:
# The database does not enforce this one - it is an acknowledgement,
# not a fact. Refusing here is what makes it a question rather than
# decoration.
raise HTTPException(
status_code=400,
detail="confirm you understand reference data is unchanged",
)
supersede = bool(supersede_previous)
with _connect("uploads") as conn:
rows = _fetch(conn, """
SELECT status, stored_path FROM doc_uploads WHERE upload_id=%s""",
(upload_id,))
if not rows:
raise HTTPException(status_code=404, detail="no such upload")
status, stored_path = rows[0]
if status != "awaiting_review":
raise HTTPException(
status_code=409, detail=f"upload is {status}, not awaiting review")
with conn.cursor() as cur:
cur.execute("""
UPDATE doc_uploads
SET status='approved', confirmed_doc_type=%s,
confirmed_doc_number=%s, confirmed_revision=%s,
confirmed_effective_date=%s, supersede_previous=%s,
reference_data_checked=TRUE, reviewed_by=%s,
reviewed_by_name=%s, reviewed_at=now()
WHERE upload_id=%s""",
(confirmed_doc_type, confirmed_doc_number.strip(),
confirmed_revision.strip(), confirmed_effective_date,
supersede, who.stored, who.name, upload_id))
conn.commit()
markdown = (_upload_dir(upload_id) / "converted.md").read_text(encoding="utf-8")
header = chunking.extract_header(markdown)
# --- chunk ------------------------------------------------------------
with _connect("ingest") as conn:
equipment = [r[0] for r in _fetch(conn, "SELECT equipment_id FROM equipment")]
records: list[tuple] = []
for page, title, body in chunking.split_sections(markdown):
for chunk in chunking.chunk_section(body, confirmed_doc_type):
records.append((page, title, chunk,
chunking.link_equipment(chunk, equipment)))
if not records:
with _connect("uploads") as uconn:
with uconn.cursor() as cur:
cur.execute(
"UPDATE doc_uploads SET status='failed', error=%s "
"WHERE upload_id=%s",
("the converted text produced no chunks", upload_id))
uconn.commit()
raise HTTPException(status_code=422, detail="nothing to publish")
vectors = _embed([r[2] for r in records])
superseded_count = 0
with conn.cursor() as cur:
# Replace any earlier ingest of this exact file, so re-publishing
# cannot double the chunks.
cur.execute("DELETE FROM doc_chunks WHERE source_file=%s", (stored_path,))
if supersede:
cur.execute("""
UPDATE doc_chunks SET superseded=TRUE
WHERE doc_number=%s AND revision <> %s AND superseded=FALSE""",
(confirmed_doc_number.strip(), confirmed_revision.strip()))
superseded_count = cur.rowcount
for (page, title, chunk, equip), vector in zip(records, vectors):
cur.execute("""
INSERT INTO doc_chunks (source_file, doc_type, doc_number,
revision, effective_date, superseded, equipment_id, page,
section_title, chunk_text, doc_title, authorising_role,
embedding)
VALUES (%s,%s,%s,%s,%s,FALSE,%s,%s,%s,%s,%s,%s,%s)""",
(stored_path, confirmed_doc_type,
confirmed_doc_number.strip(), confirmed_revision.strip(),
confirmed_effective_date, equip, page, title, chunk,
header.title, header.authorising_role,
# str(), not the list: pgvector takes its text input form,
# and this matches how ingest.py binds it. Passing the
# list itself fails at bind time.
str(vector)))
conn.commit()
with _connect("uploads") as conn:
with conn.cursor() as cur:
cur.execute("""
UPDATE doc_uploads SET status='published', chunk_count=%s,
superseded_count=%s, published_source_file=%s
WHERE upload_id=%s""",
(len(records), superseded_count, stored_path, upload_id))
conn.commit()
log.info("published %s rev %s: %d chunks, %d superseded, by %s",
confirmed_doc_number, confirmed_revision, len(records),
superseded_count, who.stored)
return RedirectResponse("/documents", status_code=303)
# --- withdraw and restore ---------------------------------------------------
# response_model=None: the return annotation is a UNION of two Response
# subclasses (the confirmation page, or the redirect after acting) and
# FastAPI would otherwise try to build a Pydantic response model from it
# and fail at import. Caught by test_the_router_can_be_mounted.
@router.post("/withdraw", response_class=HTMLResponse, response_model=None)
def withdraw(
request: Request,
doc_number: str = Form(...),
revision: str = Form(...),
reason: str = Form(""),
declared_name: str = Form(""),
) -> HTMLResponse | RedirectResponse:
"""Stop citing a document. Immediate, reversible, recorded.
Posted twice: the first time from the library with no reason, which renders
the confirmation form; the second with a reason, which acts. A withdrawal
without a stated reason is not something to make easy.
"""
if not reason.strip():
demo = settings().doc_identity_mode == "demo"
who = (Actor(stored="", display="", name=None, email=None,
groups=DEMO_GROUPS, verified=False)
if demo else _who(request))
return _page("Withdraw a document", f"""
<div class="warn"><b>{e(doc_number)} revision {e(revision)}</b> will stop being
cited from the next question asked. The chunks and the audit trail are kept, and
it can be restored.</div>
<form method=post action="/documents/withdraw">
<input type=hidden name=doc_number value="{e(doc_number)}">
<input type=hidden name=revision value="{e(revision)}">
<label for=reason>Why (required, at least {MIN_REASON} characters)</label>
<textarea id=reason name=reason rows=3 required minlength="{MIN_REASON}"></textarea>
{_name_field()}
<button>Withdraw</button>
</form>
<p><a href="/documents">Cancel</a></p>""", who)
who = require_publisher(_who(request, declared_name))
if len(reason.strip()) < MIN_REASON:
raise HTTPException(
status_code=400,
detail=f"give a reason of at least {MIN_REASON} characters",
)
action_id = str(uuid.uuid4())
with _connect("uploads") as conn:
with conn.cursor() as cur:
# Written BEFORE the act, completed after, so an interruption
# leaves evidence rather than silence.
cur.execute("""
INSERT INTO doc_actions (action_id, action, status, doc_number,
revision, actor, actor_name, actor_groups, reason)
VALUES (%s,'withdraw','pending',%s,%s,%s,%s,%s,%s)""",
(action_id, doc_number, revision, who.stored, who.name,
who.groups, reason.strip()))
conn.commit()
cur.execute("""
UPDATE doc_chunks SET superseded=TRUE
WHERE doc_number=%s AND revision=%s AND superseded=FALSE""",
(doc_number, revision))
affected = cur.rowcount
# No file move: ai-api has no write access to the document tree.
# The database flip is what stops citation, which is the part
# anybody is waiting for; file_moved_to stays NULL and says so.
cur.execute("""
UPDATE doc_actions SET status='complete', chunks_affected=%s,
completed_at=now() WHERE action_id=%s""",
(affected, action_id))
conn.commit()
log.info("withdrew %s rev %s: %d chunks, by %s",
doc_number, revision, affected, who.stored)
return RedirectResponse("/documents", status_code=303)
# response_model=None: the return annotation is a UNION of two Response
# subclasses (the confirmation page, or the redirect after acting) and
# FastAPI would otherwise try to build a Pydantic response model from it
# and fail at import. Caught by test_the_router_can_be_mounted.
@router.post("/restore", response_class=HTMLResponse, response_model=None)
def restore(
request: Request,
doc_number: str = Form(...),
revision: str = Form(...),
reason: str = Form(""),
declared_name: str = Form(""),
) -> HTMLResponse | RedirectResponse:
"""Make a withdrawn document citable again.
Uses the INGEST role, not the uploads one: db/005 has a trigger that refuses
to let the web role set superseded = FALSE. Anything that makes a document
citable goes through the role that writes chunks.
Refused while another revision of the same document is live - restoring the
old revision of a procedure alongside the new one is the failure this whole
project exists to avoid.
"""
if not reason.strip():
demo = settings().doc_identity_mode == "demo"
who = (Actor(stored="", display="", name=None, email=None,
groups=DEMO_GROUPS, verified=False)
if demo else _who(request))
return _page("Restore a document", f"""
<div class="warn"><b>{e(doc_number)} revision {e(revision)}</b> will become
citable again from the next question asked.</div>
<form method=post action="/documents/restore">
<input type=hidden name=doc_number value="{e(doc_number)}">
<input type=hidden name=revision value="{e(revision)}">
<label for=reason>Why (required, at least {MIN_REASON} characters)</label>
<textarea id=reason name=reason rows=3 required minlength="{MIN_REASON}"></textarea>
{_name_field()}
<button>Restore</button>
</form>
<p><a href="/documents">Cancel</a></p>""", who)
who = require_publisher(_who(request, declared_name))
if len(reason.strip()) < MIN_REASON:
raise HTTPException(
status_code=400,
detail=f"give a reason of at least {MIN_REASON} characters",
)
action_id = str(uuid.uuid4())
with _connect("ingest") as conn:
live = _fetch(conn, """
SELECT revision FROM live_documents
WHERE doc_number=%s AND revision <> %s""", (doc_number, revision))
if live:
raise HTTPException(
status_code=409,
detail=(
f"revision {live[0][0]} of {doc_number} is live. Withdraw it "
f"first - two live revisions of one document is the failure "
f"this refuses to create."
),
)
with conn.cursor() as cur:
cur.execute("""
INSERT INTO doc_actions (action_id, action, status, doc_number,
revision, actor, actor_name, actor_groups, reason)
VALUES (%s,'restore','pending',%s,%s,%s,%s,%s,%s)""",
(action_id, doc_number, revision, who.stored, who.name,
who.groups, reason.strip()))
conn.commit()
cur.execute("""
UPDATE doc_chunks SET superseded=FALSE
WHERE doc_number=%s AND revision=%s AND superseded=TRUE""",
(doc_number, revision))
affected = cur.rowcount
cur.execute("""
UPDATE doc_actions SET status='complete', chunks_affected=%s,
completed_at=now() WHERE action_id=%s""",
(affected, action_id))
conn.commit()
log.info("restored %s rev %s: %d chunks, by %s",
doc_number, revision, affected, who.stored)
return RedirectResponse("/documents", status_code=303)

169
api/identity.py Normal file
View file

@ -0,0 +1,169 @@
"""Who is acting, and whether they are allowed to.
Two questions, deliberately separate:
actor() WHO. In `authelia` mode this is Remote-User from the
forward-auth headers and nothing else. A missing header is a
401, never an anonymous fallback - getting to ai-api without
passing Authelia is not a state in which to accept a change to
the document set.
require_publisher() WHETHER. A name list today (DOC_PUBLISHERS), the AD
group AI_DocPublishers later. Empty list means nobody, and every
mutating endpoint 403s. Fail closed.
DEMO MODE. `DOC_IDENTITY_MODE=demo` takes the actor from a form field instead.
That is precisely what the design forbids - "identity comes from Authelia's
forwarded headers, never from the request body" - and it is here because the
customer asked for a demo with no password. Three things make it survivable:
1. It is off unless asked for.
2. Every screen carries a banner saying the name is unverified.
3. Rows are written as `demo:<name>` with actor_groups = 'DEMO-UNVERIFIED'.
(3) is the one that matters in a year. When real auth goes on, the audit trail
still has to distinguish a name somebody typed from a name Authelia proved. If
demo rows were written as bare names they would be indistinguishable from real
ones forever, and `doc_actions` is a table nothing can delete from - so the
ambiguity would be permanent.
"""
from __future__ import annotations
from dataclasses import dataclass
from fastapi import HTTPException, Request
from config import settings
# Authelia's forward-auth subrequest returns these; Caddy's shared `authelia`
# snippet on lin001 copies all four upstream (checked 2026-08-27). They are
# trusted ONLY because nothing outside the proxy network can reach this app.
# Any container on `proxy` could forge them - that assumption is exactly as
# strong as the no-published-ports rule.
_USER_HEADER = "Remote-User"
_NAME_HEADER = "Remote-Name"
_EMAIL_HEADER = "Remote-Email"
_GROUPS_HEADER = "Remote-Groups"
DEMO_GROUPS = "DEMO-UNVERIFIED"
@dataclass(frozen=True)
class Actor:
"""The person making a change, as it will be written to doc_actions."""
# What goes in doc_actions.actor / doc_uploads.*_by. Carries the `demo:`
# prefix in demo mode - do not strip it anywhere on the write path.
stored: str
# What to show on screen. No prefix.
display: str
name: str | None
email: str | None
groups: str
verified: bool
@property
def is_demo(self) -> bool:
return not self.verified
def _header(request: Request, key: str) -> str | None:
value = request.headers.get(key)
return value.strip() if value and value.strip() else None
def actor(request: Request, declared_name: str | None = None) -> Actor:
"""Resolve who is acting, or raise 401.
`declared_name` is the form field, and it is ignored outside demo mode -
passing it in `authelia` mode must not be able to change the recorded
actor, which is the whole point of taking identity from the headers.
"""
cfg = settings()
remote_user = _header(request, _USER_HEADER)
if cfg.doc_identity_mode != "demo":
if not remote_user:
raise HTTPException(
status_code=401,
detail=(
"no Remote-User header - this request did not pass through "
"Authelia. Refusing to record an anonymous document change."
),
)
return Actor(
stored=remote_user,
display=remote_user,
name=_header(request, _NAME_HEADER),
email=_header(request, _EMAIL_HEADER),
groups=_header(request, _GROUPS_HEADER) or "",
verified=True,
)
# --- demo mode ---------------------------------------------------------
typed = (declared_name or "").strip()
if not typed:
raise HTTPException(
status_code=400,
detail="a name is required - it is recorded against this change",
)
if len(typed) > 64:
raise HTTPException(status_code=400, detail="name too long")
# Record the Authelia identity too WHEN THERE IS ONE. On
# api.yokogawa.tech there usually is, because that hostname is still gated
# even though this mode ignores it for authorisation. It costs nothing and
# gives a cross-check: a demo row whose header disagrees with the typed
# name is worth a second look.
groups = DEMO_GROUPS
if remote_user:
groups = f"{DEMO_GROUPS}; authelia={remote_user}"
return Actor(
stored=f"demo:{typed}",
display=typed,
name=typed,
email=_header(request, _EMAIL_HEADER),
groups=groups,
verified=False,
)
def is_publisher(who: Actor) -> bool:
"""May this actor change the document set?
Compared against the DISPLAY name, not the stored one - the `demo:` prefix
is a provenance marker on the audit row, not part of anybody's identity.
Case-insensitive: "Admin" and "admin" are the same person, and a demo that
fails on capitalisation teaches nothing.
"""
allowed = {n.casefold() for n in settings().doc_publishers}
if not allowed:
return False
return who.display.casefold() in allowed
def require_publisher(who: Actor) -> Actor:
"""403 unless this actor may act. Called by EVERY mutating endpoint.
This is the check that actually enforces. The Authelia rule for the
document paths is the other one, and the UI hiding a button is neither -
the design is explicit that a hidden button proves nothing, and the gate
item for this phase is verified by calling the API directly as a
non-publisher.
"""
if not is_publisher(who):
if not settings().doc_publishers:
raise HTTPException(
status_code=403,
detail=(
"no publishers are configured, so nothing may be changed. "
"Set DOC_PUBLISHERS."
),
)
raise HTTPException(
status_code=403,
detail=f"{who.display} is not a publisher",
)
return who

7
api/main.py
View file

@ -21,6 +21,7 @@ from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel, Field from pydantic import BaseModel, Field
import agent import agent
import documents
from config import settings from config import settings
from contracts import ContractViolation from contracts import ContractViolation
from guardrails import GuardrailViolation from guardrails import GuardrailViolation
@ -95,6 +96,12 @@ def _langfuse():
return None return None
# The document library screens. Mounted under /documents, NOT /docs - FastAPI's
# Swagger UI already owns /docs and the customer keeps it. Two things under one
# prefix with two different access policies is what gets misread later.
app.include_router(documents.router)
class AskRequest(BaseModel): class AskRequest(BaseModel):
question: str = Field(min_length=3, max_length=1000) question: str = Field(min_length=3, max_length=1000)
question_class: str | None = Field( question_class: str | None = Field(

18
api/requirements-docs.txt Normal file
View file

@ -0,0 +1,18 @@
# Document upload (Phase 9). SEPARATE FROM requirements.txt ON PURPOSE.
#
# Installed in its own layer, AFTER the main one. A change here therefore does
# not invalidate the cached layer carrying fastapi, langgraph, langfuse and the
# rest - so adding or bumping a document dependency costs a few megabytes of
# pure-Python wheels instead of re-resolving the whole tree on a 2 vCPU host
# that also runs the demo plant's PLC.
#
# All four are PURE PYTHON: no compiler runs, no native extension is built.
#
# These extract text, not layout, and cannot read a scan. api/convert.py says
# what that costs and why review is what makes it acceptable. Anything heavier -
# in particular anything pulling torch - does not belong in this file; it
# belongs in an image built somewhere other than lin001.
python-multipart==0.0.20
pypdf==5.1.0
python-docx==1.1.2
openpyxl==3.1.5

217
api/tests/test_documents.py Normal file
View file

@ -0,0 +1,217 @@
"""The document-library rules that must hold without a database or a network.
Same principle as the contract tests: the rules that matter are in Python, so
they can be exercised without an API key, a database or a running host. What is
covered here is what would be expensive to discover on the host and impossible
to discover from `docker ps`.
"""
from __future__ import annotations
import pytest
from fastapi import HTTPException
import chunking
import convert
import identity
from config import Settings
class _Req:
"""The two attributes identity.actor() actually reads."""
def __init__(self, headers: dict[str, str] | None = None):
self.headers = headers or {}
def _use(monkeypatch, **kwargs):
"""Give identity.py a Settings built for this one test.
The real settings() is lru_cached off the process environment, which is no
use here - each test needs a different identity mode and publisher list.
monkeypatch restores the original at teardown, so nothing leaks between
tests and the cache is never touched.
"""
cfg = Settings(**kwargs)
monkeypatch.setattr(identity, "settings", lambda: cfg)
return cfg
# --- chunking ---------------------------------------------------------------
# Mirrored from ingest/ingest.py. If this drifts, the same document ingested by
# the two paths produces different chunks - see the header of api/chunking.py.
def test_a_numbered_step_sequence_in_a_procedure_is_never_split():
steps = "\n\n".join(f"{n}. Do the {n}th thing. " + "x" * 400
for n in range(1, 30))
assert chunking.approx_tokens(steps) > chunking.CHUNK_TOKEN_TARGET
assert len(chunking.chunk_section(steps, "procedure")) == 1
def test_the_same_oversized_text_IS_split_when_it_is_not_a_procedure():
# The refusal is scoped to procedures on purpose. A manual has no steps to
# break, and keeping it whole would just cost tokens.
prose = "\n\n".join("Some prose about the station. " + "x" * 400
for _ in range(30))
assert len(chunking.chunk_section(prose, "manual")) > 1
def test_sections_split_on_headings_and_keep_their_titles():
sections = chunking.split_sections(
"# Purpose\nWhy this exists.\n\n## Scope\nWhat it covers.\n")
assert [t for _, t, _ in sections] == ["Purpose", "Scope"]
def test_equipment_is_linked_only_when_it_is_unambiguous():
ids = ["PU-301", "PU-302"]
assert chunking.link_equipment("Isolate PU-301 before work.", ids) == "PU-301"
# Two units mentioned is not a tie to break - a chunk linked to the wrong
# pump is hidden from the pump it actually describes.
assert chunking.link_equipment("PU-301 and PU-302 share a header.", ids) is None
def test_header_extraction_is_a_proposal_and_may_find_nothing():
found = chunking.extract_header(
"Title: Wet Well Interlock Bypass\n"
"WRPS-OPS-014 Revision 3\n"
"Effective: 01/03/2026\n"
"Authorising role: Station Maintenance Supervisor\n")
assert found.doc_number == "WRPS-OPS-014"
assert found.revision == "3"
assert found.authorising_role == "Station Maintenance Supervisor"
empty = chunking.extract_header("A document with no header at all.")
assert empty.doc_number is None and empty.revision is None
# --- conversion -------------------------------------------------------------
def test_a_document_with_almost_no_extractable_text_is_refused():
"""A scan converts to nothing. Refusing is the point.
Storing it would put a blank document in front of a reviewer who might
approve it without noticing there is nothing in it.
"""
with pytest.raises(convert.ConversionError) as exc:
convert.convert("scan.md", b"# Title\n")
assert "OCR" in str(exc.value)
def test_an_unsupported_extension_is_refused_by_name():
with pytest.raises(convert.ConversionError) as exc:
convert.convert("drawing.dwg", b"x" * 5000)
assert ".dwg" in str(exc.value)
def test_plain_text_passes_through_and_keeps_its_headings():
body = "# Purpose\n\n" + "This station has three pumps. " * 40
result = convert.convert("notes.md", body.encode())
assert result.converter == "passthrough"
assert result.markdown.startswith("# Purpose")
# --- identity ---------------------------------------------------------------
def test_authelia_mode_refuses_a_request_with_no_remote_user(monkeypatch):
"""Reaching ai-api without passing Authelia is not a state in which to
accept a change to the document set."""
_use(monkeypatch, doc_identity_mode="authelia")
with pytest.raises(HTTPException) as exc:
identity.actor(_Req())
assert exc.value.status_code == 401
def test_authelia_mode_ignores_a_name_supplied_in_the_body(monkeypatch):
"""The form field must not be able to change the recorded actor. That is
the entire reason identity comes from the headers."""
_use(monkeypatch, doc_identity_mode="authelia")
who = identity.actor(_Req({"Remote-User": "cliu"}), declared_name="someone-else")
assert who.stored == "cliu"
assert who.verified is True
def test_demo_mode_marks_every_row_so_it_can_never_pass_as_authenticated(monkeypatch):
_use(monkeypatch, doc_identity_mode="demo")
who = identity.actor(_Req(), declared_name="admin")
assert who.stored == "demo:admin"
assert who.display == "admin"
assert who.groups == identity.DEMO_GROUPS
assert who.verified is False
def test_demo_mode_records_the_authelia_user_alongside_the_typed_name(monkeypatch):
"""A demo row whose header disagrees with the typed name is worth a look."""
_use(monkeypatch, doc_identity_mode="demo")
who = identity.actor(_Req({"Remote-User": "cliu"}), declared_name="admin")
assert "authelia=cliu" in who.groups
def test_demo_mode_still_requires_a_name(monkeypatch):
_use(monkeypatch, doc_identity_mode="demo")
with pytest.raises(HTTPException) as exc:
identity.actor(_Req(), declared_name=" ")
assert exc.value.status_code == 400
# --- authorisation ----------------------------------------------------------
def test_with_no_publishers_configured_nobody_may_change_anything(monkeypatch):
"""Fail closed. An empty list must not mean 'everyone'."""
_use(monkeypatch, doc_identity_mode="demo", doc_publishers=())
who = identity.actor(_Req(), declared_name="admin")
with pytest.raises(HTTPException) as exc:
identity.require_publisher(who)
assert exc.value.status_code == 403
def test_a_non_publisher_is_refused_even_though_authelia_let_them_in(monkeypatch):
_use(monkeypatch, doc_identity_mode="demo", doc_publishers=("admin",))
with pytest.raises(HTTPException) as exc:
identity.require_publisher(identity.actor(_Req(), declared_name="dan"))
assert exc.value.status_code == 403
def test_the_publisher_check_ignores_capitalisation(monkeypatch):
_use(monkeypatch, doc_identity_mode="demo", doc_publishers=("admin",))
assert identity.is_publisher(identity.actor(_Req(), declared_name="Admin"))
def test_the_demo_prefix_is_provenance_and_not_part_of_the_identity(monkeypatch):
"""`demo:admin` is admin. The prefix marks how we know, not who they are -
if it were matched, demo mode would authorise nobody."""
_use(monkeypatch, doc_identity_mode="demo", doc_publishers=("admin",))
who = identity.actor(_Req(), declared_name="admin")
assert who.stored == "demo:admin"
assert identity.is_publisher(who)
# --- the router itself ------------------------------------------------------
def test_the_router_can_be_mounted():
"""Import documents.py and mount it on a real FastAPI app.
This exists because it did not, and the first deployment crash-looped on
import: `withdraw` and `restore` return a UNION of two Response subclasses,
and FastAPI tried to build a Pydantic response model from the annotation.
Nothing in the rest of this file imports documents.py, so every unit test
passed against code that could not start.
A route-table assertion is not the point - reaching the assertion is. The
failure mode this catches is at import time.
"""
from fastapi import FastAPI
import documents
app = FastAPI()
app.include_router(documents.router)
paths = {r.path for r in app.routes}
for expected in ("/documents", "/documents/upload", "/documents/withdraw",
"/documents/restore", "/documents/review/{upload_id}"):
assert expected in paths, f"{expected} is not mounted"

24
compose/ai-compose.yml
View file

@ -144,15 +144,21 @@ services:
# gets. Deliberately not /datadisk/ai-docs: a file that has been uploaded # gets. Deliberately not /datadisk/ai-docs: a file that has been uploaded
# but not yet approved must not be visible to `ai-ingest --all`. # but not yet approved must not be visible to `ai-ingest --all`.
# #
# COMMENTED OUT UNTIL PHASE 9, on the same principle as caddy/ai-routes.caddy: # ENABLED 2026-08-28. Docker creates a missing bind source as a ROOT-OWNED
# add each piece at the phase that needs it. Docker creates a missing bind # directory and ai-api does not run as root, so the directory must exist
# source as a ROOT-OWNED directory, and ai-api does not run as root - so # with the right owner BEFORE this comes up:
# deploying this before the directory exists gives you an inbox the API # sudo install -d -o 10001 -g 10001 /datadisk/ai-docs-inbox
# cannot write to, on the growing disk of a live shared host. #
# Create it first, then uncomment: # 10001, not the 10002 this comment used to say. 10002 is the ingest
# sudo install -d -o 10002 -g 10002 /datadisk/ai-docs-inbox # image's user; there is no ai-docs-worker in this build, so the process
# volumes: # writing the inbox is ai-api's appuser at 10001. Check the Dockerfile
# - /datadisk/ai-docs-inbox:/inbox # rather than this line if it ever moves.
#
# /datadisk, not /: the root disk is 62 GB and uploads accumulate. And NOT
# /datadisk/ai-docs - a file that has been uploaded but not yet approved
# must not be visible to `ai-ingest --all`.
volumes:
- /datadisk/ai-docs-inbox:/inbox
healthcheck: healthcheck:
test: ["CMD", "python", "-m", "app_healthcheck"] test: ["CMD", "python", "-m", "app_healthcheck"]
interval: 30s interval: 30s