yau-plant-assistant/api/tests/test_documents.py
Claude fd85e62ebf Add the document library screens: upload, review, withdraw, restore
Phase 9's operator path, built ahead of Phase 8 at the customer's direction and
live at api.yokogawa.tech/documents. Upload, convert, review, approve, withdraw
and restore. The pool screen is explicitly out of scope.

Served by ai-api rather than ai-web, and mounted at /documents rather than
/docs. ai.yokogawa.tech is SCADA-only since 2026-08-28 and passes through no
Authelia, so it has no identity to record; publishers arrive on
api.yokogawa.tech where the forward-auth headers still do. /docs stays with
Swagger, which the customer is keeping - two things under one prefix with two
different access policies is what gets misread during a later edit.

Conversion is text extraction, not document parsing: pypdf, python-docx and
openpyxl. Docling would be better at this and pulls torch, which lin001 has
neither the memory to install nor the business running next to the demo plant's
PLC. The cost is real - no layout, no table structure, and a scan cannot be read
at all, so it is refused rather than stored empty. It is acceptable only because
the converted text is shown to a person before the document can be cited, which
is the same safety net the design already required for the header. convert.py is
the one file to change if that stops being true.

Chunking is mirrored from ingest.py rather than shared, because the two live in
different images. They must stay identical: if they drift, the same document
chunks differently depending on who loaded it, and the assistant answers or
fails to answer depending on that. The step-sequence rule is locked by a test.

Identity is self-asserted for the demo - the actor is typed on the form, which
section 16 forbids, and the publisher list is one name with no password. Rows are
written as `demo:<name>` with actor_groups = 'DEMO-UNVERIFIED' so that when real
auth goes on, a name somebody typed stays tellable from a name Authelia proved.
doc_actions cannot be deleted from, so an ambiguity there would be permanent.

Two rules the code enforces rather than documents: uploading is open to anyone
who reaches the page, because uploading changes nothing an operator can see -
approving does, and that is what is gated; and an empty publisher list means
nobody, not everybody.

Verified on the host end to end: withdraw as a non-publisher 403s, with a short
reason 400s, and as admin flips 5 chunks and writes a complete audit row;
restore puts them back and keeps both rows. The corpus is unchanged afterwards.

Requirements are split so the document dependencies install in their own layer -
a change there costs four small wheels instead of re-resolving fastapi,
langgraph and langfuse on a 2 vCPU shared host.

The five divergences from section 16 are recorded in section 14. The one with
teeth: files published through the UI stay in the inbox, so `ai-ingest --all`
cannot see them and the two paths must not be used on the same document.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-28 14:13:17 +10:00

217 lines
8.4 KiB
Python

"""The document-library rules that must hold without a database or a network.
Same principle as the contract tests: the rules that matter are in Python, so
they can be exercised without an API key, a database or a running host. What is
covered here is what would be expensive to discover on the host and impossible
to discover from `docker ps`.
"""
from __future__ import annotations
import pytest
from fastapi import HTTPException
import chunking
import convert
import identity
from config import Settings
class _Req:
"""The two attributes identity.actor() actually reads."""
def __init__(self, headers: dict[str, str] | None = None):
self.headers = headers or {}
def _use(monkeypatch, **kwargs):
"""Give identity.py a Settings built for this one test.
The real settings() is lru_cached off the process environment, which is no
use here - each test needs a different identity mode and publisher list.
monkeypatch restores the original at teardown, so nothing leaks between
tests and the cache is never touched.
"""
cfg = Settings(**kwargs)
monkeypatch.setattr(identity, "settings", lambda: cfg)
return cfg
# --- chunking ---------------------------------------------------------------
# Mirrored from ingest/ingest.py. If this drifts, the same document ingested by
# the two paths produces different chunks - see the header of api/chunking.py.
def test_a_numbered_step_sequence_in_a_procedure_is_never_split():
steps = "\n\n".join(f"{n}. Do the {n}th thing. " + "x" * 400
for n in range(1, 30))
assert chunking.approx_tokens(steps) > chunking.CHUNK_TOKEN_TARGET
assert len(chunking.chunk_section(steps, "procedure")) == 1
def test_the_same_oversized_text_IS_split_when_it_is_not_a_procedure():
# The refusal is scoped to procedures on purpose. A manual has no steps to
# break, and keeping it whole would just cost tokens.
prose = "\n\n".join("Some prose about the station. " + "x" * 400
for _ in range(30))
assert len(chunking.chunk_section(prose, "manual")) > 1
def test_sections_split_on_headings_and_keep_their_titles():
sections = chunking.split_sections(
"# Purpose\nWhy this exists.\n\n## Scope\nWhat it covers.\n")
assert [t for _, t, _ in sections] == ["Purpose", "Scope"]
def test_equipment_is_linked_only_when_it_is_unambiguous():
ids = ["PU-301", "PU-302"]
assert chunking.link_equipment("Isolate PU-301 before work.", ids) == "PU-301"
# Two units mentioned is not a tie to break - a chunk linked to the wrong
# pump is hidden from the pump it actually describes.
assert chunking.link_equipment("PU-301 and PU-302 share a header.", ids) is None
def test_header_extraction_is_a_proposal_and_may_find_nothing():
found = chunking.extract_header(
"Title: Wet Well Interlock Bypass\n"
"WRPS-OPS-014 Revision 3\n"
"Effective: 01/03/2026\n"
"Authorising role: Station Maintenance Supervisor\n")
assert found.doc_number == "WRPS-OPS-014"
assert found.revision == "3"
assert found.authorising_role == "Station Maintenance Supervisor"
empty = chunking.extract_header("A document with no header at all.")
assert empty.doc_number is None and empty.revision is None
# --- conversion -------------------------------------------------------------
def test_a_document_with_almost_no_extractable_text_is_refused():
"""A scan converts to nothing. Refusing is the point.
Storing it would put a blank document in front of a reviewer who might
approve it without noticing there is nothing in it.
"""
with pytest.raises(convert.ConversionError) as exc:
convert.convert("scan.md", b"# Title\n")
assert "OCR" in str(exc.value)
def test_an_unsupported_extension_is_refused_by_name():
with pytest.raises(convert.ConversionError) as exc:
convert.convert("drawing.dwg", b"x" * 5000)
assert ".dwg" in str(exc.value)
def test_plain_text_passes_through_and_keeps_its_headings():
body = "# Purpose\n\n" + "This station has three pumps. " * 40
result = convert.convert("notes.md", body.encode())
assert result.converter == "passthrough"
assert result.markdown.startswith("# Purpose")
# --- identity ---------------------------------------------------------------
def test_authelia_mode_refuses_a_request_with_no_remote_user(monkeypatch):
"""Reaching ai-api without passing Authelia is not a state in which to
accept a change to the document set."""
_use(monkeypatch, doc_identity_mode="authelia")
with pytest.raises(HTTPException) as exc:
identity.actor(_Req())
assert exc.value.status_code == 401
def test_authelia_mode_ignores_a_name_supplied_in_the_body(monkeypatch):
"""The form field must not be able to change the recorded actor. That is
the entire reason identity comes from the headers."""
_use(monkeypatch, doc_identity_mode="authelia")
who = identity.actor(_Req({"Remote-User": "cliu"}), declared_name="someone-else")
assert who.stored == "cliu"
assert who.verified is True
def test_demo_mode_marks_every_row_so_it_can_never_pass_as_authenticated(monkeypatch):
_use(monkeypatch, doc_identity_mode="demo")
who = identity.actor(_Req(), declared_name="admin")
assert who.stored == "demo:admin"
assert who.display == "admin"
assert who.groups == identity.DEMO_GROUPS
assert who.verified is False
def test_demo_mode_records_the_authelia_user_alongside_the_typed_name(monkeypatch):
"""A demo row whose header disagrees with the typed name is worth a look."""
_use(monkeypatch, doc_identity_mode="demo")
who = identity.actor(_Req({"Remote-User": "cliu"}), declared_name="admin")
assert "authelia=cliu" in who.groups
def test_demo_mode_still_requires_a_name(monkeypatch):
_use(monkeypatch, doc_identity_mode="demo")
with pytest.raises(HTTPException) as exc:
identity.actor(_Req(), declared_name=" ")
assert exc.value.status_code == 400
# --- authorisation ----------------------------------------------------------
def test_with_no_publishers_configured_nobody_may_change_anything(monkeypatch):
"""Fail closed. An empty list must not mean 'everyone'."""
_use(monkeypatch, doc_identity_mode="demo", doc_publishers=())
who = identity.actor(_Req(), declared_name="admin")
with pytest.raises(HTTPException) as exc:
identity.require_publisher(who)
assert exc.value.status_code == 403
def test_a_non_publisher_is_refused_even_though_authelia_let_them_in(monkeypatch):
_use(monkeypatch, doc_identity_mode="demo", doc_publishers=("admin",))
with pytest.raises(HTTPException) as exc:
identity.require_publisher(identity.actor(_Req(), declared_name="dan"))
assert exc.value.status_code == 403
def test_the_publisher_check_ignores_capitalisation(monkeypatch):
_use(monkeypatch, doc_identity_mode="demo", doc_publishers=("admin",))
assert identity.is_publisher(identity.actor(_Req(), declared_name="Admin"))
def test_the_demo_prefix_is_provenance_and_not_part_of_the_identity(monkeypatch):
"""`demo:admin` is admin. The prefix marks how we know, not who they are -
if it were matched, demo mode would authorise nobody."""
_use(monkeypatch, doc_identity_mode="demo", doc_publishers=("admin",))
who = identity.actor(_Req(), declared_name="admin")
assert who.stored == "demo:admin"
assert identity.is_publisher(who)
# --- the router itself ------------------------------------------------------
def test_the_router_can_be_mounted():
"""Import documents.py and mount it on a real FastAPI app.
This exists because it did not, and the first deployment crash-looped on
import: `withdraw` and `restore` return a UNION of two Response subclasses,
and FastAPI tried to build a Pydantic response model from the annotation.
Nothing in the rest of this file imports documents.py, so every unit test
passed against code that could not start.
A route-table assertion is not the point - reaching the assertion is. The
failure mode this catches is at import time.
"""
from fastapi import FastAPI
import documents
app = FastAPI()
app.include_router(documents.router)
paths = {r.path for r in app.routes}
for expected in ("/documents", "/documents/upload", "/documents/withdraw",
"/documents/restore", "/documents/review/{upload_id}"):
assert expected in paths, f"{expected} is not mounted"