Phase 9's operator path, built ahead of Phase 8 at the customer's direction and live at api.yokogawa.tech/documents. Upload, convert, review, approve, withdraw and restore. The pool screen is explicitly out of scope. Served by ai-api rather than ai-web, and mounted at /documents rather than /docs. ai.yokogawa.tech is SCADA-only since 2026-08-28 and passes through no Authelia, so it has no identity to record; publishers arrive on api.yokogawa.tech where the forward-auth headers still do. /docs stays with Swagger, which the customer is keeping - two things under one prefix with two different access policies is what gets misread during a later edit. Conversion is text extraction, not document parsing: pypdf, python-docx and openpyxl. Docling would be better at this and pulls torch, which lin001 has neither the memory to install nor the business running next to the demo plant's PLC. The cost is real - no layout, no table structure, and a scan cannot be read at all, so it is refused rather than stored empty. It is acceptable only because the converted text is shown to a person before the document can be cited, which is the same safety net the design already required for the header. convert.py is the one file to change if that stops being true. Chunking is mirrored from ingest.py rather than shared, because the two live in different images. They must stay identical: if they drift, the same document chunks differently depending on who loaded it, and the assistant answers or fails to answer depending on that. The step-sequence rule is locked by a test. Identity is self-asserted for the demo - the actor is typed on the form, which section 16 forbids, and the publisher list is one name with no password. Rows are written as `demo:<name>` with actor_groups = 'DEMO-UNVERIFIED' so that when real auth goes on, a name somebody typed stays tellable from a name Authelia proved. doc_actions cannot be deleted from, so an ambiguity there would be permanent. Two rules the code enforces rather than documents: uploading is open to anyone who reaches the page, because uploading changes nothing an operator can see - approving does, and that is what is gated; and an empty publisher list means nobody, not everybody. Verified on the host end to end: withdraw as a non-publisher 403s, with a short reason 400s, and as admin flips 5 chunks and writes a complete audit row; restore puts them back and keeps both rows. The corpus is unchanged afterwards. Requirements are split so the document dependencies install in their own layer - a change there costs four small wheels instead of re-resolving fastapi, langgraph and langfuse on a 2 vCPU shared host. The five divergences from section 16 are recorded in section 14. The one with teeth: files published through the UI stay in the inbox, so `ai-ingest --all` cannot see them and the two paths must not be used on the same document. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
217 lines
8.4 KiB
Python
217 lines
8.4 KiB
Python
"""The document-library rules that must hold without a database or a network.
|
|
|
|
Same principle as the contract tests: the rules that matter are in Python, so
|
|
they can be exercised without an API key, a database or a running host. What is
|
|
covered here is what would be expensive to discover on the host and impossible
|
|
to discover from `docker ps`.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
from fastapi import HTTPException
|
|
|
|
import chunking
|
|
import convert
|
|
import identity
|
|
from config import Settings
|
|
|
|
|
|
class _Req:
|
|
"""The two attributes identity.actor() actually reads."""
|
|
|
|
def __init__(self, headers: dict[str, str] | None = None):
|
|
self.headers = headers or {}
|
|
|
|
|
|
def _use(monkeypatch, **kwargs):
|
|
"""Give identity.py a Settings built for this one test.
|
|
|
|
The real settings() is lru_cached off the process environment, which is no
|
|
use here - each test needs a different identity mode and publisher list.
|
|
monkeypatch restores the original at teardown, so nothing leaks between
|
|
tests and the cache is never touched.
|
|
"""
|
|
cfg = Settings(**kwargs)
|
|
monkeypatch.setattr(identity, "settings", lambda: cfg)
|
|
return cfg
|
|
|
|
|
|
# --- chunking ---------------------------------------------------------------
|
|
# Mirrored from ingest/ingest.py. If this drifts, the same document ingested by
|
|
# the two paths produces different chunks - see the header of api/chunking.py.
|
|
|
|
|
|
def test_a_numbered_step_sequence_in_a_procedure_is_never_split():
|
|
steps = "\n\n".join(f"{n}. Do the {n}th thing. " + "x" * 400
|
|
for n in range(1, 30))
|
|
assert chunking.approx_tokens(steps) > chunking.CHUNK_TOKEN_TARGET
|
|
assert len(chunking.chunk_section(steps, "procedure")) == 1
|
|
|
|
|
|
def test_the_same_oversized_text_IS_split_when_it_is_not_a_procedure():
|
|
# The refusal is scoped to procedures on purpose. A manual has no steps to
|
|
# break, and keeping it whole would just cost tokens.
|
|
prose = "\n\n".join("Some prose about the station. " + "x" * 400
|
|
for _ in range(30))
|
|
assert len(chunking.chunk_section(prose, "manual")) > 1
|
|
|
|
|
|
def test_sections_split_on_headings_and_keep_their_titles():
|
|
sections = chunking.split_sections(
|
|
"# Purpose\nWhy this exists.\n\n## Scope\nWhat it covers.\n")
|
|
assert [t for _, t, _ in sections] == ["Purpose", "Scope"]
|
|
|
|
|
|
def test_equipment_is_linked_only_when_it_is_unambiguous():
|
|
ids = ["PU-301", "PU-302"]
|
|
assert chunking.link_equipment("Isolate PU-301 before work.", ids) == "PU-301"
|
|
# Two units mentioned is not a tie to break - a chunk linked to the wrong
|
|
# pump is hidden from the pump it actually describes.
|
|
assert chunking.link_equipment("PU-301 and PU-302 share a header.", ids) is None
|
|
|
|
|
|
def test_header_extraction_is_a_proposal_and_may_find_nothing():
|
|
found = chunking.extract_header(
|
|
"Title: Wet Well Interlock Bypass\n"
|
|
"WRPS-OPS-014 Revision 3\n"
|
|
"Effective: 01/03/2026\n"
|
|
"Authorising role: Station Maintenance Supervisor\n")
|
|
assert found.doc_number == "WRPS-OPS-014"
|
|
assert found.revision == "3"
|
|
assert found.authorising_role == "Station Maintenance Supervisor"
|
|
|
|
empty = chunking.extract_header("A document with no header at all.")
|
|
assert empty.doc_number is None and empty.revision is None
|
|
|
|
|
|
# --- conversion -------------------------------------------------------------
|
|
|
|
|
|
def test_a_document_with_almost_no_extractable_text_is_refused():
|
|
"""A scan converts to nothing. Refusing is the point.
|
|
|
|
Storing it would put a blank document in front of a reviewer who might
|
|
approve it without noticing there is nothing in it.
|
|
"""
|
|
with pytest.raises(convert.ConversionError) as exc:
|
|
convert.convert("scan.md", b"# Title\n")
|
|
assert "OCR" in str(exc.value)
|
|
|
|
|
|
def test_an_unsupported_extension_is_refused_by_name():
|
|
with pytest.raises(convert.ConversionError) as exc:
|
|
convert.convert("drawing.dwg", b"x" * 5000)
|
|
assert ".dwg" in str(exc.value)
|
|
|
|
|
|
def test_plain_text_passes_through_and_keeps_its_headings():
|
|
body = "# Purpose\n\n" + "This station has three pumps. " * 40
|
|
result = convert.convert("notes.md", body.encode())
|
|
assert result.converter == "passthrough"
|
|
assert result.markdown.startswith("# Purpose")
|
|
|
|
|
|
# --- identity ---------------------------------------------------------------
|
|
|
|
|
|
def test_authelia_mode_refuses_a_request_with_no_remote_user(monkeypatch):
|
|
"""Reaching ai-api without passing Authelia is not a state in which to
|
|
accept a change to the document set."""
|
|
_use(monkeypatch, doc_identity_mode="authelia")
|
|
with pytest.raises(HTTPException) as exc:
|
|
identity.actor(_Req())
|
|
assert exc.value.status_code == 401
|
|
|
|
|
|
def test_authelia_mode_ignores_a_name_supplied_in_the_body(monkeypatch):
|
|
"""The form field must not be able to change the recorded actor. That is
|
|
the entire reason identity comes from the headers."""
|
|
_use(monkeypatch, doc_identity_mode="authelia")
|
|
who = identity.actor(_Req({"Remote-User": "cliu"}), declared_name="someone-else")
|
|
assert who.stored == "cliu"
|
|
assert who.verified is True
|
|
|
|
|
|
def test_demo_mode_marks_every_row_so_it_can_never_pass_as_authenticated(monkeypatch):
|
|
_use(monkeypatch, doc_identity_mode="demo")
|
|
who = identity.actor(_Req(), declared_name="admin")
|
|
assert who.stored == "demo:admin"
|
|
assert who.display == "admin"
|
|
assert who.groups == identity.DEMO_GROUPS
|
|
assert who.verified is False
|
|
|
|
|
|
def test_demo_mode_records_the_authelia_user_alongside_the_typed_name(monkeypatch):
|
|
"""A demo row whose header disagrees with the typed name is worth a look."""
|
|
_use(monkeypatch, doc_identity_mode="demo")
|
|
who = identity.actor(_Req({"Remote-User": "cliu"}), declared_name="admin")
|
|
assert "authelia=cliu" in who.groups
|
|
|
|
|
|
def test_demo_mode_still_requires_a_name(monkeypatch):
|
|
_use(monkeypatch, doc_identity_mode="demo")
|
|
with pytest.raises(HTTPException) as exc:
|
|
identity.actor(_Req(), declared_name=" ")
|
|
assert exc.value.status_code == 400
|
|
|
|
|
|
# --- authorisation ----------------------------------------------------------
|
|
|
|
|
|
def test_with_no_publishers_configured_nobody_may_change_anything(monkeypatch):
|
|
"""Fail closed. An empty list must not mean 'everyone'."""
|
|
_use(monkeypatch, doc_identity_mode="demo", doc_publishers=())
|
|
who = identity.actor(_Req(), declared_name="admin")
|
|
with pytest.raises(HTTPException) as exc:
|
|
identity.require_publisher(who)
|
|
assert exc.value.status_code == 403
|
|
|
|
|
|
def test_a_non_publisher_is_refused_even_though_authelia_let_them_in(monkeypatch):
|
|
_use(monkeypatch, doc_identity_mode="demo", doc_publishers=("admin",))
|
|
with pytest.raises(HTTPException) as exc:
|
|
identity.require_publisher(identity.actor(_Req(), declared_name="dan"))
|
|
assert exc.value.status_code == 403
|
|
|
|
|
|
def test_the_publisher_check_ignores_capitalisation(monkeypatch):
|
|
_use(monkeypatch, doc_identity_mode="demo", doc_publishers=("admin",))
|
|
assert identity.is_publisher(identity.actor(_Req(), declared_name="Admin"))
|
|
|
|
|
|
def test_the_demo_prefix_is_provenance_and_not_part_of_the_identity(monkeypatch):
|
|
"""`demo:admin` is admin. The prefix marks how we know, not who they are -
|
|
if it were matched, demo mode would authorise nobody."""
|
|
_use(monkeypatch, doc_identity_mode="demo", doc_publishers=("admin",))
|
|
who = identity.actor(_Req(), declared_name="admin")
|
|
assert who.stored == "demo:admin"
|
|
assert identity.is_publisher(who)
|
|
|
|
|
|
# --- the router itself ------------------------------------------------------
|
|
|
|
|
|
def test_the_router_can_be_mounted():
|
|
"""Import documents.py and mount it on a real FastAPI app.
|
|
|
|
This exists because it did not, and the first deployment crash-looped on
|
|
import: `withdraw` and `restore` return a UNION of two Response subclasses,
|
|
and FastAPI tried to build a Pydantic response model from the annotation.
|
|
Nothing in the rest of this file imports documents.py, so every unit test
|
|
passed against code that could not start.
|
|
|
|
A route-table assertion is not the point - reaching the assertion is. The
|
|
failure mode this catches is at import time.
|
|
"""
|
|
from fastapi import FastAPI
|
|
|
|
import documents
|
|
|
|
app = FastAPI()
|
|
app.include_router(documents.router)
|
|
|
|
paths = {r.path for r in app.routes}
|
|
for expected in ("/documents", "/documents/upload", "/documents/withdraw",
|
|
"/documents/restore", "/documents/review/{upload_id}"):
|
|
assert expected in paths, f"{expected} is not mounted"
|