None of these were reachable by the tests as they stood, and all three were
silent - the screen looked correct in every case. An 8-page control philosophy
found all of them in one upload.
1. THE WHOLE DOCUMENT BECAME ONE CHUNK. pypdf emits one line per line of the
PDF and no blank lines at all: 416 lines, none blank. Section splitting looks
for Markdown headings and paragraph splitting looks for blank lines, so the
chunker was a no-op on PDF text - one 18,307-character chunk, a single
embedding vector for eight pages, and every citation reading "(untitled),
page 1". A longer document would have exceeded the embedding model's input
limit and failed to publish at all.
convert.py now recovers structure: headings from numbered and capitalised
lines, paragraphs by reflowing on line width. Heading detection is
deliberately narrow, because the dangerous direction is promoting a numbered
STEP to a heading and splitting a step sequence - so a heading must be short,
a few words, and without terminal punctuation. "1. Purpose" qualifies;
"1. Open the isolation valve and confirm zero pressure." does not.
chunking.py gains a ceiling no chunk may exceed whatever the input looks
like, falling back to line and then word boundaries. The step-sequence
refusal still holds below it and is unchanged for any realistic procedure;
past it, splitting is the lesser harm, because an embeddings call that fails
protects nobody. Two heuristics found only by running the real file:
"SCADA" and "WRPS-PRO-001" were being promoted to headings, which cut real
sections in half and re-titled the remainder with something meaningless, and
"11 August 2026" was parsing as section 11.
19 chunks now, largest 574 tokens, sections matching the document.
2. EVERY CHUNK CARRIED doc_title = "Revision". TITLE_RE used [\s:]+ for the gap
after the label, and \s includes the newline. A cover page flattens to a
label column then a value column - Title / Revision / Date - so it matched a
bare "Title" line, consumed the line break and captured the next line. Now
[ \t:]+, the same trap AUTHORISING_ROLE_RE was fixed for once already. The
document's title is now null, which is the honest answer: a citation falls
back to the section title, and a confidently wrong title falls back to
nothing. Inherited, so fixed in ingest.py too.
3. RE-PUBLISHING A DOCUMENT DUPLICATED IT. approve deleted prior chunks by
source_file, which carries the upload_id and is new on every upload -
so approving the same revision twice left 38 live chunks and the same
passage citable twice. Invisible on screen, because live_documents groups by
(doc_number, revision) and only the count moved. Now deletes by document and
revision as well, and logs how many chunks it replaced.
The two chunkers are now provably in step rather than asked to be. The header
of chunking.py claimed drift in ingest.py could not be detected from the test
suite; that was wrong, both files are on disk. The new test compares the source
of chunk_section, _split_on_lines, _split_on_words, extract_header and
approx_tokens character for character. Writing it found that one earlier edit to
ingest.py had silently not applied, leaving the two genuinely divergent, and
then that extract_header's docstring had drifted. Both fixed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ProcedureIdentity requires a title and an authorising role, and neither was
stored anywhere. The answer writer was asked for both, read them off whatever
chunk retrieval happened to return, and returned "" whenever the header chunk
was not among them.
They belong in the row for the same reason doc_number and revision do: they
are facts about the controlled document, established once when a human
confirms the header, not something to re-derive per question from whatever
text was retrieved. Denormalised onto every chunk exactly as the existing
header fields are - ingest replaces every chunk of a source_file in one
transaction, so they cannot drift within a document.
complete() deliberately still requires only doc_number, revision and
effective_date. A missing title makes an answer less useful; a wrong revision
sends somebody to the wrong document. --assume-yes must keep refusing on the
second and tolerate the first.
controlled_copy_location is NOT in the schema. It is a site fact, identical on
every row, and the one field where an invented value sends a person to a place
that does not exist. It is CONTROLLED_COPY_LOCATION in api.env, defaulting to
a string that names who to ask.
The authorising-role pattern requires the colon: without it the lazy gap
swallowed the field name and captured "role: Station Maintenance Supervisor"
as the value, which the first run caught.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two blockers stopped Phase 3 being exercised at all before Azure OpenAI exists.
Both are lifted here, and neither weakens the header confirmation - a document
still becomes citable only after a person confirms its number, revision and
effective date.
- parse_markdown() for .md, .markdown and .txt. Docling earns its place on a
PDF: it recovers structure that is not in the bytes, and that structure is
what makes section-boundary chunking possible instead of token-count
splitting that cuts step sequences in half. A Markdown file already
contains "## 2. Prerequisites". Running a document layout model over it
buys nothing and costs the entire torch stack.
It also unblocks the image. docling==2.15.1 does not resolve on
python:3.12-slim: pip backtracks through docling_ibm_models releases for
twenty minutes and exits 2. That is a real problem for Phase 3 and it is
NOT fixed here - PDFs still need Docling and the ai-ingest image still will
not build. Markdown ingestion runs from the ai-api image meanwhile.
Page is 1 for these, because a Markdown file has no pages. A citation to
one carries a section title and no meaningful page number, which is honest.
- --no-embed inserts chunks with a NULL embedding and makes no API call, so
the retrieval path can be exercised before an embeddings deployment exists.
NULL, not a zero vector: a zero vector is a point in the space, it ranks
against real queries, and it would surface as a plausible hit for anything
asked. NULL returns no similarity at all.
These chunks are invisible to vector search and findable only lexically.
The flag warns about that four times on the way past, with the SQL to clear
them, because a database half full of unembeddable chunks looks exactly
like working retrieval right up until the question that matters returns
nothing.
Before real ingestion: DELETE FROM doc_chunks WHERE embedding IS NULL;
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ingest_file() inserted every chunk with superseded = FALSE, so replacing a
document's chunks reset its withdrawal. One `ai-ingest --all` made every
superseded revision citable again - including the old revision of a procedure -
silently, and only after a bulk run, so a supersede survived exactly until the
next full ingest.
- ingest_file() reads the existing state with superseded_state() before the
delete and carries it through the insert. A withdrawn document is
re-ingested as withdrawn, and logs that it did so.
- --all skips withdrawn documents, so a bulk run does not spend an embeddings
call on a document that will not be cited either way. --include-superseded
overrides it; the chunks still come back withdrawn.
- --restore DOC_NUMBER REVISION is the counterpart to --supersede, refusing
while another revision of the same document is live. Without it the
conservative default would be a dead end - mark_superseded() only ever sets
TRUE, so there was no way back.
- Rule 5 in the module docstring, beside the other four.
Corrects a claim in the Phase 9 design: 16.10 said moving the withdrawn file
out of /datadisk/ai-docs "is not tidying" because otherwise --all resurrects it.
That was true when written and is not now. The guard belongs in the ingest code,
because a rule that depends on somebody remembering to move a file is not a
rule. The move stays as archival housekeeping and is documented as such in
16.10, db/005_doc_actions.sql and README.md.
Phase 3's gate gains the proof: supersede a revision, run --all again, ask the
question that used to cite it.
Not executed anywhere - no Postgres or Docker on this machine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ai-ingest built its DSN from PGUSER/PGPASSWORD and takes its environment from
~/ai/api.env, where PGUSER=agent_ro - SELECT and nothing else, deliberately,
because it is what the answer path runs as. So
docker compose -f ~/ai-compose.yml run --rm ai-ingest --all
connected as a role that cannot INSERT INTO doc_chunks, and Phase 3 was
unrunnable exactly as the README documents it. Nothing had reached Phase 3 yet,
so nobody had hit it.
The failure would also have landed at the worst possible moment: at the final
INSERT, after the Docling parse, after a person had typed the header
confirmations for every file, and after a billed embeddings call - with a
permission error naming no cause.
- ingest_rw moves to 003_roles.sql, at Phase 1 with the other roles. It is
not a Phase 9 concept; ingestion has needed a writing role since Phase 3
and never had one. 004 keeps only its grants on the upload queue, and its
idempotent role creation so it still applies to an older database.
- ingest.py connects through INGEST_DB_USER / INGEST_DB_PASSWORD, falling
back to PGUSER only for a local shell where one pair is set.
- require_write_access() checks INSERT, UPDATE and DELETE on doc_chunks
before anything is parsed or embedded, and fails with the fix in the
message. Falling back to PGUSER cannot smuggle agent_ro past it.
- Keyword connection parameters rather than a URL: a generated password
containing @ or / breaks a DSN string silently.
- A missing doc_chunks now says "apply 001_schema.sql" instead of raising
UndefinedTable.
Phase 1's gate gains the check that would have caught this: ingest_rw must be
able to write doc_chunks. An ingestion role that cannot write is the same class
of failure as an API role that can - it just surfaces two phases later.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Build spec and host brief carried in from C:\Claude and WRPS/02-env; the
plant model (equipment, tags, alarm bitmask, enums, unit conversions) is
derived from WRPS/04-plc/register-map.csv, WRPS/05-scada/modbus/scada-points.csv
and WRPS-CTL-003.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
