None of these were reachable by the tests as they stood, and all three were
silent - the screen looked correct in every case. An 8-page control philosophy
found all of them in one upload.
1. THE WHOLE DOCUMENT BECAME ONE CHUNK. pypdf emits one line per line of the
PDF and no blank lines at all: 416 lines, none blank. Section splitting looks
for Markdown headings and paragraph splitting looks for blank lines, so the
chunker was a no-op on PDF text - one 18,307-character chunk, a single
embedding vector for eight pages, and every citation reading "(untitled),
page 1". A longer document would have exceeded the embedding model's input
limit and failed to publish at all.
convert.py now recovers structure: headings from numbered and capitalised
lines, paragraphs by reflowing on line width. Heading detection is
deliberately narrow, because the dangerous direction is promoting a numbered
STEP to a heading and splitting a step sequence - so a heading must be short,
a few words, and without terminal punctuation. "1. Purpose" qualifies;
"1. Open the isolation valve and confirm zero pressure." does not.
chunking.py gains a ceiling no chunk may exceed whatever the input looks
like, falling back to line and then word boundaries. The step-sequence
refusal still holds below it and is unchanged for any realistic procedure;
past it, splitting is the lesser harm, because an embeddings call that fails
protects nobody. Two heuristics found only by running the real file:
"SCADA" and "WRPS-PRO-001" were being promoted to headings, which cut real
sections in half and re-titled the remainder with something meaningless, and
"11 August 2026" was parsing as section 11.
19 chunks now, largest 574 tokens, sections matching the document.
2. EVERY CHUNK CARRIED doc_title = "Revision". TITLE_RE used [\s:]+ for the gap
after the label, and \s includes the newline. A cover page flattens to a
label column then a value column - Title / Revision / Date - so it matched a
bare "Title" line, consumed the line break and captured the next line. Now
[ \t:]+, the same trap AUTHORISING_ROLE_RE was fixed for once already. The
document's title is now null, which is the honest answer: a citation falls
back to the section title, and a confidently wrong title falls back to
nothing. Inherited, so fixed in ingest.py too.
3. RE-PUBLISHING A DOCUMENT DUPLICATED IT. approve deleted prior chunks by
source_file, which carries the upload_id and is new on every upload -
so approving the same revision twice left 38 live chunks and the same
passage citable twice. Invisible on screen, because live_documents groups by
(doc_number, revision) and only the count moved. Now deletes by document and
revision as well, and logs how many chunks it replaced.
The two chunkers are now provably in step rather than asked to be. The header
of chunking.py claimed drift in ingest.py could not be detected from the test
suite; that was wrong, both files are on disk. The new test compares the source
of chunk_section, _split_on_lines, _split_on_words, extract_header and
approx_tokens character for character. Writing it found that one earlier edit to
ingest.py had silently not applied, leaving the two genuinely divergent, and
then that extract_header's docstring had drifted. Both fixed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
8d09c84fd0