Two blockers stopped Phase 3 being exercised at all before Azure OpenAI exists.
Both are lifted here, and neither weakens the header confirmation - a document
still becomes citable only after a person confirms its number, revision and
effective date.
- parse_markdown() for .md, .markdown and .txt. Docling earns its place on a
PDF: it recovers structure that is not in the bytes, and that structure is
what makes section-boundary chunking possible instead of token-count
splitting that cuts step sequences in half. A Markdown file already
contains "## 2. Prerequisites". Running a document layout model over it
buys nothing and costs the entire torch stack.
It also unblocks the image. docling==2.15.1 does not resolve on
python:3.12-slim: pip backtracks through docling_ibm_models releases for
twenty minutes and exits 2. That is a real problem for Phase 3 and it is
NOT fixed here - PDFs still need Docling and the ai-ingest image still will
not build. Markdown ingestion runs from the ai-api image meanwhile.
Page is 1 for these, because a Markdown file has no pages. A citation to
one carries a section title and no meaningful page number, which is honest.
- --no-embed inserts chunks with a NULL embedding and makes no API call, so
the retrieval path can be exercised before an embeddings deployment exists.
NULL, not a zero vector: a zero vector is a point in the space, it ranks
against real queries, and it would surface as a plausible hit for anything
asked. NULL returns no similarity at all.
These chunks are invisible to vector search and findable only lexically.
The flag warns about that four times on the way past, with the SQL to clear
them, because a database half full of unembeddable chunks looks exactly
like working retrieval right up until the question that matters returns
nothing.
Before real ingestion: DELETE FROM doc_chunks WHERE embedding IS NULL;
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
8343e4f0ac