NIST SP 800-53 Rev. 5 is 1,200+ pages of dense control language with a structure that fights every naive PDF-to-text pipeline. We learned five things while wiring it into our corpus.
1. The table of contents looks exactly like a criterion
Every TOC entry is shaped 'AC-2 Account Management ... 19'. Our first chunker treated those as control rows. Fix: skip pages flagged as 'TOC' by pypdf's outline metadata, then verify by checking that the page contains no body paragraphs longer than 200 chars.
2. Control IDs span multiple paragraphs
AC-2 has subcontrols AC-2(1) through AC-2(13). The body text describing AC-2(7) might be 4 paragraphs deep, broken across columns. Naive paragraph chunking lost the AC-2(7) header; the LLM extractor produced rows that looked correct but cited the wrong control ID. We switched to heading-rooted chunking with a forward-look that pulls every paragraph until the next heading at the same or shallower depth.
3. Withdrawn controls
Several Rev. 4 controls are 'withdrawn' in Rev. 5. The PDF still includes them with a 'WITHDRAWN' stamp. Our extractor was dutifully writing them to the corpus with full text. Fix: regex on the body for the literal string 'Withdrawn:' and skip those chunks. Logged 31 such drops on first ingest.
4. Control enhancements aren't standalone
AC-2(1) only makes sense in the context of AC-2. We started by ingesting them as separate corpus rows; the agent's retrieval would pull (1) without the parent and lose context. Now every (N) row carries a back-reference to its parent control_id and a hydrate flag the retriever uses to fetch the parent before reasoning.
5. Ingestion is idempotent or it's nothing
Re-ingesting the same PDF must be a no-op — same control_ids, same chunk hashes, no duplicate rows. We content-hash at three levels (document, chunk, criterion×chunk) so the second run does zero writes. Without this the corpus would 5x in size on every framework refresh.