Vectorless RAG

Chunking is a lossy compression of document structure. A 10-K has a hierarchy — items, sub-items, notes — that carries real meaning, and embedding-based retrieval throws it away, then tries to reconstruct context by pulling more chunks. I wanted to test the opposite bet: keep the structure, let the model reason over it, and retrieve whole sections instead of fragments.

SEC HTML converts to structured Markdown, then to a hierarchical JSON tree where every node carries a title, a summary and its content. At query time Claude reads only the tree of summaries — a table of contents, effectively — and reasons about which sections would contain the answer. Only those 1–3 sections are fetched, in full, with their context intact. No embeddings are computed at any point.

It is slow. Nine to twelve seconds against sub-second for embedding retrieval, because reasoning over the tree is an LLM call on the critical path rather than a vector lookup. It also costs more per query and scales badly with corpus size — the tree has to fit in context, which is fine for six filings and would not be fine for six thousand. This is a technique with a real envelope, and outside that envelope conventional RAG wins. That is the finding, and it is worth as much as if it had won outright.

Retrieval precision is high where it works: revenue queries return exact figures with correct year-over-year comparisons, and the gross margin breakdown correctly separates product (36.8%) from services (75.4%). Answers cite specific items and line numbers, because whole sections retain the structure that makes citation possible.