RAG Accuracy Depends on Document Type

Engineering realities for enterprise AI Q&A systems

In vendor demos, AI Q&A system accuracy is usually above ninety percent. That number is not inherently wrong — the question is what documents it was tested on.

Accuracy is not a single metric. It depends heavily on document structure, chunking strategy, and the form of the user's question. The same system placed in different document environments can perform very differently.

Document Type Sets the Accuracy Ceiling

In real enterprise environments, documents typically fall into one of three structural categories, each requiring a different approach.

List-type documents (e.g. product spec sheets, price lists, terms and conditions)

High information density. Every line carries independent meaning. No narrative connective tissue between items. Chunking by paragraph frequently separates related entries, so retrieval returns only part of the relevant information. For list-type documents of manageable length, the more reliable approach is injecting the full document as context rather than splitting and indexing.

Paragraph-type documents (e.g. operation manuals, compliance policies, product guides)

Organised by section, with clear topic boundaries at each paragraph. Chunking by paragraph and building a vector index generally achieves good retrieval accuracy — provided the paragraphs are clearly structured and each does not cover too many distinct topics.

Semantic-type documents (e.g. email threads, meeting notes, client correspondence)

Topic boundaries depend on semantic shifts, not formatting. Splitting by word count or line breaks routinely cuts a single discussion across multiple chunks, or merges unrelated topics into one vector. Chunking by semantic boundary carries higher engineering cost but is necessary for this document type.

All three types coexisting in a single enterprise knowledge base is the norm, not the exception. A chunking strategy optimised for one document type applied to a mixed environment typically produces a significant accuracy drop.

A Fundamental Limitation of Vector Search

A RAG system converts the user's question into a vector, finds the semantically closest chunks in the document library, and passes those to a language model to generate an answer. The underlying assumption is that the user's question is semantically similar to the relevant content in the documents.

This holds in many cases. It fails systematically on one type of query: open-ended questions. When a user asks "what should I watch out for in this contract?", the question itself has no specific semantic anchor. Vector search cannot determine which paragraphs in the document correspond to "things to watch out for" — because the document was never organised from that angle. The system may return a semantically adjacent but actually irrelevant chunk, and the language model generates an answer that sounds reasonable but is built on the wrong information. This is not a model failure — it is a structural limitation of vector retrieval on open-ended queries. Identifying which question types fall into this blind spot is part of system design, not something to address after the fact.

Email and Mixed Document Environments: Closest to Real Business

The working knowledge of many enterprises is distributed across: client email threads in Gmail; quotes, contracts, and audit reports sent as PDFs; Excel reports or Word documents received as attachments; WhatsApp message logs between departments. This combination is close to the standard reality for Hong Kong SMEs. Each source has a different structure and requires different handling.

Email, for example: Gmail provides a Google Apps Script interface for automating attachment extraction, email body parsing, and integration with backend systems. But email body text is semantic-type content, requiring the corresponding chunking approach. Attached PDFs introduce their own accuracy questions — the processing cost difference between a scanned PDF and a native PDF can be several times higher. Integrating these sources into a single queryable system requires a separate ingestion pipeline for each document type, not a single generic pipeline. If a vendor proposal describes only "an AI Q&A system" without specifying how each document type is handled, that is a gap worth pressing on.

Verify the Test Conditions Before Evaluating Accuracy

Before engaging with an AI Q&A system vendor, these four questions help determine whether the accuracy figures they cite apply to your environment:

One: what type of documents were used in the test? If the demo used cleanly structured policy documents and your actual environment is primarily email and mixed PDFs, the accuracy figures are not comparable.

Two: is the chunking strategy designed for your document types? Ask the vendor to explain what chunking logic they use for list-type, paragraph-type, and semantic-type documents separately — and how mixed environments are handled.

Three: what is covered by open-ended question testing? Request a test using your actual business questions, not the vendor's pre-selected demo questions.

Four: is the email and attachment ingestion pipeline included in the design? If your business depends heavily on Gmail correspondence, confirm the system can handle email body text, PDF attachments, and mixed attachment formats. A ninety-percent accuracy claim is only meaningful if it was measured on your documents. For the RAG vs fine-tuning decision framework, see RAG vs fine-tuning enterprise decision guide. For automating the Gmail PDF ingestion side, see Gmail PDF automation with Google Apps Script.