How to Improve RAG Chunking Quality: Preprocessing Strategies Preserving Document Hierarchical Structure

Low RAG (Retrieval-Augmented Generation) chunking quality can lead to incorrect answers when documents are uploaded.

While it may seem that documents are organized into searchable units through parsing and chunking, actual results often show titles separated from the body text and table headers split from their values into different chunks.

When chunking occurs without the document’s hierarchical structure, the retrieval stage may fetch chunks irrelevant to the answer or miss the chunk containing the correct information. In this post, we identify the causes of poor chunking by type and summarize how to improve chunking quality through RAG preprocessing that preserves document structure.

Why RAG Chunking Quality Determines Answer Accuracy

Chunking is a preprocessing task that divides long documents into small units for retrieval, and how documents are divided at this stage directly impacts the accuracy of RAG (Retrieval-Augmented Generation) answers.

RAG works by searching for content related to a question within documents and using it as the basis for Large Language Model (LLM) answers. Since a chunk is the minimum unit of retrieval, chunk quality directly affects retrieval quality and answer accuracy. Chunks must be divided into semantic units to find the evidence matching the question, ensuring the context of the answer remains stable.

If chunks are too large, multiple topics mix, increasing retrieval noise; if they are too small, the surrounding context is lost. Ultimately, the core of chunking lies not in simply cutting documents to a certain length, but in dividing them into units where meaning is preserved. If context is broken at this stage, it is difficult to fully compensate for retrieval quality even by adjusting embeddings or vector databases.

Causes of Poor RAG Chunking Quality: Fixed-Size Chunking and Ignoring Document Structure

A primary cause of low RAG chunking quality is dividing text without reflecting the document structure.

Typical examples include fixed-size chunking based only on character or token counts, splitting that misses the hierarchical relationship between titles and body text, and cases where row-column relationships in tables or parent-child relationships in lists are broken. Ultimately, treating a document as sequential text rather than content with a semantic structure can result in chunks that lack the necessary context during the retrieval stage.

Why Fixed-Size Chunking Breaks Context

Fixed-size chunking is the simplest method that cuts documents into set lengths based only on character or token counts. While easy to implement, it does not consider semantic units, causing sentences or tables to be cut in the middle and information to be scattered across two chunks.

For instance, mechanical splitting at set character intervals can separate a single paragraph into multiple chunks regardless of semantic units, or include table headers and data in different chunks. Because length alone cannot determine where a semantic unit ends, the cut point often passes right through the middle of a sentence or table.

If a sentence required for the correct answer is split into two chunks, retrieval may only fetch one side, leading to an answer with insufficient evidence. Microsoft explains that the structure must be identified before cutting the document to determine a chunking strategy.

Problems with Losing Document Hierarchical Structure

Documents have a hierarchical structure leading from titles to subtitles and then to the body text. Splitting without reflecting this structure can result in fragments where it is difficult to tell which section the content belongs to just by looking at the chunk.

For example, if a subtitle like ‘3.2 Refund Policy’ is separated from the body text into a different chunk, the clue that the remaining body chunk explains the refund policy is weakened. Since titles serve as the criteria for identifying the topic of the body text, breaking the connection between the title and body makes it difficult to determine which topic a sentence explains. In this case, even if a user asks about the refund period, the retrieval relevance of that body chunk may be rated low, preventing it from appearing at the top of search results.

Dividing documents while maintaining hierarchical structure is more difficult than it seems. In fact, even in widely used open-source splitting tools, issues have been reported where paragraph distinctions weaken when cutting based on Markdown headers, which can affect subsequent splitting and analysis.

Why Row and Column Information is Missing in Table Lists

Tables and lists are structures where meaning is determined by the relationship between each element. In tables, the relationship between headers and values, and rows and columns, must be maintained; in lists, the connection between parent and child items must be preserved. Cutting documents based only on length can separate these relationships into different chunks, making it difficult to understand what the data means.

For example, if the header ‘Q1 2026 Revenue’ and the numbers below it are split into different chunks, the chunk containing only numbers will have difficulty explaining which item those values represent. While a table’s meaning becomes clear when headers and values are together, methods that only divide text sequentially fail to sufficiently reflect cell positional relationships and header connections.

The technology required here is TSR (Table Structure Recognition). TSR recognizes the row-column structure and header-value relationships within a table, structuring table data into meaningful units. If this process is insufficient, the retrieval stage may fetch a numeric chunk missing its header, and the Large Language Model (LLM) may misinterpret which item that figure refers to. Especially for long or complex tables, it is important to preprocess so that header information and row-column relationships can be referenced together at the chunk level.

The causes of these three types can ultimately be summarized as one: cutting based only on length without sufficiently reflecting the document’s structural information. Therefore, to improve chunking quality, a process of structuring the document hierarchy and table relationships before cutting is necessary.

Methods to Reduce Context Loss in Document Structure-Based Chunking

Document hierarchical structure-based chunking is a method that uses structural elements of a document—such as titles, subtitles, body text, and tables—as boundaries for splitting.

Because it cuts by semantic units rather than document length, titles and body text are grouped in the same chunk, and table headers and values are kept together.

By including the title path in each chunk to show which section the fragment belongs to, both the body text and the context can be utilized during retrieval. The RAG process proceeds in the order of ‘Document Upload → Parsing → Chunking → Embedding → Vector DB Storage → Retrieval → Answer Generation.’

Here, embedding is the task of converting text into numeric vectors containing meaning, and the Vector DB is the repository that stores and searches those vectors. If hierarchical information is lost during the chunking stage, it will affect the subsequent embedding and retrieval quality.

The level to which the document’s hierarchical structure is preserved and extracted is also important. According to the HiChunk study, which compared hierarchical chunking, performance in retrieving all necessary evidence sentences for an answer gradually improved as the hierarchy increased from 1 to 3 levels, with no significant changes observed after 3 levels.

Conversely, limiting the hierarchy to 1 level caused the semantic range included in a single chunk to become excessively broad, degrading RAG retrieval performance.

RAG Preprocessing Methods for Structure-Based Chunking

Three preprocessing steps are required before cutting to preserve the document’s hierarchical structure.

These include distinguishing title, body, and table areas in the document, attaching the title path as metadata to each chunk, and converting the document into a format where the structure is preserved. Only through this preprocessing can the meaning and context of the document be preserved in the subsequent chunking stage.

Distinguishing Title, Body, and Table Areas with DLA

DLA (Document Layout Analysis) is a structural analysis technology that uses AI deep learning to distinguish various elements within a document—such as text, images, tables, and graphics—and accurately identify the position and relationship of each element.

Identifying this structure before chunking can reduce problems where titles and body text are incorrectly grouped or tables are separated from the body text.

For documents containing tables, TSR should also be checked. TSR restores the row-column relationships of a table, helping to ensure that headers and values are not disconnected.

Preserving Title Hierarchy as Metadata

Preserving the title hierarchy as metadata means storing which title each fragment falls under within each chunk. Attaching a title path like ‘Document Name > Chapter 3 > 3.2 Refund Policy’ to a chunk allows even a chunk containing only body text to be understood within its context.

Metadata is additional information stored along with the data. If titles are maintained rather than deleted, both the body and title context are reflected during the retrieval stage. Thus, when a user asks about the refund period, the chunk containing ‘Refund Policy’ in its title path is retrieved first. Databricks explains that hierarchical chunking preserves the major/minor sections and paragraph structure of a document across multiple levels.

Converting to Structured Data for Chunking and Embedding

Converting to structured data refers to the task of changing a document into a format that can contain structural information, such as JSON, HTML, or Markdown. This allows chunking tools to recognize title hierarchies and table structures much more easily.

When chunking proceeds with the document structure preserved, each chunk can utilize information about which title it is under and which item a value in a table is connected to. Conversely, cutting only text without structural information can result in chunks with broken meaning.

Ultimately, structured data conversion can be seen as preparatory work to better utilize the document’s context during the chunking and embedding stages.

To divide a document based on titles or tables during the chunking stage, the distinction between titles, body text, and tables must first remain in the input data.

This is because once the document structure is lost, it is difficult to accurately reconnect titles with body text or tables with data.

Hancom Data Loader is a document parsing solution that converts documents into data with preserved structural information at this front end.

💡 You can see for yourself how preprocessing that preserves document structure makes a difference by uploading a document in the Hancom Data Loader live demo.

👉 Go to Hancom Data Loader Live Demo

Hancom Data Loader’s Hierarchical Structure-Based Preprocessing Method

When Hancom Data Loader provides structured data that preserves the document’s hierarchical structure and table relationships, it becomes much easier to use title, body, and table boundaries as criteria in the subsequent chunking stage.

Parsing that Preserves Structure with DLA and TSR – Improving Chunking Quality

Hancom Data Loader provides structured data that maintains document context by distinguishing title, body, and table areas with DLA and restoring table structures with TSR in the pre-cutting stage.

Data organized this way makes it easy to group titles with body text and table headers with values in subsequent chunking, serving as a foundation for improving retrieval quality. It processes OCR (Optical Character Recognition), DLA, and TSR in a single pipeline rather than splitting them into different tools.

The reduction in connection errors and management burden that occurs when linking multiple open-source tools is also effective in B2B environments. Additionally, the Level Inference Hierarchical Structure feature, which automatically tags the hierarchy of titles, subtitles, and body text based on indentation, numbering systems, and font size, allows this hierarchical information to serve as the basis for chunking.

Structured Data Conversion by Format and Customer Cases

Hancom Data Loader flexibly applies processing methods according to document formats and characteristics, distinguishing between text-centric documents, image-based documents, and documents where the original structure must be read directly to convert them into structured data.

Hancom Data Loader is a document parsing solution that converts HWP, HWPX, PDF, and OOXML into structured data. For HWP and HWPX, it parses the original directly to extract data while maintaining hierarchy and table structures.

Hancom Data Loader was utilized in the establishment of the Gyeonggi Provincial Office of Education AI Digital Platform. It converted approximately 2,800 school websites, 40,000 information materials, and 7,000 guidelines and laws from HWP, HWPX, and PDF into structured data, including hierarchical information and complex tables such as nested tables and merged cells.

🖥️Hancom Data Loader

Ultimately, RAG chunking quality is not determined solely at the chunking stage. If the relationships between titles, body text, tables, and images are lost, it is difficult to stably divide semantic units regardless of the chunking strategy used.

Hancom Data Loader converts HWP, HWPX, PDF, and OOXML documents into structured data, establishing a preprocessing foundation that allows high-quality data to be utilized in subsequent processes.

If necessary, the extracted structured data can be used in the RAG pipeline in conjunction with Hancompedia.

If you want to improve RAG answer quality, you can start by checking whether the original document structure is being extracted correctly.

👉 Go to Hancom Data Loader Live Demo

👉 Inquire about Hancom Data Loader implementation

FAQ: Frequently Asked Questions About RAG Chunking and Preprocessing

---

References

1. IBM, “Chunking strategies for RAG tutorial using Granite”
2. arXiv, Lu et al., “HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunking”, 2025
3. arXiv, Nguyen et al., “Enhancing Retrieval Augmented Generation with Hierarchical Text Segmentation Chunking”, 2025
4. Databricks, Debu Sinha, “The Ultimate Guide to Chunking Strategies for RAG Applications”
5. Microsoft Learn, “Developing Azure RAG Solutions – Preparation Phase”