What is RAG? The Role of Retrieval-Augmented Generation and Preprocessing <\/strong><\/h2>\n\nWhat are the Meaning and Basic Concepts of RAG?<\/strong><\/h3>\n\nUnlike general LLMs that rely solely on training data, RAG is an AI architecture that searches external documents\u2014such as internal corporate documents, in-house manuals, regulations, and reports\u2014and generates answers based on highly relevant content. The quality of document preprocessing, the very first stage of the pipeline, has a significant impact on subsequent retrieval accuracy and answer quality.<\/p>\n\n
RAG System Architecture and the Role of Preprocessing<\/strong><\/h3>\n\nRAG works in the following order: Document Upload \u2192 Parsing \u2192 Chunking \u2192 Embedding \u2192 Vector DB (embedding storage and retrieval database) Storage \u2192 Retrieval \u2192 Answer Generation<\/strong>. If preprocessing is inadequate, the AI may pull irrelevant information and generate answers that differ from the facts\u2014i.e., hallucinations, where the AI outputs incorrect content as if it were true without any basis. <\/p>\n\nIf the preprocessing stage is insufficient, errors can carry over into the subsequent chunking, embedding, and retrieval stages. In particular, for formats widely used in Korea such as HWP and HWPX, most major global Document AI solutions do not support direct parsing of the original files, making it difficult to fully preserve their structure with general-purpose tools. <\/p>\n\n
Worried about the burden of document preprocessing and data loss when building a RAG pipeline?<\/strong><\/p>\n\n\ud83d\udc49 <\/a> Explore Hancom Data Loader<\/a><\/p>\n\n<\/div>\n\n![\"Image](\"https:\/\/blog.hancom.com\/wp-content\/uploads\/2026\/07\/rag-data-preprocessing-rag-workflow-six-steps-1024x576.png\")
<\/figure>\n\n<\/div>\n\nWhy is Structured Conversion Necessary for Unstructured Documents?<\/strong><\/h2>\n\nUnstructured documents such as HWP, HWPX, PDF, and OOXML cannot be directly understood by AI. Therefore, structured data conversion must precede for RAG to function correctly.<\/p>\n\n
Why Search Errors Occur in Documents with Lost Structure<\/strong><\/h3>\n\nWhen document structure is lost, the AI generates irrelevant answers based on it.<\/p>\n\n
Formats that only support text extraction, such as OOXML, or simple OCR (Optical Character Recognition) methods may not preserve table structure, paragraph hierarchy, or reading order information. <\/p>\n\n
For example, if numbers in a table are mixed with body text, AI cannot distinguish which numbers are table values and which content is descriptive text. As a result, it may fail to find relevant information or provide factually incorrect answers. If structural information like table layouts or paragraph hierarchies is lost, the context that AI relies on is diminished. <\/p>\n\n
Structural loss is not just a formatting issue. It becomes a cause of RAG quality degradation, lowering chunking, embedding, and retrieval accuracy as well. <\/p>\n\n
<\/div>\n\nHow are OCR and DLA different?<\/strong><\/h2>\n\nOCR is a character recognition technology, while DLA (Document Layout Analysis) is a technology that understands the overall structure of a document. To improve RAG quality, both technologies are needed together. <\/p>\n\n
Why Building RAG with OCR Alone is Difficult<\/strong><\/h3>\n\nOCR is a technology that converts characters from images or scanned documents into text. Since it is specialized for character recognition, it is difficult to preserve document structure information such as table structures, paragraph hierarchies, and reading order. In PDFs, the visual order and the actual extraction order can differ, leading to issues where semantic units get mixed up when using OCR alone. <\/p>\n\n
Meanwhile, DLA identifies the structural relationship of extracted text, determining whether it’s a table, heading, or body content. This understanding of structural relationships allows the AI to interpret the same number completely differently depending on whether it’s a value within a table or part of a body explanation. <\/p>\n\n
In summary, OCR is the technology that reads characters, and DLA is the technology that identifies the context\u2014whether those characters belong to a table or a heading. Both technologies must work together within a single pipeline to improve RAG quality. <\/p>\n\n
<\/div>\n\n![\"Image](\"https:\/\/blog.hancom.com\/wp-content\/uploads\/2026\/07\/rag-data-preprocessing-ocr-vs-hancom-data-loader-difference-1024x576.png\")
<\/figure>\n\n<\/div>\n\nThe Impact of Preserving HWP\/HWPX Structure on Search Accuracy<\/strong><\/h3>\n\nHWP and HWPX are formats with proprietary file structures. Using only general OCR or PDF conversion results in the loss of information regarding tables, footnotes, and indentation, which decreases RAG search accuracy.<\/p>\n\n
Although these are the most commonly used document formats in Korean public institutions, legal, and financial sectors, most major global Document AI solutions only support general-purpose formats like PDF, JPEG, PNG, and TIFF, and do not support direct parsing of HWP\/HWPX originals. Direct parsing of the originals is essential to maintain RAG search accuracy. <\/p>\n\n
Directly parsing HWP\/HWPX originals requires core technology that can access the proprietary format structure. If processed by converting to PDF like general tools, key formatting such as footnotes, merged cells, and indentation may be lost. <\/p>\n\n
\ud83d\udca1 How to directly parse HWP\/HWPX originals without conversion, based on Hancom’s 30 years of technology?<\/p>\n\n
\ud83d\udc49Explore Hancom Data Loader<\/a><\/p>\n\n<\/div>\n\n\n![\"RAG](\"https:\/\/blog.hancom.com\/wp-content\/uploads\/2026\/07\/rag-data-preprocessing-rag-search-flow-diagram-1-1024x576.png\")
<\/figure>\n<\/figure>\n\n<\/div>\n\nRAG Chunking and Embedding: The Impact of Preprocessing Quality<\/strong><\/h2>\n\nChunking is the process of dividing a document into semantic units, and embedding is the process of converting text into numerical vectors to make it searchable. Both stages depend on the quality of preprocessing. <\/p>\n\n
Why Chunking Quality Determines Search Accuracy<\/strong><\/h3>\n\nThe choice of chunking strategy directly determines the retrieval accuracy of a RAG system.<\/p>\n\n
The key to chunking is not just cutting text into small pieces, but dividing it while ensuring the meaning of paragraphs is fully maintained. Both Semantic Chunking and Hierarchical Chunking work effectively only when the document structure has been preserved beforehand. If DLA is not performed first, any chunking method becomes less effective. <\/p>\n\n
Whether it is semantic-based or hierarchy-based chunking, the document’s headings, paragraphs, and table hierarchies must first be accurately extracted to work effectively. This is why document parsing quality must be secured before choosing any chunking strategy. <\/p>\n\n
Why Poor Chunking and Embedding Degrade AI Answer Quality<\/strong><\/h3>\n\nWhen chunking quality is low, context is separated or semantic units are lost, and the quality of embedding\u2014which converts text into numeric vectors\u2014also declines. Ultimately, vector DB search accuracy and AI answer quality both decline together. <\/p>\n\n
Ultimately, vector DB search accuracy drops, increasing the likelihood of RAG hallucinations. This is why, in most cases, the cause of degraded RAG embedding quality lies in the preprocessing stage rather than the LLM. <\/p>\n\n
<\/div>\n\n![\"Image](\"https:\/\/blog.hancom.com\/wp-content\/uploads\/2026\/07\/rag-data-preprocessing-hancom-data-loader-core-technology-1-1024x576.png\")
<\/figure>\n\n<\/div>\n\nFrequently Asked Questions (FAQ) on RAG Preprocessing<\/strong><\/h2>\n\n<\/div>\n\n\n\nQ1. Why is document preprocessing important in RAG? <\/strong><\/h3>\n\n\nThis is because if the document structure is lost, the quality of chunking and embedding decreases, leading to a drop in both search accuracy and AI answer quality. RAG Retrieval Accuracy is determined at the preprocessing stage. <\/p>\n\n<\/div>\n<\/div>\n
\nQ2. Is it possible to build RAG using only OCR? <\/strong><\/h3>\n\n\nIt is possible, but limitations may arise unless the documents are simple. If only OCR is used, only the character information of the document is extracted, so structural information such as paragraph types (headings, body, lists), table structures, and reading order may be lost. <\/p>\n\n<\/div>\n<\/div>\n
\nQ3. Can HWP\/HWPX files also be used for RAG? <\/strong><\/h3>\n\n\nThey can be used, but structural information may be lost through general OCR or PDF conversion due to their proprietary file structures. A solution that directly parses HWP\/HWPX originals is required. Hancom Data Loader processes HWP\/HWPX originals without conversion based on Hancom’s core technology, minimizing structural loss. <\/p>\n\n<\/div>\n<\/div>\n
\nQ4. What criteria should be used to select a RAG document preprocessing solution? <\/strong><\/h3>\n\n\nKey selection criteria include support for direct parsing of HWP\/HWPX originals, a single pipeline for DLA, OCR, and TSR, and the ability to run on-premise in a closed network. Hancom Data Loader meets all three conditions and is a proven solution in public, financial, and legal environments with GS certification. <\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
<\/div>\n\n![\"Image](\"https:\/\/blog.hancom.com\/wp-content\/uploads\/2026\/07\/rag-data-preprocessing-hancom-data-loader-features-summary-1-1024x576.png\")
<\/figure>\n\n<\/div>\n\nHow to Solve RAG Document Preprocessing with Hancom Data Loader<\/strong><\/h2>\n\nAs we have seen, RAG performance is determined at the preprocessing stage before the LLM. The document structure must be fully preserved for chunking to maintain semantic units, for embedding quality to improve, and for vector DB search accuracy to be secured. If preprocessing quality declines, all subsequent steps degrade with it. <\/p>\n\n
The problem is that most documents handled in actual corporate environments, such as HWP, HWPX, PDF, and OOXML, are in formats that are difficult to parse while preserving structure. In particular, since global Document AI solutions do not support direct parsing of HWP\/HWPX originals, it is difficult to connect them to a RAG pipeline without structural loss using conventional methods. <\/p>\n\n
Hancom Data Loader supports the entire process as a single pipeline, from document parsing\u2014the very first step of RAG preprocessing\u2014to structured data conversion.<\/strong><\/p>\n\n\u2705 Processes DLA, OCR, and TSR composite AI in a single pipeline to accurately extract structured text and object information from documents <\/p>\n\n
\u2705 Supports all formats actually used in corporate environments, including HWP, HWPX, PDF, and OOXML, and directly parses HWP\/HWPX original files without conversion <\/p>\n\n
\u2705 Automatically determines the hierarchy of headings, paragraphs, and tables through Level Inference-based hierarchical structure extraction and outputs it as structured data, contributing to quality improvement in the subsequent chunking stage. <\/p>\n\n
\u2705 Prevents external leakage of documents even in network-separated or closed-network environments by fully internalizing on-premise with container-based REST API<\/p>\n\n
\u2705 A proven solution in public, financial, and legal environments with GS certification <\/p>\n\n
If your current RAG answers are not meeting expectations, check your preprocessing environment with Hancom Data Loader before looking at the LLM.<\/p>\n\n
\ud83d\udc49 Go use the Hancom Data Loader live demo<\/a> <\/p>\n\n\ud83d\udc49 Inquire About Hancom Data Loader<\/a><\/p>\n\n
\n\nReferences<\/strong><\/p>\n\n\n- arXiv<\/a><\/strong>, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” Lewis et al., NeurIPS 2020<\/li>\n\n\n\n
- AWS<\/strong><\/a>, “What is Retrieval-Augmented Generation (RAG)?”<\/li>\n\n\n\n
- AWS<\/strong><\/a>, “What is OCR?”<\/li>\n\n\n\n
- Microsoft<\/strong><\/a>, “Chunk large documents for RAG and vector search in Azure AI Search”<\/li>\n\n\n\n
- arXiv<\/strong><\/a>, “Revolutionizing RAG with Enhanced PDF Structure Recognition,” Lin, arXiv:2401.12599<\/li>\n\n\n\n
- arXiv<\/strong><\/a>, “Document Parsing Unveiled: Techniques, Challenges, and Prospects,” Lin, arXiv:2410.21169<\/li>\n\n\n\n
- arXiv<\/strong><\/a>, “SCAN: Semantic Document Layout Analysis for Textual and Visual RAG,” Ueda et al., arXiv:2505.14381, EACL 2026<\/li>\n\n\n\n
- AWS Textract<\/strong><\/a>, “Supported Document Formats”<\/li>\n\n\n\n
- Microsoft Azure<\/strong><\/a>, “Azure Document Intelligence Overview”<\/li>\n\n\n\n
- Google<\/strong><\/a>, “Document AI Supported File Types”<\/li>\n<\/ol>\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
The quality of RAG data preprocessing is determined at the document structure analysis stage, not by the LLM. We have summarized the core principles\u2014from chunking and embedding to RAG search accuracy\u2014in a way that even non-developers can understand. <\/p>\n","protected":false},"author":2,"featured_media":1392,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[29],"tags":[],"class_list":["post-854","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ax-glossary"],"_links":{"self":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts\/854","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/comments?post=854"}],"version-history":[{"count":18,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts\/854\/revisions"}],"predecessor-version":[{"id":1838,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts\/854\/revisions\/1838"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/media\/1392"}],"wp:attachment":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/media?parent=854"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/categories?post=854"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/tags?post=854"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Unlike general LLMs that rely solely on training data, RAG is an AI architecture that searches external documents\u2014such as internal corporate documents, in-house manuals, regulations, and reports\u2014and generates answers based on highly relevant content. The quality of document preprocessing, the very first stage of the pipeline, has a significant impact on subsequent retrieval accuracy and answer quality.<\/p>\n\n
RAG System Architecture and the Role of Preprocessing<\/strong><\/h3>\n\nRAG works in the following order: Document Upload \u2192 Parsing \u2192 Chunking \u2192 Embedding \u2192 Vector DB (embedding storage and retrieval database) Storage \u2192 Retrieval \u2192 Answer Generation<\/strong>. If preprocessing is inadequate, the AI may pull irrelevant information and generate answers that differ from the facts\u2014i.e., hallucinations, where the AI outputs incorrect content as if it were true without any basis. <\/p>\n\nIf the preprocessing stage is insufficient, errors can carry over into the subsequent chunking, embedding, and retrieval stages. In particular, for formats widely used in Korea such as HWP and HWPX, most major global Document AI solutions do not support direct parsing of the original files, making it difficult to fully preserve their structure with general-purpose tools. <\/p>\n\n
Worried about the burden of document preprocessing and data loss when building a RAG pipeline?<\/strong><\/p>\n\n\ud83d\udc49 <\/a> Explore Hancom Data Loader<\/a><\/p>\n\n<\/div>\n\n<\/figure>\n\n<\/div>\n\nWhy is Structured Conversion Necessary for Unstructured Documents?<\/strong><\/h2>\n\nUnstructured documents such as HWP, HWPX, PDF, and OOXML cannot be directly understood by AI. Therefore, structured data conversion must precede for RAG to function correctly.<\/p>\n\n
Why Search Errors Occur in Documents with Lost Structure<\/strong><\/h3>\n\nWhen document structure is lost, the AI generates irrelevant answers based on it.<\/p>\n\n
Formats that only support text extraction, such as OOXML, or simple OCR (Optical Character Recognition) methods may not preserve table structure, paragraph hierarchy, or reading order information. <\/p>\n\n
For example, if numbers in a table are mixed with body text, AI cannot distinguish which numbers are table values and which content is descriptive text. As a result, it may fail to find relevant information or provide factually incorrect answers. If structural information like table layouts or paragraph hierarchies is lost, the context that AI relies on is diminished. <\/p>\n\n
Structural loss is not just a formatting issue. It becomes a cause of RAG quality degradation, lowering chunking, embedding, and retrieval accuracy as well. <\/p>\n\n
<\/div>\n\nHow are OCR and DLA different?<\/strong><\/h2>\n\nOCR is a character recognition technology, while DLA (Document Layout Analysis) is a technology that understands the overall structure of a document. To improve RAG quality, both technologies are needed together. <\/p>\n\n
Why Building RAG with OCR Alone is Difficult<\/strong><\/h3>\n\nOCR is a technology that converts characters from images or scanned documents into text. Since it is specialized for character recognition, it is difficult to preserve document structure information such as table structures, paragraph hierarchies, and reading order. In PDFs, the visual order and the actual extraction order can differ, leading to issues where semantic units get mixed up when using OCR alone. <\/p>\n\n
Meanwhile, DLA identifies the structural relationship of extracted text, determining whether it’s a table, heading, or body content. This understanding of structural relationships allows the AI to interpret the same number completely differently depending on whether it’s a value within a table or part of a body explanation. <\/p>\n\n
In summary, OCR is the technology that reads characters, and DLA is the technology that identifies the context\u2014whether those characters belong to a table or a heading. Both technologies must work together within a single pipeline to improve RAG quality. <\/p>\n\n
<\/div>\n\n<\/figure>\n\n<\/div>\n\nThe Impact of Preserving HWP\/HWPX Structure on Search Accuracy<\/strong><\/h3>\n\nHWP and HWPX are formats with proprietary file structures. Using only general OCR or PDF conversion results in the loss of information regarding tables, footnotes, and indentation, which decreases RAG search accuracy.<\/p>\n\n
Although these are the most commonly used document formats in Korean public institutions, legal, and financial sectors, most major global Document AI solutions only support general-purpose formats like PDF, JPEG, PNG, and TIFF, and do not support direct parsing of HWP\/HWPX originals. Direct parsing of the originals is essential to maintain RAG search accuracy. <\/p>\n\n
Directly parsing HWP\/HWPX originals requires core technology that can access the proprietary format structure. If processed by converting to PDF like general tools, key formatting such as footnotes, merged cells, and indentation may be lost. <\/p>\n\n
\ud83d\udca1 How to directly parse HWP\/HWPX originals without conversion, based on Hancom’s 30 years of technology?<\/p>\n\n
\ud83d\udc49Explore Hancom Data Loader<\/a><\/p>\n\n<\/div>\n\n\n<\/figure>\n<\/figure>\n\n<\/div>\n\nRAG Chunking and Embedding: The Impact of Preprocessing Quality<\/strong><\/h2>\n\nChunking is the process of dividing a document into semantic units, and embedding is the process of converting text into numerical vectors to make it searchable. Both stages depend on the quality of preprocessing. <\/p>\n\n
Why Chunking Quality Determines Search Accuracy<\/strong><\/h3>\n\nThe choice of chunking strategy directly determines the retrieval accuracy of a RAG system.<\/p>\n\n
The key to chunking is not just cutting text into small pieces, but dividing it while ensuring the meaning of paragraphs is fully maintained. Both Semantic Chunking and Hierarchical Chunking work effectively only when the document structure has been preserved beforehand. If DLA is not performed first, any chunking method becomes less effective. <\/p>\n\n
Whether it is semantic-based or hierarchy-based chunking, the document’s headings, paragraphs, and table hierarchies must first be accurately extracted to work effectively. This is why document parsing quality must be secured before choosing any chunking strategy. <\/p>\n\n
Why Poor Chunking and Embedding Degrade AI Answer Quality<\/strong><\/h3>\n\nWhen chunking quality is low, context is separated or semantic units are lost, and the quality of embedding\u2014which converts text into numeric vectors\u2014also declines. Ultimately, vector DB search accuracy and AI answer quality both decline together. <\/p>\n\n
Ultimately, vector DB search accuracy drops, increasing the likelihood of RAG hallucinations. This is why, in most cases, the cause of degraded RAG embedding quality lies in the preprocessing stage rather than the LLM. <\/p>\n\n
<\/div>\n\n<\/figure>\n\n<\/div>\n\nFrequently Asked Questions (FAQ) on RAG Preprocessing<\/strong><\/h2>\n\n<\/div>\n\n\n\nQ1. Why is document preprocessing important in RAG? <\/strong><\/h3>\n\n\nThis is because if the document structure is lost, the quality of chunking and embedding decreases, leading to a drop in both search accuracy and AI answer quality. RAG Retrieval Accuracy is determined at the preprocessing stage. <\/p>\n\n<\/div>\n<\/div>\n
\nQ2. Is it possible to build RAG using only OCR? <\/strong><\/h3>\n\n\nIt is possible, but limitations may arise unless the documents are simple. If only OCR is used, only the character information of the document is extracted, so structural information such as paragraph types (headings, body, lists), table structures, and reading order may be lost. <\/p>\n\n<\/div>\n<\/div>\n
\nQ3. Can HWP\/HWPX files also be used for RAG? <\/strong><\/h3>\n\n\nThey can be used, but structural information may be lost through general OCR or PDF conversion due to their proprietary file structures. A solution that directly parses HWP\/HWPX originals is required. Hancom Data Loader processes HWP\/HWPX originals without conversion based on Hancom’s core technology, minimizing structural loss. <\/p>\n\n<\/div>\n<\/div>\n
\nQ4. What criteria should be used to select a RAG document preprocessing solution? <\/strong><\/h3>\n\n\nKey selection criteria include support for direct parsing of HWP\/HWPX originals, a single pipeline for DLA, OCR, and TSR, and the ability to run on-premise in a closed network. Hancom Data Loader meets all three conditions and is a proven solution in public, financial, and legal environments with GS certification. <\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
<\/div>\n\n<\/figure>\n\n<\/div>\n\nHow to Solve RAG Document Preprocessing with Hancom Data Loader<\/strong><\/h2>\n\nAs we have seen, RAG performance is determined at the preprocessing stage before the LLM. The document structure must be fully preserved for chunking to maintain semantic units, for embedding quality to improve, and for vector DB search accuracy to be secured. If preprocessing quality declines, all subsequent steps degrade with it. <\/p>\n\n
The problem is that most documents handled in actual corporate environments, such as HWP, HWPX, PDF, and OOXML, are in formats that are difficult to parse while preserving structure. In particular, since global Document AI solutions do not support direct parsing of HWP\/HWPX originals, it is difficult to connect them to a RAG pipeline without structural loss using conventional methods. <\/p>\n\n
Hancom Data Loader supports the entire process as a single pipeline, from document parsing\u2014the very first step of RAG preprocessing\u2014to structured data conversion.<\/strong><\/p>\n\n\u2705 Processes DLA, OCR, and TSR composite AI in a single pipeline to accurately extract structured text and object information from documents <\/p>\n\n
\u2705 Supports all formats actually used in corporate environments, including HWP, HWPX, PDF, and OOXML, and directly parses HWP\/HWPX original files without conversion <\/p>\n\n
\u2705 Automatically determines the hierarchy of headings, paragraphs, and tables through Level Inference-based hierarchical structure extraction and outputs it as structured data, contributing to quality improvement in the subsequent chunking stage. <\/p>\n\n
\u2705 Prevents external leakage of documents even in network-separated or closed-network environments by fully internalizing on-premise with container-based REST API<\/p>\n\n
\u2705 A proven solution in public, financial, and legal environments with GS certification <\/p>\n\n
If your current RAG answers are not meeting expectations, check your preprocessing environment with Hancom Data Loader before looking at the LLM.<\/p>\n\n
\ud83d\udc49 Go use the Hancom Data Loader live demo<\/a> <\/p>\n\n\ud83d\udc49 Inquire About Hancom Data Loader<\/a><\/p>\n\n
\n\nReferences<\/strong><\/p>\n\n\n- arXiv<\/a><\/strong>, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” Lewis et al., NeurIPS 2020<\/li>\n\n\n\n
- AWS<\/strong><\/a>, “What is Retrieval-Augmented Generation (RAG)?”<\/li>\n\n\n\n
- AWS<\/strong><\/a>, “What is OCR?”<\/li>\n\n\n\n
- Microsoft<\/strong><\/a>, “Chunk large documents for RAG and vector search in Azure AI Search”<\/li>\n\n\n\n
- arXiv<\/strong><\/a>, “Revolutionizing RAG with Enhanced PDF Structure Recognition,” Lin, arXiv:2401.12599<\/li>\n\n\n\n
- arXiv<\/strong><\/a>, “Document Parsing Unveiled: Techniques, Challenges, and Prospects,” Lin, arXiv:2410.21169<\/li>\n\n\n\n
- arXiv<\/strong><\/a>, “SCAN: Semantic Document Layout Analysis for Textual and Visual RAG,” Ueda et al., arXiv:2505.14381, EACL 2026<\/li>\n\n\n\n
- AWS Textract<\/strong><\/a>, “Supported Document Formats”<\/li>\n\n\n\n
- Microsoft Azure<\/strong><\/a>, “Azure Document Intelligence Overview”<\/li>\n\n\n\n
- Google<\/strong><\/a>, “Document AI Supported File Types”<\/li>\n<\/ol>\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
The quality of RAG data preprocessing is determined at the document structure analysis stage, not by the LLM. We have summarized the core principles\u2014from chunking and embedding to RAG search accuracy\u2014in a way that even non-developers can understand. <\/p>\n","protected":false},"author":2,"featured_media":1392,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[29],"tags":[],"class_list":["post-854","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ax-glossary"],"_links":{"self":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts\/854","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/comments?post=854"}],"version-history":[{"count":18,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts\/854\/revisions"}],"predecessor-version":[{"id":1838,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts\/854\/revisions\/1838"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/media\/1392"}],"wp:attachment":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/media?parent=854"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/categories?post=854"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/tags?post=854"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
If the preprocessing stage is insufficient, errors can carry over into the subsequent chunking, embedding, and retrieval stages. In particular, for formats widely used in Korea such as HWP and HWPX, most major global Document AI solutions do not support direct parsing of the original files, making it difficult to fully preserve their structure with general-purpose tools. <\/p>\n\n
Worried about the burden of document preprocessing and data loss when building a RAG pipeline?<\/strong><\/p>\n\n \ud83d\udc49 <\/a> Explore Hancom Data Loader<\/a><\/p>\n\n Unstructured documents such as HWP, HWPX, PDF, and OOXML cannot be directly understood by AI. Therefore, structured data conversion must precede for RAG to function correctly.<\/p>\n\n When document structure is lost, the AI generates irrelevant answers based on it.<\/p>\n\n Formats that only support text extraction, such as OOXML, or simple OCR (Optical Character Recognition) methods may not preserve table structure, paragraph hierarchy, or reading order information. <\/p>\n\n For example, if numbers in a table are mixed with body text, AI cannot distinguish which numbers are table values and which content is descriptive text. As a result, it may fail to find relevant information or provide factually incorrect answers. If structural information like table layouts or paragraph hierarchies is lost, the context that AI relies on is diminished. <\/p>\n\n Structural loss is not just a formatting issue. It becomes a cause of RAG quality degradation, lowering chunking, embedding, and retrieval accuracy as well. <\/p>\n\n OCR is a character recognition technology, while DLA (Document Layout Analysis) is a technology that understands the overall structure of a document. To improve RAG quality, both technologies are needed together. <\/p>\n\n OCR is a technology that converts characters from images or scanned documents into text. Since it is specialized for character recognition, it is difficult to preserve document structure information such as table structures, paragraph hierarchies, and reading order. In PDFs, the visual order and the actual extraction order can differ, leading to issues where semantic units get mixed up when using OCR alone. <\/p>\n\n Meanwhile, DLA identifies the structural relationship of extracted text, determining whether it’s a table, heading, or body content. This understanding of structural relationships allows the AI to interpret the same number completely differently depending on whether it’s a value within a table or part of a body explanation. <\/p>\n\n In summary, OCR is the technology that reads characters, and DLA is the technology that identifies the context\u2014whether those characters belong to a table or a heading. Both technologies must work together within a single pipeline to improve RAG quality. <\/p>\n\n HWP and HWPX are formats with proprietary file structures. Using only general OCR or PDF conversion results in the loss of information regarding tables, footnotes, and indentation, which decreases RAG search accuracy.<\/p>\n\n Although these are the most commonly used document formats in Korean public institutions, legal, and financial sectors, most major global Document AI solutions only support general-purpose formats like PDF, JPEG, PNG, and TIFF, and do not support direct parsing of HWP\/HWPX originals. Direct parsing of the originals is essential to maintain RAG search accuracy. <\/p>\n\n Directly parsing HWP\/HWPX originals requires core technology that can access the proprietary format structure. If processed by converting to PDF like general tools, key formatting such as footnotes, merged cells, and indentation may be lost. <\/p>\n\n \ud83d\udca1 How to directly parse HWP\/HWPX originals without conversion, based on Hancom’s 30 years of technology?<\/p>\n\n \ud83d\udc49Explore Hancom Data Loader<\/a><\/p>\n\n Chunking is the process of dividing a document into semantic units, and embedding is the process of converting text into numerical vectors to make it searchable. Both stages depend on the quality of preprocessing. <\/p>\n\n The choice of chunking strategy directly determines the retrieval accuracy of a RAG system.<\/p>\n\n The key to chunking is not just cutting text into small pieces, but dividing it while ensuring the meaning of paragraphs is fully maintained. Both Semantic Chunking and Hierarchical Chunking work effectively only when the document structure has been preserved beforehand. If DLA is not performed first, any chunking method becomes less effective. <\/p>\n\n Whether it is semantic-based or hierarchy-based chunking, the document’s headings, paragraphs, and table hierarchies must first be accurately extracted to work effectively. This is why document parsing quality must be secured before choosing any chunking strategy. <\/p>\n\n When chunking quality is low, context is separated or semantic units are lost, and the quality of embedding\u2014which converts text into numeric vectors\u2014also declines. Ultimately, vector DB search accuracy and AI answer quality both decline together. <\/p>\n\n Ultimately, vector DB search accuracy drops, increasing the likelihood of RAG hallucinations. This is why, in most cases, the cause of degraded RAG embedding quality lies in the preprocessing stage rather than the LLM. <\/p>\n\n This is because if the document structure is lost, the quality of chunking and embedding decreases, leading to a drop in both search accuracy and AI answer quality. RAG Retrieval Accuracy is determined at the preprocessing stage. <\/p>\n\n<\/div>\n<\/div>\n It is possible, but limitations may arise unless the documents are simple. If only OCR is used, only the character information of the document is extracted, so structural information such as paragraph types (headings, body, lists), table structures, and reading order may be lost. <\/p>\n\n<\/div>\n<\/div>\n They can be used, but structural information may be lost through general OCR or PDF conversion due to their proprietary file structures. A solution that directly parses HWP\/HWPX originals is required. Hancom Data Loader processes HWP\/HWPX originals without conversion based on Hancom’s core technology, minimizing structural loss. <\/p>\n\n<\/div>\n<\/div>\n Key selection criteria include support for direct parsing of HWP\/HWPX originals, a single pipeline for DLA, OCR, and TSR, and the ability to run on-premise in a closed network. Hancom Data Loader meets all three conditions and is a proven solution in public, financial, and legal environments with GS certification. <\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n As we have seen, RAG performance is determined at the preprocessing stage before the LLM. The document structure must be fully preserved for chunking to maintain semantic units, for embedding quality to improve, and for vector DB search accuracy to be secured. If preprocessing quality declines, all subsequent steps degrade with it. <\/p>\n\n The problem is that most documents handled in actual corporate environments, such as HWP, HWPX, PDF, and OOXML, are in formats that are difficult to parse while preserving structure. In particular, since global Document AI solutions do not support direct parsing of HWP\/HWPX originals, it is difficult to connect them to a RAG pipeline without structural loss using conventional methods. <\/p>\n\n Hancom Data Loader supports the entire process as a single pipeline, from document parsing\u2014the very first step of RAG preprocessing\u2014to structured data conversion.<\/strong><\/p>\n\n \u2705 Processes DLA, OCR, and TSR composite AI in a single pipeline to accurately extract structured text and object information from documents <\/p>\n\n \u2705 Supports all formats actually used in corporate environments, including HWP, HWPX, PDF, and OOXML, and directly parses HWP\/HWPX original files without conversion <\/p>\n\n \u2705 Automatically determines the hierarchy of headings, paragraphs, and tables through Level Inference-based hierarchical structure extraction and outputs it as structured data, contributing to quality improvement in the subsequent chunking stage. <\/p>\n\n \u2705 Prevents external leakage of documents even in network-separated or closed-network environments by fully internalizing on-premise with container-based REST API<\/p>\n\n \u2705 A proven solution in public, financial, and legal environments with GS certification <\/p>\n\n If your current RAG answers are not meeting expectations, check your preprocessing environment with Hancom Data Loader before looking at the LLM.<\/p>\n\n \ud83d\udc49 Go use the Hancom Data Loader live demo<\/a> <\/p>\n\n \ud83d\udc49 Inquire About Hancom Data Loader<\/a><\/p>\n\n References<\/strong><\/p>\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" The quality of RAG data preprocessing is determined at the document structure analysis stage, not by the LLM. We have summarized the core principles\u2014from chunking and embedding to RAG search accuracy\u2014in a way that even non-developers can understand. <\/p>\n","protected":false},"author":2,"featured_media":1392,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[29],"tags":[],"class_list":["post-854","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ax-glossary"],"_links":{"self":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts\/854","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/comments?post=854"}],"version-history":[{"count":18,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts\/854\/revisions"}],"predecessor-version":[{"id":1838,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts\/854\/revisions\/1838"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/media\/1392"}],"wp:attachment":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/media?parent=854"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/categories?post=854"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/tags?post=854"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}<\/figure>\n\nWhy is Structured Conversion Necessary for Unstructured Documents?<\/strong><\/h2>\n\n
Why Search Errors Occur in Documents with Lost Structure<\/strong><\/h3>\n\n
How are OCR and DLA different?<\/strong><\/h2>\n\n
Why Building RAG with OCR Alone is Difficult<\/strong><\/h3>\n\n
<\/figure>\n\nThe Impact of Preserving HWP\/HWPX Structure on Search Accuracy<\/strong><\/h3>\n\n
<\/figure>\n<\/figure>\n\nRAG Chunking and Embedding: The Impact of Preprocessing Quality<\/strong><\/h2>\n\n
Why Chunking Quality Determines Search Accuracy<\/strong><\/h3>\n\n
Why Poor Chunking and Embedding Degrade AI Answer Quality<\/strong><\/h3>\n\n
<\/figure>\n\nFrequently Asked Questions (FAQ) on RAG Preprocessing<\/strong><\/h2>\n\n
Q1. Why is document preprocessing important in RAG? <\/strong><\/h3>\n
Q2. Is it possible to build RAG using only OCR? <\/strong><\/h3>\n
Q3. Can HWP\/HWPX files also be used for RAG? <\/strong><\/h3>\n
Q4. What criteria should be used to select a RAG document preprocessing solution? <\/strong><\/h3>\n
<\/figure>\n\nHow to Solve RAG Document Preprocessing with Hancom Data Loader<\/strong><\/h2>\n\n
\n\n\n