{"id":1630,"date":"2026-08-14T09:00:00","date_gmt":"2026-08-14T00:00:00","guid":{"rendered":"https:\/\/blog.hancom.com\/hwp-rag-preprocessing-binary-parsing\/"},"modified":"2026-08-25T11:08:07","modified_gmt":"2026-08-25T02:08:07","slug":"hwp-rag-preprocessing-binary-parsing","status":"publish","type":"post","link":"https:\/\/blog.hancom.com\/en\/hwp-rag-preprocessing-binary-parsing\/","title":{"rendered":"Why HWP RAG Preprocessing Fails | Limitations of General Parsing Tools and Solutions"},"content":{"rendered":"\n

When public and financial institutions build in-house LLM and RAG systems, the first thing to check is the preprocessing method for existing HWP documents. If the decades of accumulated official documents, reports, and proposals cannot be transformed into structured data that AI can read, it becomes difficult to properly build the knowledge base that RAG relies on. <\/p>\n\n

HWP is a format that is difficult to process with simple text extraction alone. Structural elements like tables, footnotes, and paragraph hierarchies must be preserved together to maintain stable search and answer quality. This article explains why general parsing tools show limitations with HWP and how PDF conversion, HWPX conversion, and direct original binary parsing methods differ. <\/p>\n\n

<\/div>\n\n

Why HWP Document Preprocessing is Crucial in the RAG Pipeline<\/strong><\/h2>\n\n
<\/div>\n\n

AI Adoption by Public Institutions and the HWP Format: Why Preprocessing is a Prerequisite<\/strong><\/h3>\n\n

The quality of AI adoption by public institutions can vary depending on how accurately existing HWP documents are structured. A significant portion of administrative, public, and financial knowledge is accumulated in HWP documents, so if these cannot be converted into structured data that AI can read, it becomes difficult to sufficiently build the knowledge base for RAG. <\/p>\n\n

HWP is a document format that has been widely used in the domestic business environment for a long time, including government official documents, bid proposals, and court documents. Therefore, in HWP preprocessing, it is crucial to preserve structural information such as tables, paragraph hierarchies, and footnotes, beyond simple text extraction. <\/p>\n\n

<\/div>\n\n
\"Policy<\/figure>\n\n
<\/div>\n\n

Policy is also accelerating this task. The Ministry of Economy and Finance<\/a> decided to reflect ‘AI utilization and innovation’ in the 2026 public institution management evaluation, and the Ministry of Interior and Safety<\/a> mandated the attachment of documents in the Onnara System to be in an open format (HWPX). However, even if new documents are created in HWPX, existing HWP documents still remain, so the demand for preprocessing to convert these legacy documents into AI knowledge assets continues. <\/p>\n\n

Such use cases are also emerging in actual public projects. Hancom Data Loader was utilized in the Gyeonggi Provincial Office of Education’s AI Digital Platform construction project<\/a> to convert school website guides, guidelines, and legal data into a format that AI can learn from. By organizing scattered administrative documents into structured data, a foundation can be created for school staff to find necessary information based on internal documents and use it for Q&A. <\/p>\n\n

Structural Limitations of HWP Format Vulnerable to Simple Text Extraction Parsing<\/strong><\/h2>\n\n

Necessity of Understanding CFB and Document Structure<\/strong><\/h3>\n\n

HWP has a structure that makes it difficult for general text extraction tools to directly read its main content. This is because the file’s internal structure is based on CFB (Compound File Binary, or Compound File Binary format), divided into Storage and Stream, requiring an understanding of the document’s structure first to reliably extract data. <\/p>\n\n

The characters and tables displayed on the screen are not stored as a single plain text. Document information, body text, tables, and formatting information are managed in separate streams and record units. Therefore, to parse HWP, one must first identify the Storage and Stream structure within the CFB and sequentially interpret the body records to reliably extract text and document structure. <\/p>\n\n

Hancom Tech’s HWP format parsing<\/a> analysis also explains that document information and body text are stored distributed within the CFB structure, and encrypted documents require a separate processing step. If this structure is not properly interpreted, text may be omitted, or structural data such as tables, paragraphs, and formatting information may be lost. This is why general parsers struggle to produce stable extraction results from HWP. <\/p>\n\n

Information Omission and Order Distortion During Variable-Length Data Record Parsing<\/strong><\/h3>\n\n

HWP body text and document information are stored sequentially as variable-length records. Therefore, if a dedicated engine fails to accurately interpret the record header, it can miscalculate the stream pointer’s position, leading to table structure damage or omission of body text. This occurs because data within a stream is not divided into fixed sizes; instead, each piece of information is stored in ‘records’ of varying lengths. Accurately reading the header is essential to determine the starting position of the next data. <\/p>\n\n

<\/div>\n\n
\"HWP<\/figure>\n\n
<\/div>\n\n

Each record header contains information about the record type (Tag ID), logical hierarchy (Level), and data length (Size). If the byte order for reading the header, or the length and hierarchy information, is misinterpreted by even one step, the starting position of the next record can be miscalculated, leading to damaged table cell boundaries or omitted paragraphs. This difficulty in interpreting variable-length records is why simple text extraction tools often damage HWP tables. <\/p>\n\n

*Source: <\/em>Hancom Tech<\/em><\/a>, \u300cHangul Document File Format: HWP Format Parsing with Python (2)\u300d, 2025<\/em><\/p>\n\n

<\/div>\n\n

Solutions for HWP Hangul Document Parsing Issues<\/strong><\/h2>\n\n
<\/div>\n\n
\"\"<\/figure>\n\n
<\/div>\n\n

Comparison of 3 HWP Preprocessing Methods<\/strong><\/h3>\n\n
Category<\/strong><\/td>Conversion Process<\/strong><\/td>Structure\/Table Preservation<\/strong><\/td>RAG Suitability<\/strong><\/td><\/tr>
PDF Conversion<\/td>HWP \u2192 PDF \u2192 Parsing<\/td>Potential hierarchy\/table damage during conversion<\/td>Chunking boundaries blurred<\/td><\/tr>
HWPX Conversion<\/td>HWP \u2192 HWPX \u2192 Parsing<\/td>Easy access to XML structure<\/td>Usable as standard data<\/td><\/tr>
Direct Original Parsing<\/td>Direct Binary Structure Interpretation<\/td>Minimizes conversion loss<\/td>Better for semantic chunking<\/td><\/tr><\/tbody><\/table><\/figure>\n\n

Advantages and Limitations of HWP Format PDF Conversion Method<\/strong><\/h3>\n\n

The method of converting HWP to PDF and then parsing it offers the advantage of utilizing general PDF parsers and OCR tools in the post-conversion stage. PDF processing tools are numerous, making it relatively easy to integrate them into existing document processing pipelines. <\/p>\n\n

However, this method adds an extra step of converting the original HWP to PDF. During this process, paragraph hierarchy, line breaks, and table row\/column structures may be rearranged differently from the original, and subsequent parsing results may become unstable, especially with Hangul administrative documents that have multi-column layouts or many merged cells. <\/p>\n\n

Processing time can also increase during conversion. According to tests<\/a>, a task that took 9.5 seconds for direct extraction from HWPX took 150.3 seconds when extracted after PDF conversion, which is approximately 15.8 times longer.<\/p>\n\n

*Source: <\/em>Hancom Tech<\/em><\/a>, \u300cHangul Documents for the AX Era: Analysis of Open Source Ecosystem Status and Activation Strategy\u300d, 2025<\/em><\/p>\n\n

Since PDFs often prioritize display layout information over logical document structure, from a RAG preprocessing perspective, additional work may be required to restore the reading order and table structure.<\/p>\n\n

Data Standardization through Open Document Format (HWPX) Conversion<\/strong><\/h3>\n\n

The core of converting HWP to HWPX is the ability to standardize and process existing HWP documents into an XML-based structure. HWPX is an open document format that contains XML documents and resources within a ZIP package, making it easier to access text and metadata structures than directly interpreting HWP’s CFB structure and zlib compressed streams. <\/p>\n\n

Furthermore, HWPX’s technical standard is publicly available, which is advantageous for analyzing internal document structures and linking with other systems. However, even during the conversion from HWP to HWPX, it is necessary to verify whether the original formatting and table structures are maintained as intended. When utilizing large volumes of documents for RAG preprocessing, it is crucial to check the quality of the extracted structure and text after conversion, rather than simply using the converted results as-is. <\/p>\n\n

Hancom provides a free converter for HWP to HWPX, allowing up to 30 files to be converted at once.<\/p>\n\n

Preserving Data with Direct Parsing of Original HWP Binary<\/strong><\/h3>\n\n

Direct parsing of the original HWP binary interprets the record and compression structure of the binary file itself to extract text, tables, and hierarchies as they are, without intermediate conversion steps like PDF or HWPX. This approach reduces the potential for data loss during conversion. It also prevents line break errors or table structure damage that typically occur during conversion. <\/p>\n\n

In RAG preprocessing, the accuracy of document structure extraction can affect answer quality. If table row\/column relationships or paragraph hierarchies are lost during the retrieval phase, the context of the evidentiary data referenced by the LLM also weakens. Indeed, a ChatDOC study<\/a> evaluating RAG with enhanced PDF structure recognition showed that systems applying a structure recognition parser provided better answers in approximately 47% of questions compared to the baseline, were equivalent in 38%, and inferior in 15%. <\/p>\n\n

This demonstrates that how accurately structural information is preserved during the parsing stage is a critical factor in determining subsequent search and answer quality.<\/p>\n\n

*This study does not specifically address direct HWP parsing, but it serves as a reference demonstrating the importance of preserving document structure.<\/em><\/p>\n\n

\ud83d\udca1 You can check how original HWP documents are converted into structured data.<\/p>\n\n

\ud83d\udc49 Go to Hancom Data Loader Live Demo<\/a><\/p>\n\n

<\/div>\n\n
\"Hancom<\/figure>\n\n
<\/div>\n\n

Optimizing Hangul Document RAG with Hancom Data Loader<\/strong><\/h2>\n\n
<\/div>\n\n

Hancom Data Loader is essential for building Hangul document RAG because it directly parses original HWP and HWPX files without going through a PDF conversion step, converting them into structured data without conversion loss.<\/p>\n\n

Eliminating Conversion Loss with Direct Original Parsing<\/strong><\/h3>\n\n

Direct original parsing helps reduce structural loss that can occur during the conversion process. While PDF conversion can lead to line breaks, reading order, and table row\/column relationships being rearranged differently from the original, directly analyzing the original document can mitigate these issues. The extraction results are provided as structured data, such as JSON, which can be utilized in the RAG preprocessing stage. <\/p>\n\n

Preserving Hierarchy and Tables with Document Layout Analysis (DLA) and Table Structure Recognition (TSR)<\/strong><\/h3>\n\n

DLA (Document Layout Analysis) and TSR (Table Structure Recognition) play crucial roles in preserving the semantic units of a document. DLA is used to distinguish document components such as titles, body text, tables, and images, and to identify the reading order. TSR restores row\/column relationships in complex table structures, such as tables without borders, merged cells, or tables within tables, which can reduce problems where numbers or cell values are separated without context during the RAG retrieval stage. <\/p>\n\n

Integration with Hancom Pedia for RAG Search and Answer Generation<\/strong><\/h3>\n\n

Hancom Data Loader extracts and parses text, tables, and paragraph hierarchy information from documents, converting them into structured data. Hancom Pedia then receives this data and connects it to the chunking, embedding, vector DB storage, and RAG search and answer generation stages. In essence, Hancom Data Loader handles document preprocessing, while Hancom Pedia manages the search and answer generation stages. <\/p>\n\n

On-premise deployment is also possible for environments with high security requirements. Hancom Data Loader can be deployed within an internal network as a container-based REST API, making it suitable for public and financial environments where documents cannot be transmitted to external servers. In fact, Hancom is building RAG implementation cases for public documents by supplying Hancom Data Loader and Hancom Pedia for the Ministry of Interior and Safety’s Pan-Government Super-Large AI Common Foundation Implementation Project<\/a>. <\/p>\n\n

<\/div>\n\n

Frequently Asked Questions Regarding HWP Document Processing in RAG Systems<\/strong><\/h2>\n
\n
\n
\n

If table structures are lost when converting HWP to PDF, does this affect RAG performance?<\/strong><\/strong><\/strong><\/h3>\n
\n\n

If tables are treated as images or paragraph hierarchies are flattened during the PDF conversion step, chunking boundaries become inaccurate, and this loss propagates to the embedding and retrieval stages, degrading RAG answer quality.<\/p>\n\n<\/div>\n<\/div>\n

\n

Is there a way to parse HWP or HWPX files directly without going through PDF?<\/strong><\/strong><\/strong><\/h3>\n
\n\n

If a dedicated parsing engine with core HWP and HWPX format technology is available, it can directly analyze the binary and XML structures for extraction without PDF conversion. Reading the original directly preserves hierarchies, tables, and formatting without a conversion step, and Hancom Data Loader uses this direct original parsing method. <\/p>\n\n<\/div>\n<\/div>\n

\n

HWP files are not supported by AI frameworks like LangChain. What should I do?<\/strong><\/strong><\/strong><\/h3>\n
\n\n

Since LangChain does not have an official standard HWP loader, it is more practical to first structure the data using a separate parsing pipeline or a dedicated solution, and then integrate it with the framework. The open-source pyhwpx<\/a> operates via COM automation, which only works in environments where Hangul is installed on Windows, making it unsuitable for Linux servers or closed network environments. <\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

\ud83d\udda5\ufe0f<\/strong>Hancom Data Loader<\/strong><\/a><\/h3>\n\n

Hancom Data Loader is a document parsing solution that converts HWP, HWPX, and PDF documents into structured data that AI can utilize. OOXML documents are processed using text extraction. The extracted and parsed data can be integrated with Hancom Pedia to extend into a RAG search and answer pipeline. If you require preprocessing that preserves the structure of original HWP and HWPX documents, you can find it in Hancom Data Loader. <\/p>\n\n

\ud83d\udc49 Go to Hancom Data Loader Live Demo<\/a><\/p>\n\n

\ud83d\udc49 Inquire about Hancom Data Loader implementation<\/a><\/p>\n\n

\n\n

References<\/strong><\/h2>\n\n
    \n
  1. ZDNet Korea<\/a>, \u201cPublic documents to mandate \u2018HWPX\u2019 from the 18th\u2026Transition to an AI-friendly administrative system,\u201d 2026<\/li>\n\n\n\n
  2. Ministry of Economy and Finance<\/a>, \u300cPublic Institutions to Lead AI Adoption and Utilization\u300d, 2025<\/li>\n\n\n\n
  3. AWS<\/a>, \u201cWhat is Retrieval-Augmented Generation (RAG)?\u201d<\/li>\n\n\n\n
  4. arXiv<\/a>, \u201cDocument Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction,\u201d 2024<\/li>\n<\/ol>\n\n

    <\/p>\n","protected":false},"excerpt":{"rendered":"

    Are you curious why global open-source parsing tools struggle with HWP, the Korean standard administrative format? We summarize solutions for RAG document preprocessing, including PDF conversion, HWPX conversion, and direct original binary parsing. <\/p>\n","protected":false},"author":2,"featured_media":1631,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[26],"tags":[],"class_list":["post-1630","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ax-insight"],"_links":{"self":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts\/1630","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/comments?post=1630"}],"version-history":[{"count":6,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts\/1630\/revisions"}],"predecessor-version":[{"id":1767,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/posts\/1630\/revisions\/1767"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/media\/1631"}],"wp:attachment":[{"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/media?parent=1630"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/categories?post=1630"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.hancom.com\/en\/wp-json\/wp\/v2\/tags?post=1630"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}