Boosting RAG Data Preprocessing Performance with VLM (Vision-Language Model)-Based Document Parsing

While VLM (Vision-Language Model) and multimodal AI seem capable of understanding tables and charts in documents at once, actual RAG (Retrieval-Augmented Generation) preprocessing results often show table structures being lost or chart data failing to extract properly. Text may be extracted, but critical data can be lost in the process.

This article examines how to effectively combine VLM with document parsing, and outlines the inefficiencies and considerations that can arise during implementation.

Limitations of Standard OCR and Simple Text Parsers in Document Processing

Standard OCR (Optical Character Recognition) and simple text parsers only read character coordinates in a document—they fail to capture visual context and structural relationships, such as whether text is a table header or a footnote. Even when characters are accurately recognized, the information about where those characters belong is lost, frequently causing data structure damage during the RAG preprocessing stage.

Why Simple Text Extraction Misses Visual Context

Simple text extraction misses visual context because it only extracts text without reading the role that text plays within the document.

For example, when a report is processed with standard OCR, sentences are extracted, but the connection to which chart the sentence describes or which table cell it belongs to is not captured.

As a result, the link between table titles and cell values breaks, the reading order in multi-column layouts gets scrambled, and even when the same number is read, it becomes difficult to distinguish which item that number represents.

According to research on document parsing technology, methods that only extract characters lose connection information such as ‘which item this number belongs to,’ making the results difficult to use directly in RAG. Therefore, DLA (Document Layout Analysis), which distinguishes titles from body text, must accompany extraction so that the results can be utilized as data.

*Source: arXiv, Zhang et al., “Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction“, 2024

The Problem of Unstructured Visual Data Being Lost Without Use

Charts, graphs, and diagrams containing key information in corporate reports can have their data omitted by simple parsers, preventing them from being loaded into RAG systems at all.

Revenue trends and comparison graphs needed for decision-making risk being entirely omitted at the early stages of the data pipeline.

Errors that occur at this stage propagate to subsequent retrieval and generation stages. The OHR-Bench study, which measured how OCR errors affect RAG performance, found that even using the best-performing OCR resulted in an overall accuracy drop of approximately 14% (5 points on the F1 scale), with greater losses at the retrieval and generation stages. The complete omission of visual data can also be problematic—because only text is read, without capturing where that text is positioned in the document.

*Source: arXiv, Zhang et al., “OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation” (OHR-Bench, ICCV 2025), 2024

Why VLM Is Necessary for In-House RAG Preprocessing

What Is VLM?

VLM is a multimodal AI that combines visual information with language. It processes images and text simultaneously to describe images or answer questions about them. By analyzing an image, it explains what is inside using full sentences.

Structurally, it combines a vision encoder that interprets images with a language model, passing image features extracted by the vision encoder to the language model through cross-attention layers. The difference from LLMs also emerges at this point. While LLMs are strong with text, they cannot directly handle non-linguistic information such as images or video—VLM addresses this limitation by incorporating a vision encoder, extending the scope of interpretation to include visual elements within documents.

Want to learn more about how VLM understands images?

👉 What Is VLM? Understanding the Concept and Document AI Applications

Converting Charts and Diagrams into Text Data Through VLM Image Captioning

VLM visually recognizes charts and diagrams that simple parsers skip, converting them into natural language captions that RAG can search—such as ‘bar graph comparing quarterly revenue’ or ‘image showing key product components.’ This conversion technology, which transforms images into sentences that search engines can handle, is called image captioning.

The NVIDIA Technical Blog notes that RAG’s utility greatly increases when it handles not only text but also tables, graphs, charts, and diagrams together. Only when visual information is converted to text does the entire document become searchable.

*Source: NVIDIA Technical Blog, “An Easy Introduction to Multimodal Retrieval-Augmented Generation“

Improving Document Understanding Through Structural Layout Recognition

Document understanding improves when table titles and cell values, and the positional relationships between images and surrounding explanatory text, are preserved and passed to the LLM. An LLM that receives contextually connected data can properly link items with values when answering.

Therefore, in addition to converting visual information to text through captioning, preprocessing that preserves layout structure must also take place.

The effect of layout-preserving preprocessing is confirmed numerically. According to the VisRAG study, which proposed a method for understanding documents as images with VLM rather than first converting them to text, preserving layout and visual information resulted in retrieval and answer accuracy improvements of 20–40% over text-based RAG.

*Source: arXiv, Yu et al., “VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents“, 2024

Potential Issues When Adopting External VLM APIs

Integrating a separate parser with an external VLM can cause token costs and response delays to accumulate with each document processed.

Additionally, passing tables and images directly to an external VLM raises concerns about hallucination—such as generating nonexistent values—and in air-gapped environments, security constraints prevent the use of external APIs altogether.

Token Costs, Time Delays, and Hallucination in Large-Scale Document Processing

When processing large volumes of documents, integrating an external VLM separately causes both token costs and response delays to grow as the number of documents increases. This is because the structure sends parser-extracted results to an external model for interpretation.

In fact, an article covering a practitioner engineer’s implementation experience analyzed that each external VLM call during the retrieval stage adds approximately 5–7 seconds of response time.

Notably, these call issues occur not once at ingestion time, but every time a question is asked—so the more questions, the more costs multiply.

Cost and delay are not the only problems. VLMs are also not entirely free from the risk of hallucination, such as describing content not present in an image or misinterpreting cell values in complex tables.

For example, passing a table as an image directly to an external VLM risks misreading row-column relationships or generating values that do not exist in the actual document.

Therefore, the issue to address here is not VLM itself, but the approach of relying entirely on an external VLM for interpretation without first organizing document structure. First structuring tables and layouts with DLA and TSR (Table Structure Recognition), then applying VLM only to visual information—such as charts or diagrams—that is difficult to explain through text extraction alone, is a more stable approach. In other words, when document parsing lays the foundation and VLM supplements visual information, the reliability of RAG preprocessing results can be improved.

*Source: arXiv, Kogilathota et al., “HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token“, 2026

*Source: Towards AI, “Your Multimodal RAG Pipeline Should Look at Images Twice“, 2026

Constraints on Building On-Premises Secure Environments Due to External Cloud Integration

To use external VLM APIs or cloud-based document recognition services, internal documents must be transmitted to external cloud environments for analysis. For this reason, in environments such as public sector, finance, and legal where external document transmission is restricted, security policy and compliance review must come before functional performance considerations. You must first confirm whether the approach can be used in that environment, rather than focusing on cost or speed.

Particularly in closed networks or air-gapped environments where external communication is blocked, the method of sending documents outside can become a constraint on adoption. Therefore, in such environments, a configuration that can handle document extraction, captioning, and structural analysis within the internal network is critical.

Ultimately, to reduce cost, response delay, hallucination risk, and security constraints together, a solution is needed in which VLM-based captioning and DLA, OCR, and TSR are internalized within a single pipeline.

Hancom Data Loader: A Document Parsing Solution That Enhances Accuracy with VLM Capabilities

🖥️Hancom Data Loader

From Direct HWP/HWPX Parsing to Proprietary VLM

Hancom Data Loader is a document parsing solution that converts HWP, HWPX, PDF, and OOXML into structured data. Notably, HWP and HWPX are parsed directly from the original format without conversion to PDF, reducing the loss of table structures, footnotes, and metadata during conversion.

Text and tables are extracted through direct parsing, while visual information that is difficult to capture as text—such as charts and images—is processed by the internalized proprietary VLM through image captioning. (Image captioning is currently in the PoC stage; the commercial release schedule will be announced later.)

By not passing all elements wholesale to an external VLM, the cost, speed, and hallucination burdens associated with external VLM integration can be reduced.

Extraction That Preserves Table and Layout Structure with DLA and TSR

DLA and TSR are the two pillars that extract tables and layouts—which are easily damaged—while preserving their structure. When DLA distinguishes title, body, table, and image regions and restores reading order, TSR restores the table interior at the cell level.

TSR recognizes row-column relationships even in merged cells, multi-level headers, borderless tables, and tables within tables, outputting results in Markdown, JSON, or HTML. Table data that would scatter into running text with simple extraction is restored as structured data maintaining cell-level relationships—when both technologies work together, extraction accuracy can be improved even in complex documents. However, processing methods vary by input format. The same technology is not uniformly applied to all formats.

Input Format	Processing Method
HWP, HWPX	Direct Original Parsing, DLA/TSR Application, OCR Selected by API Type
PDF	All DLA, TSR, and OCR Applied
PNG, JPG, BMP	OCR-Based Recognition
OOXML	Text Extraction

The hierarchical information extracted in this way continues to the next stage. Structured data with preserved title, subtitle, and body hierarchy serves as the basis for semantic chunking in Hancompedia, our RAG solution. The division of roles is clear: Hancom Data Loader handles extraction and parsing, while Hancompedia manages the RAG pipeline from chunking onward.

SaaS API and On-Premises: Choosing Based on Environment

Hancom Data Loader can be deployed as either on-premises or SaaS API, depending on the implementation environment.

In environments such as public sector and finance where external transmission is restricted, the on-premises method—installed on internal servers—is appropriate. Documents can be processed within the internal network without sending them outside, making it easier to meet security requirements even in closed or air-gapped environments.

Conversely, when initial validation or rapid integration is needed, the SaaS API can be used. Without building a separate server, documents can be integrated via API, and with page-based billing, you can first verify the extraction results of actual documents.

The core of RAG preprocessing is reading documents not just as characters, but with their structure intact. Hancom Data Loader processes DLA, OCR, TSR, and VLM captioning in a single pipeline to reduce information loss. Actual document extraction results can be verified in the live demo.

👉 Curious how tables and charts are extracted? Check the live demo

👉 Inquire about document parsing solution implementation tailored to your document environment

Frequently Asked Questions

---

References

1. arXiv, Zhang et al., “Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction”, 2024
2. arXiv, Zhang et al., “OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation” (OHR-Bench, ICCV 2025), 2024
3. NVIDIA Technical Blog, “An Easy Introduction to Multimodal Retrieval-Augmented Generation”
4. arXiv, Yu et al., “VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents”, 2024