Building your first RAG system? A step-by-step guide from document parsing to Q
“They’re telling me to build a chatbot with internal documents, but I’m not even sure what RAG is.”
“There are too many components—where do I even start?”
Everyone faces challenges when first building a RAG (Retrieval-Augmented Generation) system. RAG is a technology in which an LLM retrieves external documents as supporting evidence before generating an answer, but a closer look reveals a string of unfamiliar terms such as chunking, embedding, and vector DB. This article provides a step-by-step overview of the entire workflow, from the concept of RAG and its seven core components to implementation methods using LangChain and LlamaIndex, as well as document parsing and preprocessing that determine accuracy.
What is a RAG system? The concept of retrieval-augmented generation and why it emerged
RAG (retrieval-augmented generation) is a technique that searches external documents and uses them as evidence before an LLM answers, introduced to address the limits of training data and the hallucination problem.
RAG concept and definition
RAG is a technique in which an LLM first retrieves relevant information from reliable external knowledge sources before generating an answer, then produces the answer grounded in that information. As the name suggests, it refers to a structure that combines three actions in order: Retrieval + Augmented + Generation.
When documents run to thousands of pages, you can’t feed them into an LLM all at once—RAG retrieves only the necessary parts and provides them as evidence. The paper on general-purpose fine-tuning methods for RAG also reports that combining an external retrieval index makes answers more specific and closer to factual.
Limitations of using an LLM alone and the hallucination problem
Because an LLM does not know information after its training point or domain knowledge it was never trained on, hallucinations can occur—where it makes up unknown content as if it were fact.
There are two causes: the knowledge cutoff, which prevents it from knowing information after training ends, and a lack of domain knowledge for internal policies or specialized documents it never learned. IBM also explains that hallucinations are more likely when a model cannot learn new data.
Why RAG is needed
Because RAG retrieves external data and uses it as evidence for answers, it can reflect up-to-date information and reduce hallucinations without retraining the model.
Fine-tuning requires retraining the model, but RAG can reflect the latest information by updating only the external index. It can also present the documents used for retrieval, making it easier to verify the basis for an answer. Reflecting current information without retraining costs—and being able to check sources—are key reasons teams choose RAG.
RAG system diagram: seven core components
RAG system diagram: from document extraction to LLM answers
A RAG system works in the order: document extraction → parsing → chunking → embedding → vector DB storage → retrieval → LLM answer. Even though it looks like many steps, it becomes easier to understand when you group them into two parts.
- Document extraction – pulls content from the original document
- Parsing – extracts text and structural information
- Chunking – splits the document into smaller units
- Embedding – converts the split text into numeric vectors
- Vector DB storage – loads the converted vectors into a retrieval store
- Retrieval – finds the content most similar to the question
- LLM answer – generates an answer grounded in the retrieved content
The first five steps are the indexing process that prepares data in advance, while the last two steps (retrieval and LLM answering) run each time a question comes in.
The AWS guide also notes that embedding storage is a one-time step, while retrieval, augmentation, and generation repeat for every question. This distinction between one-time preparation and repeated execution becomes a key lens for understanding the implementation methods covered later.
Document parsing: the first step in RAG and the baseline for quality
Document parsing is the step that extracts text and structural information from original documents in formats such as HWP, HWPX, and PDF, and it determines the quality of subsequent chunking, embedding, and retrieval.
If structural information is damaged at this stage, every subsequent step inherits data with that structure already lost. As described in Unstructured’s open source, the core of parsing is splitting a document into elements such as Title, NarrativeText, and Table—but in the Korean environment, the difficulty rises further with HWP and HWPX, which are optimized for local use. Hancom Data Loader directly parses these HWP/HWPX files from the original binary without converting them to PDF, preserving table structures, footnotes, and metadata.
The roles of chunking and embeddings
Chunking is the process of splitting a document into smaller units, and embedding is the process of converting those split texts into numeric vectors that capture meaning.
If chunks are too large, different topics get mixed together and retrieval becomes inaccurate; if they’re too small, context breaks and meaning becomes hard to interpret. Traditional approaches that cut text only by character count fail to reflect document structure such as titles and paragraphs. The study on RAG chunks and frameworks points out that this approach cannot produce chunks with sufficient meaning.
Vector databases and Top-K retrieval
A vector database is a retrieval-focused store that saves embedded vectors and quickly finds the vectors most similar to a user’s question. Common vector databases include FAISS and Chroma.
Top-K is the value that determines how many high-similarity items to return from retrieval results. If K is too large, low-relevance information may be pulled in; if it’s too small, needed information can be missed—so choosing an appropriate value matters.
The LLM answer generation stage
The LLM takes the retrieved document chunks as context and generates the final answer based on a prompt that combines them with the user’s question.
A prompt is the combination of retrieved document content and the user’s question. Because the model answers based on retrieved evidence rather than its own memory, how accurately the earlier stages retrieve the right materials largely determines answer quality.
How to build a RAG system: from preprocessing to answer generation
Building a RAG system can be divided into a preprocessing domain that extracts and structures documents, and a retrieval/generation domain that handles everything from chunking to answer generation—then connecting the two domains with the right solutions.
Building a RAG system: documents
The RAG system build process is divided into a preprocessing domain that extracts and structures documents and a retrieval/generation domain that handles everything from chunking to answer generation—then connecting each domain with the right solutions.
Step 1: Document structure analysis and RAG document preprocessing
In the document preprocessing stage, you analyze the original document’s layout, tables, and paragraph hierarchy and convert them into structured data. The quality of this output determines chunking accuracy later.
The core of this process is DLA (Document Layout Analysis), which distinguishes text, images, tables, and graphics and identifies their positions and relationships. The results are organized into a standard structure, like the Document objects produced by LangChain’s DocumentLoader. However, because you must account for formats, table structures, and Korean-language document characteristics, teams often place a specialized preprocessing solution at the front to handle this stage.
Step 2: Chunking, embedding, and vector DB storage
Preprocessed data is chunked, vectorized via an embedding model, and stored in a vector database for later retrieval.
This is the indexing stage that prepares retrieval data in advance before questions arrive, and RAG frameworks typically handle chunking, embedding, and vector database storage.
Step 3: Retrieval and LLM answer generation
When a user question comes in, it is vectorized with the same embedding model, similarity search is performed, and the retrieved documents are passed to the LLM to generate the final answer.
This RAG retrieval process converts the question into an embedding, extracts relevant chunks via similarity search in the vector DB, assembles them into a prompt, and has the LLM answer. The question must be vectorized in the same way as stored documents to correctly find semantically similar content.
How to implement RAG: LangChain vs. LlamaIndex
RAG can be implemented with frameworks such as LangChain and LlamaIndex, and each framework differs in how it composes components from document loading through retrieval and generation.
How to implement RAG with LangChain
LangChain is a framework that builds a RAG pipeline by combining components such as DocumentLoader, Text Splitter, Retriever, and Chains.
The implementation flow is: load documents with a DocumentLoader, split them with a Text Splitter, embed and store them in a vector store, retrieve results with a Retriever, connect them via a Chain, and call the LLM. Because it’s a modular, component-assembly structure, it can flexibly support complex workflows.
How to implement RAG with LlamaIndex
LlamaIndex is a framework specialized in indexing diverse data sources, with strengths in connecting documents and building indexes.
The core is a data connector that converts content from multiple sources into Document objects and builds them into an index. The official LlamaIndex documentation describes this loading process as a flow of loading, transforming, indexing, and storing. Even within RAG, if LangChain emphasizes chain/agent composition, LlamaIndex’s strengths lie in data indexing and retrieval optimization.
Criteria for choosing RAG document preprocessing tools
The right RAG document preprocessing tool depends on the document formats you need to handle and the complexity of their structure. Plain text and PDFs are often fine with basic document loaders, but complex tables or Korean domestic formats typically require separate preprocessing.
LangChain and LlamaIndex document loaders handle pipeline wiring—such as chunking, retrieval, and chain composition—well, but because they’re optimized mainly for PDFs and text, they may struggle to preserve structure for tables with merged cells or domestic formats like HWP and HWPX.
According to reporting by AI Times, Hancom Data Loader is a solution that extracts text and various objects from PDFs and Office documents and preprocesses them into data for AI training. It later expanded support to structured data extraction across HWP, HWPX, PDF, and OOXML overall, based on DLA and OCR (Optical Character Recognition) and TSR (Table Structure Recognition).
💡 If table structures or HWP/HWPX parsing are bottlenecks in your RAG pipeline, you can upload real documents in the Hancom Data Loader live demo and check the extraction results directly.
👉 Go to Hancom Data Loader Live Demo
LangChain vs. LlamaIndex: what’s different?
LangChain is strong at composing diverse LLM chains and agents, while LlamaIndex excels at indexing large document sets and optimizing retrieval—so the right choice depends on your project goals.
IBM distinguishes LangChain as an orchestration framework for LLM applications and LlamaIndex as a data orchestration framework.
Comparison of strengths and best-fit use cases for LangChain and LlamaIndex
| Category | LangChain | LlamaIndex |
| Strength area | Workflow orchestration such as chains and agents | Data indexing and retrieval optimization |
| Best fit | Complex workflows, integrating a variety of tools | Large-scale document Q, lightweight RAG |
| Specialization | General-purpose LLM application composition | Connecting data sources and building indexes |
However, for both frameworks, performance ultimately depends on the quality of the input documents. No matter how well you assemble the pipeline, it’s hard to overcome the limits of poor initial extraction.
Common issues when building a RAG system and how to address them
Common issues in RAG builds include hallucinations and reduced retrieval accuracy due to loss of table and hierarchical structure—both of which are directly tied to the quality of input document preprocessing.
Core principles for preventing hallucinations in RAG
To reduce hallucinations in RAG, a key principle is to explicitly instruct the prompt to answer only from the retrieved context, and to constrain the model to say it doesn’t know when the answer cannot be found in that context.
Even so, RAG reduces hallucinations but does not eliminate them entirely. According to the study on LLM errors, even when relevant context is provided, LLMs still often include unsupported information or contradictions. In other words, the fundamental path to reducing hallucinations ultimately lies in document quality at the input stage.
TSR: how to extract merged cells and multi-level headers
TSR is a technique that restores row/column relationships between cells, converting tables with merged cells and multi-level headers into structured data.
In many cases, retrieval accuracy drops because table structure is lost at the input stage. When a table is handled as plain text, the linkage information between cells—what tells you which values belong to which fields—disappears.
The study on the STC (Structure-Aware Tabular Chunking) framework measured accuracy metrics (MRR and Recall@1; closer to 1 means it finds the correct answer well) on table-heavy data such as legal contracts, evaluating how well question-relevant document chunks rise to the top of retrieval results.
As a result, when chunking preserved table structure, the metric improved from around 0.36 to about 0.6–0.75 depending on the retrieval method. This indicates that structure-preserving chunking helps improve retrieval performance. Therefore, using a preprocessing solution that supports TSR can help increase retrieval accuracy even for documents with many complex tables.
The relationship between document preprocessing quality and RAG performance
Document preprocessing quality determines the upper bound of overall RAG system performance, and errors introduced during extraction propagate 그대로 into chunking, retrieval, and generation.
Text without structural information is hard to split into meaningful units, and when context-broken chunks are retrieved, the LLM generates answers that don’t fit the context. In ChatDOC’s own comparative test, structure-aware parsing produced better results—47% better, 38% tied, and 15% worse—than general extraction. Ultimately, checking from the document extraction stage is the key to building RAG.
Building a RAG system: where should you start?
Pre-build checklist for RAG
Before building a RAG system, you should first review four items: the document formats to process, the security environment, infrastructure constraints, and the proportion of tables/images.
These four items are important criteria that determine which solution combination you need. By checking where your documents and environment fall before choosing a model, you can significantly reduce trial and error.
Four items to review before building RAG and what to check
| Review item | What to check |
| Document formats | Do you need to handle domestic formats such as HWP/HWPX? |
| Security environment | Is it a closed network environment where external transfer is restricted? |
| Infrastructure constraints | Do you need to run in a constrained environment, such as CPU-only? |
| Table/image proportion | Are there many complex tables/images, making AI pipeline integration important? |
If you’re in a closed network where it’s difficult to send documents outside, an on-premises approach installed directly on internal servers is a good fit. If you want to start with a small validation first, a SaaS approach billed per page works well for pilot use.
What to check when you get stuck on document preprocessing
If tables or Korean document structure break during document preprocessing, the cause is often not the RAG framework but the preprocessing solution that structures the input documents.
The hallucinations, table-structure loss, and reduced retrieval accuracy we’ve covered may look different on the surface, but they share one common cause: document preprocessing quality.
According to reporting by CIO Korea, RAG is drawing attention as a way to reduce LLM hallucinations, but extracting data from unstructured enterprise documents is often not easy. As a result, preprocessing technology that refines documents into AI-friendly forms is becoming increasingly important.
✅ Loss of table and hierarchical structure
If tables with merged cells or multi-level headers, or the hierarchy between titles and body text, are not properly distinguished in retrieval results, it’s highly likely that structure is being lost during extraction.
✅ Domestic document formats and closed-network environments
If HWP/HWPX make up a large share or you’re in an environment where external transfer is blocked, general-purpose document loaders have limits—so it’s worth considering a separate preprocessing solution.
💻 Hancom Data Loader
Hancom Data Loader is a document parsing solution that converts HWP, HWPX, PDF, and OOXML into structured data. Based on DLA, OCR, and TSR, it extracts not only text but also structural information such as tables, hierarchies, and coordinates, and it supports on-premises environments that do not transmit data externally.
In practice, in the Gyeonggi-do Office of Education AI Digital Platform 구축 project, Hancom Data Loader converted AI-trainable data from about 2,800 school websites, around 40,000 guidance materials, and about 7,000 guideline/legal datasets. Based on this data, HancomPedia is providing accurate QA to faculty and staff.
What matters in building RAG is not simply choosing a model, but verifying that your company’s documents are properly converted into data that AI can read.
You can start with document preprocessing that agents can trust—with Hancom Data Loader.
👉 Inquire About Hancom Data Loader
References
- arXiv, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” 2020
- IBM Think, “What is RAG (Retrieval Augmented Generation)?”
- AWS Prescriptive Guidance, “Understanding Retrieval Augmented Generation”
- Unstructured, “Partitioning”
- arXiv, “Enhancing Retrieval Augmented Generation with Hierarchical Text Segmentation Chunking”, 2025
- IBM Think, “Llamaindex vs Langchain: What’s the difference?”
- arXiv, “Structure-Aware Chunking for Tabular Data in Retrieval-Augmented Generation”, 2026
- arXiv, “Revolutionizing Retrieval-Augmented Generation with Enhanced PDF Structure Recognition,” 2024
- arXiv, “Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards”, 2025
- AI Times, “Hancom launches ‘Hancom Data Loader,’ which extracts AI data from documents”, 2024
- CIO Korea, “‘Extracting AI data from documents’… Hancom releases the Hancom Data Loader SDK globally”, 2024
- ChosunBiz, “Hancom participates in the Gyeonggi-do Office of Education AI Digital Platform project”
