StructRAG: Succeeding Where Graph RAG Fails

Enhancing Knowledge-Intensive Reasoning in LLMs through Information Structurization

Effectively leveraging external knowledge to enhance the performance of large language models (LLMs) on knowledge-based tasks has become a crucial challenge. Retrieval-augmented generation (RAG) methods have emerged as a promising solution, providing LLMs with relevant information from external sources to improve their factual accuracy and reasoning capabilities. However, existing RAG approaches struggle with knowledge-intensive reasoning tasks, where the required information is often scattered across multiple documents, making it difficult to accurately identify and integrate key pieces of information for global reasoning.

Inspired by cognitive theories suggesting that humans convert raw information into structured knowledge when tackling complex reasoning tasks, researchers from the Chinese Academy of Sciences and Alibaba Group have proposed a new framework called StructRAG. This framework introduces a hybrid information structurization mechanism that constructs and utilizes structured knowledge in the most suitable format based on the specific requirements of the task at hand. By mimicking human-like thinking processes, StructRAG aims to enhance the performance of LLMs on knowledge-intensive reasoning tasks.

In this article, we will delve into the details of the StructRAG framework, exploring its key components and how they work together to improve RAG performance. We will also discuss the training process of the hybrid structure router, a critical module in StructRAG, and present the experimental results demonstrating the effectiveness of this approach.

Overview of the StructRAG framework. Source.

The Framework

The StructRAG framework consists of three main modules that work sequentially to identify the optimal structure type, construct structured knowledge, and utilize that knowledge for accurate reasoning. Let's take a closer look at each of these modules.

1. Hybrid Structure Router: The hybrid structure router is the core component of StructRAG, responsible for determining the most appropriate structure type for a given task. It takes the question and the core content of the documents as input and outputs the optimal structure type. The router considers five candidate structure types: table, graph, algorithm, catalogue, and chunk, each suited for different kinds of knowledge-intensive tasks.

The selection of the optimal structure type is crucial, as it directly impacts the effectiveness of the subsequent modules. To train the router, the authors propose a novel method based on the Decision Transformer with Preference Optimization (DPO) algorithm, which follows reinforcement learning principles without requiring additional reward models. The training data for the router is generated through a synthesizing-simulating-judging pipeline, which creates high-quality synthetic preference pairs for various tasks and structure types.

2. Scattered Knowledge Structurizer: Once the optimal structure type is identified, the scattered knowledge structurizer comes into play. This module is responsible for extracting relevant information scattered across the raw documents and reconstructing it into structured knowledge in the chosen format. The structurizer leverages the powerful understanding and generation capabilities of LLMs to perform this complex task.

The structurizer takes the question, the selected structure type, and each raw document as input. It then extracts the structured knowledge from the document and generates a description of the structured knowledge. The output structured knowledge is collected and combined to form the overall structured knowledge for the given task.

3. Structured Knowledge Utilizer: The final module in the StructRAG framework is the structured knowledge utilizer, which performs reasoning to answer the question based on the constructed structured knowledge. This module is designed to handle complex, combinatorial questions that may hinder the direct identification and utilization of relevant information.

The utilizer employs an LLM-based approach to facilitate question decomposition, precise knowledge extraction, and final answer inference. It first breaks down the original question into several simpler sub-questions based on the overall description of the structured knowledge. Then, it extracts precise knowledge for each sub-question from the structured knowledge. Finally, the utilizer integrates all the sub-questions and their corresponding precise knowledge to generate the final answer.

Training the Hybrid Structure Router

The performance of the hybrid structure router is critical to the overall effectiveness of the StructRAG framework. To train the router, the authors propose a novel method that combines a synthesizing-simulating-judging pipeline for generating training data and the DPO algorithm for preference training.