❓ Are You Doing RAG Right?

Find out with these best practices backed-up by science

In this issue:

1. RAG best practices

2. 1.000.000 LLM personas

3. Memory x Memory x Memory

   ---

LLM Watch is a reader-funded publication. However, I also offer limited slots for newsletter ads and Linkedin posts.

If you want to become a sponsor and get your product in front of an international tech audience, find more information below:

---

1. Searching for Best Practices in Retrieval-Augmented Generation

Watching: RAG Best Practices (paper)

What problem does it solve? Retrieval-augmented generation (RAG) techniques have shown great promise in enhancing the performance of large language models (LLMs) by integrating up-to-date information, reducing hallucinations, and improving response quality, especially in specialized domains. However, existing RAG approaches often suffer from complex implementations and prolonged response times, which can hinder their practical application. The multiple processing steps involved in a typical RAG workflow can be executed in various ways, leading to a wide range of possible combinations and configurations.

How does it solve the problem? To address the challenges associated with RAG approaches, the researchers conducted extensive experiments to identify optimal RAG practices that strike a balance between performance and efficiency. By investigating existing RAG approaches and their potential combinations, they aimed to streamline the RAG workflow and minimize the complexity of implementation. The study also explored the use of multimodal retrieval techniques to enhance question-answering capabilities when dealing with visual inputs. Additionally, they proposed a "retrieval as generation" strategy to accelerate the generation of multimodal content, further improving the efficiency of RAG-based systems.

What's next? The findings of this study provide valuable insights into the optimal deployment of RAG techniques, paving the way for more efficient and effective integration of retrieval-based methods with LLMs. As the demand for specialized and up-to-date information continues to grow, the development of streamlined RAG approaches will become increasingly important. Future research could focus on further refining these techniques, exploring new multimodal retrieval strategies, and investigating the potential of RAG in various domains, such as healthcare, finance, and education.

---

---

2. Scaling Synthetic Data Creation with 1,000,000,000 Personas

Watching: Persona Hub (paper/code)

What problem does it solve? Large Language Models (LLMs) have shown impressive capabilities in various tasks, but their performance is often limited by the diversity and quality of the training data. Generating high-quality, diverse synthetic data at scale remains a challenge, especially when it comes to capturing different perspectives and knowledge domains. This is where persona-driven data synthesis comes into play, leveraging the vast knowledge encapsulated within LLMs to create diverse and relevant synthetic data.

How does it solve the problem? The researchers propose a novel approach called Persona Hub, which automatically curates a collection of 1 billion diverse personas from web data. These personas act as distributed carriers of world knowledge, allowing the LLM to tap into various perspectives and generate synthetic data accordingly. By utilizing these personas, the LLM can create diverse and high-quality synthetic data for a wide range of scenarios, such as mathematical and logical reasoning problems, instructions, knowledge-rich texts, game NPCs, and tools (functions). This persona-driven approach ensures that the generated data is versatile, scalable, and flexible.

What's next? The introduction of Persona Hub and persona-driven data synthesis has the potential to drive a paradigm shift in synthetic data creation and its applications in practice. As this approach gains traction, we can expect to see more advanced and specialized personas being curated, leading to even more diverse and relevant synthetic data for various domains and tasks.

---

3. Memory³: Language Modeling with Explicit Memory

Watching: Memory³ (paper)

What problem does it solve? Training Large Language Models (LLMs) is expensive, both in terms of compute and storage. While a lot of the knowledge that gets encoded into the model parameters during training is explicit and could theoretically be stored more efficiently, the implicit knowledge that enables LLMs to generalize so well is what makes them powerful. Memory3 aims to reduce the cost of LLMs by externalizing explicit knowledge into a dedicated memory format that is cheaper than model parameters.

How does it solve the problem? Memory3 introduces a third form of memory in addition to the implicit knowledge stored in model parameters and the short-term working memory used during inference (context key-values). This explicit memory is designed to store factual knowledge more efficiently than model parameters. The researchers also developed techniques to make this approach feasible, including a memory sparsification mechanism to reduce storage requirements and a two-stage pretraining scheme to facilitate the formation of the explicit memory during training.

What's next? The results of Memory3 are showing that a 2.4B parameter model with explicit memory can outperform much larger models and maintain higher decoding speed than retrieval-augmented generation (RAG) approaches. However, this is just a preliminary proof of concept. Further research is needed to explore the full potential of this approach, such as scaling up the model size, optimizing the memory format and retrieval mechanisms, and evaluating its performance on a wider range of tasks. Additionally, the memory circuitry theory introduced in this paper could inspire new architectures and training techniques for more efficient and capable LLMs.

---

Papers of the Week:

• SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention

• ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models

• Investigating How Large Language Models Leverage Internal Knowledge to Perform Complex Reasoning

• xLSTM-UNet can be an Effective 2D & 3D Medical Image Segmentation Backbone with Vision-LSTM (ViL) better than its Mamba Counterpart

• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator

• ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos

• AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation

• GraCoRe: Benchmarking Graph Comprehension and Complex Reasoning in Large Language Models

---