Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrievers

2404.07220

YC

0

Reddit

0

Published 4/12/2024 by Kunal Sawarkar, Abhilasha Mangal, Shivam Raj Solanki
Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrievers

Abstract

Retrieval-Augmented Generation (RAG) is a prevalent approach to infuse a private knowledge base of documents with Large Language Models (LLM) to build Generative Q&A (Question-Answering) systems. However, RAG accuracy becomes increasingly challenging as the corpus of documents scales up, with Retrievers playing an outsized role in the overall RAG accuracy by extracting the most relevant document from the corpus to provide context to the LLM. In this paper, we propose the 'Blended RAG' method of leveraging semantic search techniques, such as Dense Vector indexes and Sparse Encoder indexes, blended with hybrid query strategies. Our study achieves better retrieval results and sets new benchmarks for IR (Information Retrieval) datasets like NQ and TREC-COVID datasets. We further extend such a 'Blended Retriever' to the RAG system to demonstrate far superior results on Generative Q&A datasets like SQUAD, even surpassing fine-tuning performance.

Get summaries of the top AI research delivered straight to your inbox:

Overview

  • This paper introduces "Blended RAG", a novel approach to improving the accuracy of Retriever-Augmented Generation (RAG) models for question answering tasks.
  • The key innovations are the use of semantic search and hybrid query-based retrievers to enhance the performance of RAG models.
  • The authors demonstrate the effectiveness of their approach through experiments on various question answering benchmarks.

Plain English Explanation

The paper focuses on improving the performance of question-answering models that use a combination of retrieval and generation, known as Retriever-Augmented Generation (RAG) models. The authors propose a new approach called "Blended RAG" that incorporates two main improvements:

  1. Semantic Search: Instead of relying solely on keyword-based retrieval, Blended RAG uses semantic search, which looks for passages that are semantically related to the question, rather than just matching keywords. This helps the model find more relevant information to answer the question.

  2. Hybrid Query-Based Retrievers: Blended RAG uses a combination of different retrieval methods, including sparse (keyword-based) and dense (semantic) retrievers. This "hybrid" approach allows the model to take advantage of the strengths of both types of retrievers, leading to more accurate results.

By incorporating these two innovations, the authors demonstrate that Blended RAG can outperform traditional RAG models on various question-answering benchmarks. This research is significant because it shows how combining different retrieval techniques can improve the performance of AI models that need to find relevant information to answer questions.

Technical Explanation

The paper introduces the "Blended RAG" approach, which builds upon the Retriever-Augmented Generation (RAG) model architecture. RAG models use a retriever component to find relevant information from a large knowledge base, and then a generator component to produce an answer based on the retrieved information.

The key innovations in Blended RAG are:

  1. Semantic Search: The authors replace the sparse (keyword-based) retriever in the original RAG model with a dense retriever that uses semantic search. This dense retriever encodes the question and the passages in the knowledge base into dense vector representations, and then uses a similarity metric to find the most relevant passages.

  2. Hybrid Query-Based Retrievers: Blended RAG combines the semantic dense retriever with a sparse (keyword-based) retriever. The outputs of these two retrievers are then blended together to provide the final set of retrieved passages to the generator component.

The authors evaluate Blended RAG on several question-answering datasets, including Natural Questions, WebQuestions, and HotpotQA. They find that Blended RAG outperforms the original RAG model, as well as other state-of-the-art question-answering systems.

Critical Analysis

The authors acknowledge several limitations of their work:

  • The performance of Blended RAG is still dependent on the quality of the underlying retrieval index and the knowledge base. Improving the coverage and accuracy of these components could further boost the model's performance.
  • The authors only evaluate Blended RAG on a limited set of question-answering datasets. Testing the model's generalization to other tasks or domains would help validate its broader applicability.
  • The paper does not provide a detailed analysis of the relative contributions of the sparse and dense retrievers in the hybrid approach. Understanding the optimal balance between these components could lead to further improvements.

Additionally, one could question whether the performance gains of Blended RAG justify the increased complexity and computational cost of the hybrid retriever. The trade-offs between model accuracy and efficiency should be carefully considered in practical applications.

Conclusion

The Blended RAG approach introduced in this paper represents a promising step forward in improving the accuracy of retriever-augmented generation models for question answering. By incorporating semantic search and a hybrid retriever, the authors have demonstrated that it is possible to enhance the information-gathering capabilities of these models, leading to more reliable and informative responses.

This research highlights the value of combining multiple retrieval techniques, as well as the importance of leveraging semantic information in addition to keyword-based search. As AI systems continue to play a growing role in information discovery and knowledge-intensive tasks, innovations like Blended RAG will be crucial in ensuring the reliability and robustness of these models.



This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Related Papers

๐Ÿ›ธ

DuetRAG: Collaborative Retrieval-Augmented Generation

Dian Jiao, Li Cai, Jingsheng Huang, Wenqiao Zhang, Siliang Tang, Yueting Zhuang

YC

0

Reddit

0

Retrieval-Augmented Generation (RAG) methods augment the input of Large Language Models (LLMs) with relevant retrieved passages, reducing factual errors in knowledge-intensive tasks. However, contemporary RAG approaches suffer from irrelevant knowledge retrieval issues in complex domain questions (e.g., HotPot QA) due to the lack of corresponding domain knowledge, leading to low-quality generations. To address this issue, we propose a novel Collaborative Retrieval-Augmented Generation framework, DuetRAG. Our bootstrapping philosophy is to simultaneously integrate the domain fintuning and RAG models to improve the knowledge retrieval quality, thereby enhancing generation quality. Finally, we demonstrate DuetRAG' s matches with expert human researchers on HotPot QA.

Read more

5/24/2024

โ†—๏ธ

RAG Does Not Work for Enterprises

Tilmann Bruckhaus (Strative.ai)

YC

0

Reddit

0

Retrieval-Augmented Generation (RAG) improves the accuracy and relevance of large language model outputs by incorporating knowledge retrieval. However, implementing RAG in enterprises poses challenges around data security, accuracy, scalability, and integration. This paper explores the unique requirements for enterprise RAG, surveys current approaches and limitations, and discusses potential advances in semantic search, hybrid queries, and optimized retrieval. It proposes an evaluation framework to validate enterprise RAG solutions, including quantitative testing, qualitative analysis, ablation studies, and industry case studies. This framework aims to help demonstrate the ability of purpose-built RAG architectures to deliver accuracy and relevance improvements with enterprise-grade security, compliance and integration. The paper concludes with implications for enterprise deployments, limitations, and future research directions. Close collaboration between researchers and industry partners may accelerate progress in developing and deploying retrieval-augmented generation technology.

Read more

6/10/2024

DR-RAG: Applying Dynamic Document Relevance to Retrieval-Augmented Generation for Question-Answering

New!DR-RAG: Applying Dynamic Document Relevance to Retrieval-Augmented Generation for Question-Answering

Zijian Hei, Weiling Liu, Wenjie Ou, Juyi Qiao, Junming Jiao, Zhiqing Zhu, Guowen Song

YC

0

Reddit

0

Retrieval-Augmented Generation (RAG) has significantly demonstrated the performance of Large Language Models (LLMs) in the knowledge-intensive tasks, such as Question-Answering (QA). RAG expands the query context by incorporating external knowledge bases to enhance the response accuracy. However, it would be inefficient to access LLMs multiple times for each query and unreliable to retrieve all the relevant documents by a single query. We find that even though there is low relevance between some critical documents and query, it is possible to retrieve the remaining documents by combining parts of the documents with the query. To mine the relevance, a two-stage retrieval framework called Dynamic-Relevant Retrieval-Augmented Generation (DR-RAG) is proposed to improve document retrieval recall and the accuracy of answers while maintaining efficiency. Also, a small classifier is applied to two different selection strategies to determine the contribution of the retrieved documents to answering the query and retrieve the relatively relevant documents. Meanwhile, DR-RAG call the LLMs only once, which significantly improves the efficiency of the experiment. The experimental results on multi-hop QA datasets show that DR-RAG can significantly improve the accuracy of the answers and achieve new progress in QA systems.

Read more

6/13/2024

๐Ÿงช

A Multi-Source Retrieval Question Answering Framework Based on RAG

Ridong Wu, Shuhong Chen, Xiangbiao Su, Yuankai Zhu, Yifei Liao, Jianming Wu

YC

0

Reddit

0

With the rapid development of large-scale language models, Retrieval-Augmented Generation (RAG) has been widely adopted. However, existing RAG paradigms are inevitably influenced by erroneous retrieval information, thereby reducing the reliability and correctness of generated results. Therefore, to improve the relevance of retrieval information, this study proposes a method that replaces traditional retrievers with GPT-3.5, leveraging its vast corpus knowledge to generate retrieval information. We also propose a web retrieval based method to implement fine-grained knowledge retrieval, Utilizing the powerful reasoning capability of GPT-3.5 to realize semantic partitioning of problem.In order to mitigate the illusion of GPT retrieval and reduce noise in Web retrieval,we proposes a multi-source retrieval framework, named MSRAG, which combines GPT retrieval with web retrieval. Experiments on multiple knowledge-intensive QA datasets demonstrate that the proposed framework in this study performs better than existing RAG framework in enhancing the overall efficiency and accuracy of QA systems.

Read more

5/30/2024