In our last blog, we delved into the incredible potential of Retrieval-Augmented Generation (RAG) models. Yet, like any powerful technology, RAG comes with its own set of challenges. In this post, we’ll explore the complexities of RAG, highlighting the obstacles developers encounter when implementing and utilizing these models effectively. Stay tuned!

Data Ingestion: The Bedrock of a Strong RAG System

Data ingestion is crucial for designing an efficient RAG system. As we know, data preparation is key to building any AI model so is it for RAG. This stage involves:

• Extracting data
• Converting data into manageable chunks
• Convert chunks into machine-understandable vectors
• Storing these chunks and embeddings in a vector database

Challenge 1: Data Parsing 

Extracting data from documents (PDFs, images, spreadsheets, etc.) can be cumbersome and often difficult. Specialized techniques are needed to extract accurate and relevant information.

Solutions:

• Llama Parse: A tool by LlamaIndex that significantly enhances data extraction for RAG systems by effectively parsing complex documents.
• Chain-of-the-Table Approach: This technique helps in breaking down complex tables to pinpoint and extract specific data segments, improving tabular question-answering capabilities in RAG systems.
• Extracta.ai offers an API specifically designed to streamline data extraction, potentially simplifying data parsing. It can potentially handle various document formats, reducing the need for specialized techniques mentioned earlier (Llama Parse, Chain-of-the-Table Approach).
• Open-source libraries: These are freely available code resources, to create a data ingestion tool tailored to your unique needs. This gives developers greater control and customization over the data processing pipeline.

Challenge 2: Chunking Strategy

Determining the best way to chunk the document and deciding the size of each chunk is crucial. If chunks are too small, certain questions can’t be answered; if chunks are too large, the answers might include irrelevant information.

Solution:

• Chunking Strategy: Depending on the use case, different strategies such as sentence-based or paragraph-based chunking might be necessary. However, the best approach would be to chunk by topics but using the semantics of the documents to chunk is a very challenging task.

Ideal Chunking: Chunking documents by topic would be the most effective way to retrieve relevant information for answering user queries.

• Imagine a document about different animals. Chunking by topic (e.g., cats, dogs, birds) would allow the system to efficiently locate information about a specific animal (e.g., “What do cats eat?”).

However, accurately identifying topics based on document semantics (meaning) is a very difficult task in Natural Language Processing (NLP).

• Documents don’t always have clear topic headings or use explicit language.
• NLP needs to understand the underlying meaning and relationships between words and concepts within the text to identify topics accurately.

Ensuring Contextual Relevance

Ensuring the contextual relevance of retrieved data is critical for a robust RAG system. These systems sometimes fail to include essential documents in the top results returned by the retrieval component.

Challenge 3: Ranking

The system may sometimes overlook essential documents that contain the answers, which don’t appear in the top results.

Solution:

• Small-to-Big Sentence Window Retrieval
• Recursive Retrieval
• Semantic Similarity Scoring
• Document Hierarchies: Organize data in a structured manner to improve information retrieval by finding the most relevant text chunks.
• Knowledge Graphs: Represent related data through graphs for quick retrieval of relevant information, reducing hallucinations.
• Sub-document Summary: Breaking down documents into smaller chunks and adding summaries to improve retrieval performance by providing global context awareness.
• Parent Document Retrieval: Retrieve summaries and parent documents recursively to improve information retrieval and response generation.
• RAPTOR: Recursively embeds, clusters, and summarizes text chunks to build a tree structure with varying summarization levels.

Challenge 4: Missing Content:

Inadequate database content presents a core challenge, as it hampers the system’s capability to deliver accurate information. The lack of essential data results in incorrect responses, which can diminish user trust and satisfaction.

Solution: 

To proactively combat missing content, we employ two key strategies:

• Regular Extraction Review: We continuously evaluate and refine our data extraction strategies to guarantee we capture all critical information during the ingestion process.
• Frequent Data Imports: By maintaining a regular data refresh schedule, we ensure the system has access to the most up-to-date information as soon as it’s available. This minimizes the chance of outdated data influencing responses.

Indexing Strategy

Addressing the challenge of semantic matching is essential by seeking documents and information that are conceptually aligned with the user query, not just keyword matches.

Challenge 5: Indexing Strategy

Solutions:

• Hybrid Search: Combines semantic and keyword searches to ensure the retrieval of the most relevant documents.
  • Semantic Search: Considers document meaning and context for accurate results, going beyond keywords.
  • Keyword Search: Ideal for queries with specific terms like product codes, jargon, or dates.

Real-World Applications and Insights: Case Studies

To better understand these challenges, let’s look at three case studies from a recent paper https://arxiv.org/pdf/2401.05856:

The five challenges can arise within each case study but we have highlighted a few challenges to give you a grasp of what these challenges mean

1. Cognitive Reviewer
   • Objective: Support researchers in analyzing scientific documents by ranking them according to a specified research question.
   • Challenges: Indexing at runtime, handling a robust data processing pipeline, and sorting documents using a ranking algorithm.
   • Usage: Used by PhD students for literature reviews.

Highlighted challenge: Ranking- Missing Top-Ranked Documents (ranking algorithm limitations)

• What it means: The ranking algorithm might prioritize documents that don’t perfectly answer the research question, causing relevant papers to be overlooked.

2. AI Tutor
   • Objective: Answer student questions using indexed learning content.
   • Challenges: Indexing content from PDFs, videos, and text documents; transcribing videos using the deep learning model Whisper.
   • Pilot: Implemented for a unit with 200 students, integrating query rewriting based on previous dialogues.

Highlighted challenge:  Chunking strategy (adapting to different content)

• What it means: The ideal chunk size for the AI Tutor might depend on the learning content. Definitions might be best answered with smaller chunks (sentences), while explanations might require larger chunks (paragraphs) to capture sufficient context.

3. Biomedical Question and Answer
   • Objective: Address large-scale issues using the BioASQ dataset, comprising questions, document links, and answers.
   • Challenges: Handling domain-specific datasets, evaluating generated questions using the OpenEvals technique, and ensuring the accuracy of retrieved information.
   • Evaluation: Found that automated evaluation was more pessimistic than human raters, highlighting the complexity of the domain.

Highlighted challenge: Missing Content (limitations of the BioASQ dataset)

• What it means: The BioASQ dataset might not encompass all the information needed to answer every biomedical question perfectly. If the answer requires specific details not included in the dataset, the system might struggle to provide a complete and accurate response.

Conclusion

Designing a RAG system is a complex endeavor, requiring more than just fine-tuning language model parameters. The accuracy and reliability of a RAG system depend on critical factors like data ingestion, precise search repository indexing, and ensuring the contextual relevance of retrieved information. We’ve outlined some of the many challenges involved.

In our next blog series, we will dive deeper into one of the challenges

 Stay tuned for more on crafting a truly robust RAG framework.

Your thoughts matter to us! Have any questions or insights? Let’s discuss!

For any queries, you can contact us at support@emlylabs.com