RAG Evaluation

You are an AI/ML engineer specializing in Retrieval-Augmented Generation (RAG) systems. The user wants to evaluate RAG pipelines using RAGAS (RAG Assessment) metrics and establish benchmarks for retrieval quality, generation quality, and end-to-end performance.

What to check first

• Install ragas library: pip install ragas datasets (requires Python 3.9+)
• Verify you have an LLM API key configured (OpenAI, Anthropic, or local LLM) via environment variables
• Prepare a dataset with: questions, ground_truth_answers, contexts, and generated_answers fields in HuggingFace Dataset format

Steps

1. Import RAGAS metrics: from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
2. Load your evaluation dataset using datasets.load_dataset() or create one from dict with the four required columns
3. Initialize your LLM evaluator with LangchainLLMWrapper or compatible evaluator class for metric computation
4. Set the embeddings model explicitly via ragas.run_evaluation() parameter (e.g., HuggingFace embeddings)
5. Call evaluate() on your dataset with selected metrics to compute scores for each sample
6. Aggregate results to get mean scores across the dataset—these become your baseline benchmarks
7. Log metrics to tracking system (Weights & Biases, MLflow) to monitor RAG performance over iterations
8. Compare against baseline thresholds: typically aim for answer_relevancy > 0.7, context_precision > 0.6

Code

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset
from langchain_openai import ChatOpenAI
from langchain_openai.embeddings import OpenAIEmbeddings
import os

# Sample evaluation data
data = {
    "question": [
        "What is the capital of France?",
        "How does photosynthesis work?",
    ],
    "answer": [
        "Paris is the capital of France.",
        "Photosynthesis converts light energy into chemical energy.",
    ],
    "contexts": [
        [["Paris is the capital and most populous city of France."]],
        [["Photosynthesis is a process used by plants to convert light energy."]],
    ],
    "ground_truth": [
        "Paris",
        "Light energy is converted to chemical energy through chlorophyll.",
    ],
}

# Create Dataset
dataset = Dataset.from_dict(data)

# Initialize LLM for metric computation
llm = ChatOpenAI(
    model="gpt-4-turbo",
    api_key=os.getenv("OPENAI_API_KEY"),
    temperature=0,
)

# Initialize embeddings
embeddings = Open

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Forgetting to handle rate limits — Anthropic returns 429 errors that need exponential backoff
• Hardcoding the model name in 50 places — use a single config so you can swap models in one place
• Not setting a timeout on API calls — a hanging request can lock your worker indefinitely
• Logging API responses with sensitive data — PII can end up in your logs without realizing
• Treating the API as deterministic — same prompt, different output. Test on multiple runs

When NOT to Use This Skill

• For deterministic tasks where regex or rule-based code would work — LLMs add cost and latency for no benefit
• When you need 100% accuracy on a known schema — use structured output APIs or fine-tuning instead
• For real-time low-latency applications under 100ms — even the fastest LLM is too slow

How to Verify It Worked

• Test with malformed inputs, empty strings, and edge cases — APIs often behave differently than docs suggest
• Verify your error handling on all 4xx and 5xx responses — most code only handles the happy path
• Run a load test with 10x your expected traffic — rate limits hit fast
• Check token usage matches your estimate — surprises here become surprises on your bill

Production Considerations

• Set a daily spend cap on your Anthropic console — prevents runaway costs from bugs or attacks
• Use prompt caching for static parts of your prompts — can cut costs by 50-90%
• Stream responses for any user-facing output — perceived latency drops by 70%
• Have a fallback model ready — if Claude is down, you should be able to swap to a backup with one config change