← Compare all runs

Evaluating the RAG pipeline

13 test questions, each run through the real pipeline and scored. Retrieval is scored with plain arithmetic. The answers are scored by a second LLM acting as judge, using metrics modelled on Ragas.

answers openai/gpt-5-mini judge google/gemini-2.5-flash embeddings openai/text-embedding-3-small run one-per-artist retrieval one-per-artist, top 6 chunks chunks 800 chars, 100 overlap run 2026-10-08 15:01 UTC · 81 s

Averages

retrieval 1.00 Context recall Share of the expected artists that appear in the retrieved chunks. No LLM. n = 10
retrieval 0.99 Context precision Are the chunks from expected artists ranked near the top? Rank-weighted. No LLM. n = 10
generation 0.90 Faithfulness Share of the answer’s claims that the retrieved chunks back up. Judge splits the answer into claims, then checks each. n = 10
generation 0.66 Answer relevancy Judge writes 3 questions the answer would fit; their embedding similarity to the real question. 0 if evasive. n = 10
generation 0.83 Correctness Judge compares the answer with our reference answer: correct 1, partial 0.5, incorrect 0. n = 9
generation 12/13 Refusal Did it refuse exactly when it should? Out-of-scope questions should be refused; answerable ones not. n = 13

Every question, every metric

Darker is better. “–” means the metric doesn’t apply: no expected artists, no reference answer, or a question that should be refused. Select a question to see what the judge saw and decided.

QuestionContext recallContext precisionFaithfulnessAnswer relevancyCorrectnessRefusal
genre
Which artists in the data make Afrobeat or other African music?1.001.001.000.760.00pass · answered
Which K-pop groups are in the data?1.001.001.000.781.00pass · answered
Which Latin music artists are in the data?1.001.001.000.841.00pass · answered
Which artists are known for Indian film music or Bollywood?1.001.001.000.731.00pass · answered
Which classic soul or Motown singers are in the data?1.000.920.000.430.50pass · answered
open
Which artists blend hip hop with R&B, and what are some of their albums?––1.000.00–fail · refused
fact
In what year did Fela Kuti release Zombie?1.001.001.000.691.00pass · answered
Which country is Stromae from?1.001.001.000.901.00pass · answered
When did Adele release 21?1.001.001.000.651.00pass · answered
Where are Daft Punk from, and since when have they been active?1.001.001.000.841.00pass · answered
trap
How many Grammy awards has Beyoncé won?1.001.00–––pass · refused
out-of-scope
What is the capital of Australia?–––––pass · refused
Who won the 2018 FIFA World Cup?–––––pass · refused
score 0–0.2 0.2–0.4 0.4–0.6 0.6–0.8 0.8–1

Reading these numbers

  • The judge is an LLM too, and can be wrong. Before trusting a score, read a few claim verdicts and see if you agree. Checking the judge against your own labels is the next step.
  • Faithfulness isn’t correctness. An answer can stick perfectly to the retrieved chunks and still be incomplete, if retrieval missed something.
  • Small numbers. 13 questions is enough to spot problems, not to compare two setups that differ by a few points.
  • Answers vary between runs. The same question can get a slightly different answer, and different scores, next time.