← Compare all runs

Evaluating the RAG pipeline

13 test questions, each run through the real pipeline and scored. Retrieval is scored with plain arithmetic. The answers are scored by a second LLM acting as judge, using metrics modelled on Ragas.

answers openai/gpt-5-mini judge google/gemini-2.5-flash embeddings openai/text-embedding-3-small run long-context-a retrieval none: every profile in the prompt chunks 800 chars, 100 overlap run 2026-10-09 13:51 UTC · 56 s

Averages

retrieval n/a Context recall Share of the expected artists that appear in the retrieved chunks. No LLM. n = 0
retrieval n/a Context precision Are the chunks from expected artists ranked near the top? Rank-weighted. No LLM. n = 0
generation 1.00 Faithfulness Share of the answer’s claims that the retrieved chunks back up. Judge splits the answer into claims, then checks each. n = 10
generation 0.58 Answer relevancy Judge writes 3 questions the answer would fit; their embedding similarity to the real question. 0 if evasive. n = 10
generation 1.00 Correctness Judge compares the answer with our reference answer: correct 1, partial 0.5, incorrect 0. n = 9
generation 12/13 Refusal Did it refuse exactly when it should? Out-of-scope questions should be refused; answerable ones not. n = 13

Every question, every metric

Darker is better. “–” means the metric doesn’t apply: no expected artists, no reference answer, or a question that should be refused. Select a question to see what the judge saw and decided.

QuestionContext recallContext precisionFaithfulnessAnswer relevancyCorrectnessRefusal
genre
Which artists in the data make Afrobeat or other African music?––1.000.771.00pass · answered
Which K-pop groups are in the data?––1.000.501.00pass · answered
Which Latin music artists are in the data?––1.000.651.00pass · answered
Which artists are known for Indian film music or Bollywood?––1.000.841.00pass · answered
Which classic soul or Motown singers are in the data?––1.000.001.00pass · answered
open
Which artists blend hip hop with R&B, and what are some of their albums?––1.000.00–fail · refused
fact
In what year did Fela Kuti release Zombie?––1.000.911.00pass · answered
Which country is Stromae from?––1.000.351.00pass · answered
When did Adele release 21?––1.000.921.00pass · answered
Where are Daft Punk from, and since when have they been active?––1.000.891.00pass · answered
trap
How many Grammy awards has Beyoncé won?–––––pass · refused
out-of-scope
What is the capital of Australia?–––––pass · refused
Who won the 2018 FIFA World Cup?–––––pass · refused
score 0–0.2 0.2–0.4 0.4–0.6 0.6–0.8 0.8–1

Reading these numbers

  • The judge is an LLM too, and can be wrong. Before trusting a score, read a few claim verdicts and see if you agree. Checking the judge against your own labels is the next step.
  • Faithfulness isn’t correctness. An answer can stick perfectly to the retrieved chunks and still be incomplete, if retrieval missed something.
  • Small numbers. 13 questions is enough to spot problems, not to compare two setups that differ by a few points.
  • Answers vary between runs. The same question can get a slightly different answer, and different scores, next time.