Two ways to evaluate the same RAG pipeline
The same music RAG pipeline (top 6 chunks, answers by gpt-5-mini) was evaluated twice. A is rag-langchain’s own evaluation: RAGAS’s ideas reimplemented in TypeScript, judged by google/gemini-2.5-flash. B is this project: the real RAGAS library in Python, judged by openai/gpt-4o-mini. This report compares the two processes: what each measures, what each found and missed, and how far either judge can be trusted.
In short
- A measures more; B only measures generation. A scores retrieval without an LLM (context recall and precision against expected artists) and four answer metrics. B scores faithfulness and refusal only. B’s answers were faithful but incomplete, and B couldn’t see it.
- Each process caught something the other missed. A found that one artist crowding the top 6 caused every retrieval miss. B found a mislabelled field (birth dates shown as “Active since”) that faithfulness passes as fully supported.
- The judge matters more than the library. On the same 161 claims, the judges agree on 90%, and Gemini was right on 12 of the 16 disagreements. The choice of judge moves faithfulness by up to 0.17, more than the 0.10 between A’s retrieval strategies, and reverses their ranking.
- Agreement is mostly trustworthy. In a sample of 20 claims both judges agreed on, 19 were right.
- Recommendation: keep A as the main evaluation, and take three things from B: the label fix, a rule for plain “I don’t know” answers, and a second judge run on a sample, with disagreements checked by hand.
At a glance
| A rag-langchain | B ragas-evaluation | |
|---|---|---|
| Built with | TypeScript only. The judge is called through LangChain with typed (zod) output. | TypeScript pipeline, Python RAGAS 0.4.3. One JSONL file sits between them. |
| Judge | google/gemini-2.5-flash (a different family from the answering model) | openai/gpt-4o-mini (also a different model from the answering one) |
| Retrieval metrics | Context recall and precision, plain arithmetic over expected artists | None |
| Answer metrics | Faithfulness, answer relevancy, correctness against a reference, refusal | Faithfulness, refusal |
| Test set | 13 questions, grouped (genre, fact, open, trap, out of scope), each with expected artists and a reference answer | 12 questions, each labelled “answer” or “refuse”, including 3 traps the model could answer from memory |
| Experiments | 4 runs: top-k twice (to measure noise), 2 per artist, 1 per artist | One run with the label fix, then the two-judge comparison on A’s runs |
| Setup | Node only | Node, plus uv with Python 3.12 or 3.13 (RAGAS doesn’t install on 3.14), pinned LangChain versions, analytics switched off |
| Output | A page per run and a comparison of runs | A step-by-step walkthrough, and the two-judge comparison |
How each process runs
A
- Questionswith expected artists and a reference answer
- Retrieve and answerthe real pipeline, with a chosen retrieval strategy
- Retrieval metricsrecall and precision: arithmetic, no LLM
- Judge calls, in parallelfaithfulness (2 calls), relevancy (1 call + embeddings), correctness, refusal
- One JSON file and page per runthen a page comparing the runs
B
- Questionslabelled “answer” or “refuse”
- Retrieve and answer (TypeScript)written to samples.jsonl in RAGAS’s field names
- Refusal check (Python)a rule for plain “I don’t know”, otherwise a RAGAS DiscreteMetric
- Faithfulness, answers onlyRAGAS: split into claims, then a verdict for each
- results.json and a walkthrough pageevery judge prompt and verdict kept
Faithfulness: the same idea, different details
Both split the answer into claims and check each claim against the retrieved chunks. Score = supported ÷ all. The details differ, and they matter:
| Detail | A | B (RAGAS) |
|---|---|---|
| “I don’t know” and “Artists used: …” | The claim prompt tells the judge to skip them, so a refusal has no claims and no score | Split into claims like any other text and scored at random (0, 0.5 or 1), so B only runs faithfulness on answers |
| Where the support is | The judge names the numbered chunk that supports each claim | A reason only, no chunk number |
| The chunks the judge sees | Numbered, exactly as in the answer prompt | Joined into one text |
| Prompt | Short, our own wording | RAGAS’s, with worked examples (about 3,000 characters before the chunks) |
What each process found, and missed
| Finding | Found by | Why the other missed it |
|---|---|---|
| Every retrieval miss was crowding: one artist filling several of the 6 slots | A | B has no expected artists, so it can’t measure recall. B’s own soul answer used Aretha Franklin in 5 of 6 slots and still scored 1.0. |
| Faithful but incomplete: answers stuck to what was retrieved, so only correctness dropped | A | B has no reference answers, so it has no correctness metric. |
| Answer relevancy measured the metric more than the answers | A | B doesn’t run answer relevancy. |
| Capping chunks per artist trades depth for breadth | A | B ran one strategy only. |
| “Active since” showed a person’s birth date as their career start, and faithfulness passed it | B | A’s only date question is about Daft Punk, a group, for whom the label is correct. Only a question about a person’s career exposes it. |
| RAGAS scores “I don’t know” answers at random | B | A’s own claim prompt already avoids it. |
| A small judge misreads plain “I don’t know”; a rule is more reliable | B | A’s Gemini judge got the refusals right (13/13 in the first run). |
| Judges are noisy: the same answer gets different scores across runs | A B | A ran top-k twice; B rescored the same answers. |
| Judges hesitate to turn tags into a category | B | Seen in the two-judge comparison of A’s answers: Gemini rejected “Marvin Gaye is a classic soul / Motown singer” with both tags in the chunk, and both judges rejected Aretha Franklin’s “soul / Motown” with her soul tags in the chunk. |
Scores as each process reported them
A per run:
| Run | Recall | Precision | Faithfulness | Relevancy | Correctness | Refusal right |
|---|---|---|---|---|---|---|
| top-k-a | 0.90 | 1.00 | 0.97 | 0.63 | 0.83 | 13/13 |
| top-k-b | 0.90 | 1.00 | 1.00 | 0.66 | 0.83 | 13/13 |
| two-per-artist | 0.96 | 1.00 | 1.00 | 0.69 | 0.89 | 12/13 |
| one-per-artist | 1.00 | 0.99 | 0.90 | 0.66 | 0.83 | 12/13 |
B: faithfulness 0.96 over 7 answers; refusal right 12/12, 5 of them decided by the rule. The questions differ from A’s, so these numbers can’t be compared with A’s directly. The next section can.
How far can the judge be trusted?
To compare the judges directly, the 161 claims from A’s 40 answers were judged again by B’s judge with RAGAS’s prompt, against the same chunks. Every disagreement was settled from the chunk text, or by hand where it was a judgement call. Full comparison →
90%same verdict (145 of 161)
12 – 4right on the 16 disagreements: Gemini – RAGAS
19 / 20sampled agreements that were right
| A’s run | Gemini’s verdicts | RAGAS’s verdicts | Checked |
|---|---|---|---|
| top-k-a | 0.97 | 0.87 | 1.00 |
| top-k-b | 1.00 | 0.83 | 1.00 |
| two-per-artist | 1.00 | 0.85 | 1.00 |
| one-per-artist | 0.90 | 0.95 | 0.91 |
- The judge moves the score more than the strategies do. Gemini’s faithfulness per run is 0.90–1.00; RAGAS’s is 0.83–0.95. Gemini puts one-per-artist last; RAGAS puts one-per-artist first. A conclusion like “strategy X is more faithful” needs the judge named next to it.
- The checked scores are close to Gemini’s. RAGAS’s judge mostly erred by rejecting facts in plain sight: release years, values in tag lists, and reading “or” as “and”.
- What counts as “supported” is a choice. Is one crowd-sourced tag, or an album title, evidence? No judge prompt settles that. Those cases were decided by hand and recorded with the reason.
Recommendation
- Keep A as the main evaluation. It measures retrieval as well as generation, has the more reliable judge, and names the chunk behind each verdict, which makes checking quick.
- Port the label fix to rag-langchain (
Born:for people,Formed:for groups), reindex, and rerun. Its published runs still use “Active since”. - Add B’s kind of questions to A’s set: a person’s career start, and traps the model could answer from memory (members, sales). Ask some questions that want more detail (“and their albums”), not just “which X are in the data”: they test retrieval depth.
- Add a rule for plain “I don’t know” answers before the refusal judge, so a judge change can’t break it.
- Use a second judge as a check, not a second score. Run it on a sample, read the disagreements, and report scores with the judge’s name. Treat differences smaller than the judge noise (about 0.03 between A’s two identical top-k runs) as no difference.
- Use B for learning, not for the main number. The RAGAS library is a useful reference implementation and a ready second judge, but its default prompts with a small model were the least reliable part of either process.
Limits
- Small numbers: 13 and 12 questions, 161 claims, one pass of B’s judge on A’s claims, 20 sampled agreements.
- In the two-judge comparison Gemini wrote the claims, which may suit Gemini’s verdicts. RAGAS’s claim-splitting step was skipped so the verdicts could be compared one to one.
- B’s numbers come from a different question set and a fixed label, so A’s and B’s scores in the tables above aren’t directly comparable. Only the two-judge comparison uses identical inputs.
- The judgement calls reflect one person’s reading of “supported”, recorded with their reasons in
eval/cross-judge-labels.jsonandeval/agreement-sample.json.
Pages: A’s run comparison · B’s walkthrough · Two-judge comparison