Two judges, the same claims
rag-langchain scored its answers with google/gemini-2.5-flash as the judge. Here the same 161 claims from 40 answers (4 runs) were judged again by openai/gpt-4o-mini with RAGAS’s own verdict prompt, against the same numbered chunks. Where the two disagree, the chunk text decides who was right.
Where they agree
Rows are Gemini’s verdict, columns RAGAS’s. Blue cells are agreement; orange cells are the disagreements listed below. Most disagreements are claims Gemini accepted and RAGAS rejected: RAGAS’s judge is stricter, but stricter isn’t the same as right.
| openai/gpt-4o-mini (RAGAS prompt) | ||
|---|---|---|
| google/gemini-2.5-flash | supported | not supported |
| supported | 143 | 12 |
| not supported | 4 | 2 |
What it does to the score
The same answers, scored with each judge’s verdicts (supported ÷ claims per answer, averaged per run). This is how much a faithfulness score depends on which judge you pick. Checked uses the shared verdict where the judges agree, and the decision below where they don’t.
| Run | Answers | Gemini | RAGAS | Checked |
|---|---|---|---|---|
| top-k-a | 10 | 0.97 | 0.87 | 1.00 |
| top-k-b | 10 | 1.00 | 0.83 | 1.00 |
| two-per-artist | 10 | 1.00 | 0.85 | 1.00 |
| one-per-artist | 10 | 0.90 | 0.95 | 0.91 |
What went wrong
Each disagreement, grouped by the kind of mistake, and which judge made it.
| Pattern | Claims | Gemini wrong | RAGAS wrong |
|---|---|---|---|
| Tag list missed | 6 | 1 | 5 |
| Year in a release list missed | 3 | 3 | |
| “or” read as “and” | 2 | 2 | |
| Inference from a title | 2 | 1 | 1 |
| Is a tag enough? | 1 | 1 | |
| Outside knowledge used | 1 | 1 | |
| Nothing in the chunk | 1 | 1 |
Every disagreement
Decisions are in eval/cross-judge-labels.json. Clear cases were decided from the chunk text (the evidence line is quoted). Dashed cards are judgement calls, left for a person to decide.
#1 · top-k-a · Which classic soul or Motown singers are in the data? Tag list missed
“Marvin Gaye is a classic soul / Motown singer.”
✗ Gemini The context provides tags like 'motown' and 'soul' for Marvin Gaye, but it does not explicitly state he is a 'classic soul / Motown singer'.
✓ RAGAS The context describes Marvin Gaye as an artist associated with Motown and soul music.
Evidence: Tags: …; soul; …; motown; … (Marvin Gaye, chunk [2])
Decided: supported → RAGAS was right (claude, from the chunk text)
#2 · top-k-a · Which artists blend hip hop with R&B, and what are some of their albums? Tag list missed
“JAŸ-Z blends hip hop with R&B.”
✓ Gemini The context lists 'hip hop rnb and dance hall' as one of JAŸ-Z's tags.
✗ RAGAS The context does not provide specific information about JAŸ-Z's musical style or genre blending.
Evidence: Tags: …; hip hop rnb and dance hall; … (JAŸ-Z, chunk [3])
Decided: supported → Gemini was right (claude, from the chunk text)
#3 · top-k-a · Which artists blend hip hop with R&B, and what are some of their albums? Tag list missed
“Beyoncé blends hip hop with R&B.”
✓ Gemini The context lists 'hip hop rnb and dance hall' and 'hip hop soul' as tags for Beyoncé.
✗ RAGAS The context does not provide specific information about Beyoncé's musical style or genre blending.
Evidence: Tags: …; hip hop rnb and dance hall; hip hop soul; … (Beyoncé, chunk [6])
Decided: supported → Gemini was right (claude, from the chunk text)
#4 · top-k-a · When did Adele release 21? Year in a release list missed
“Adele released 21 in 2011.”
✓ Gemini The context block [1] lists '21 (2011)' under the artist Adele, indicating its release in that year.
✗ RAGAS The context mentions Adele's album '21' but does not specify the release year as 2011; it only states '21 (2011)' without confirming it as a release date.
Evidence: - 21 (2011) (Adele, chunk [1])
Decided: supported → Gemini was right (claude, from the chunk text)
#5 · top-k-b · Which classic soul or Motown singers are in the data? “or” read as “and”
“2. Aretha Franklin is a classic soul or Motown singer.”
✓ Gemini The context mentions that Aretha Franklin's tags include 'soul' and 'motown'.
✗ RAGAS The context identifies Aretha Franklin as the 'Queen of Soul' and associates her with soul music, which supports her classification as a classic soul singer, but does not explicitly mention Motown.
Evidence: Tags: …; soul; …; the queen of soul; … (Aretha Franklin, chunk [5]). Soul alone makes the “or” true.
Decided: supported → Gemini was right (claude, from the chunk text)
#6 · top-k-b · Which artists blend hip hop with R&B, and what are some of their albums? Tag list missed
“Beyoncé's tags include "hip hop rnb and dance hall".”
✓ Gemini The tags for Beyoncé include "hip hop rnb and dance hall".
✗ RAGAS The context does not provide any tags for Beyoncé, so this statement cannot be verified.
Evidence: Tags: …; hip hop rnb and dance hall; … (Beyoncé, chunk [6])
Decided: supported → Gemini was right (claude, from the chunk text)
#7 · top-k-b · Which artists blend hip hop with R&B, and what are some of their albums? Tag list missed
“Beyoncé's tags include "contemporary r&b".”
✓ Gemini The tags for Beyoncé include "contemporary r&b".
✗ RAGAS The context does not provide any tags for Beyoncé, so this statement cannot be verified.
Evidence: Tags: …; contemporary r&b; … (Beyoncé, chunk [6])
Decided: supported → Gemini was right (claude, from the chunk text)
#8 · top-k-b · When did Adele release 21? Year in a release list missed
“Adele released 21 in 2011.”
✓ Gemini The context states '21 (2011)' under the artist Adele.
✗ RAGAS The context mentions Adele's album '21' but does not specify the release year as 2011; it only states that she was active since 1988 and lists various releases without confirming the year for '21'.
Evidence: - 21 (2011) (Adele, chunk [1])
Decided: supported → Gemini was right (claude, from the chunk text)
#9 · two-per-artist · Which classic soul or Motown singers are in the data? “or” read as “and”
“Aretha Franklin is a classic soul or Motown singer.”
✓ Gemini Aretha Franklin's tags include 'soul' and 'motown' is not listed.
✗ RAGAS The context describes Aretha Franklin as the 'Queen of Soul' and associates her with soul music, which supports her classification as a classic soul singer, but does not explicitly mention Motown.
Evidence: Tags: …; soul; …; the queen of soul; … (Aretha Franklin, chunk [4]). Soul alone makes the “or” true.
Decided: supported → Gemini was right (claude, from the chunk text)
#10 · two-per-artist · When did Adele release 21? Year in a release list missed
“Adele released 21 in 2011.”
✓ Gemini The context states that Adele released "21 (2011)".
✗ RAGAS The context does not mention the album '21' being released by Adele in 2011; it only lists her releases without specifying '21'.
Evidence: - 21 (2011) (Adele, chunk [1])
Decided: supported → Gemini was right (claude, from the chunk text)
#11 · one-per-artist · Which artists in the data make Afrobeat or other African music? Tag list missed
“Prince makes Afro house music.”
✓ Gemini Prince's tags include "afro house".
✗ RAGAS There is no mention of 'Afro house' in the context related to Prince.
Evidence: Tags: …; afro house; … (Prince, chunk [6])
Decided: supported → Gemini was right (claude, from the chunk text)
#12 · one-per-artist · Which artists in the data make Afrobeat or other African music? Is a tag enough?
“Prince makes African music.”
✓ Gemini Prince's tags include "african".
✗ RAGAS While Prince is an American artist, the context does not explicitly state that he makes African music.
Evidence: Prince has the tag “african”. Does one crowd-sourced tag support “makes African music”?
Decided: not supported → RAGAS was right (the author's judgement call)
#13 · one-per-artist · Which artists are known for Indian film music or Bollywood? Outside knowledge used
“A. R. Rahman is known for Bollywood.”
✓ Gemini The context lists several releases with titles such as 'Bombay' and 'Rangeela', which are common Bollywood film titles, suggesting an association with Bollywood.
✗ RAGAS The context does not explicitly mention A. R. Rahman being known for Bollywood specifically, as it focuses on his film soundtracks without specifying the industry.
Evidence: The chunk is only release titles (Rangeela, Bombay, …). Nothing says film music or Bollywood; knowing those are Bollywood films is outside knowledge.
Decided: not supported → RAGAS was right (claude, from the chunk text)
#14 · one-per-artist · Which classic soul or Motown singers are in the data? Inference from a title
“Marvin Gaye is a classic soul or Motown singer.”
✗ Gemini The context provides a list of Marvin Gaye's releases but does not describe his musical genre or association with Motown.
✓ RAGAS The context lists Marvin Gaye as an artist associated with Motown, indicating his genre.
Evidence: No tags in this chunk. One release is “Motown Superstar Series, Volume 15 (1980)”. Does that support “soul or Motown singer”?
Decided: supported → RAGAS was right (the author's judgement call)
#15 · one-per-artist · Which classic soul or Motown singers are in the data? Inference from a title
“Aretha Franklin is a classic soul or Motown singer.”
✗ Gemini The context lists Aretha Franklin's releases but does not state her musical genre or connection to Motown.
✓ RAGAS The context lists Aretha Franklin as an artist associated with soul music, confirming her genre.
Evidence: No tags in this chunk. Releases include “Queen of Soul (1968)” and “Lady Soul (1968)”. Does an album title support “soul singer”?
Decided: not supported → Gemini was right (the author's judgement call)
#16 · one-per-artist · Which classic soul or Motown singers are in the data? Nothing in the chunk
“Stevie Wonder is a classic soul or Motown singer.”
✗ Gemini The context provides a list of Stevie Wonder's releases but does not describe his musical genre or association with Motown.
✓ RAGAS The context lists Stevie Wonder as an artist associated with Motown, indicating his genre.
Evidence: No tags in this chunk, and no release mentions soul or Motown.
Decided: not supported → Gemini was right (claude, from the chunk text)
Can we trust it when they agree?
Two judges agreeing doesn’t make them right. 20 agreed claims were checked against the chunks: the 2 that both judges rejected, plus 18 drawn at random from the ones both accepted (seed 42, eval/sample_agreements.py). 19 of 20 were right. Where they were wrong, both judges made the same mistake.
| Claim | Evidence in the chunks | |
|---|---|---|
| ✗ | Aretha Franklin is a classic soul / Motown singer. top-k-a/soul/5 · both judges: not supported | Tags: …; soul; …; the queen of soul; … (Aretha Franklin, chunk [5]). Nothing says Motown; “/” read as “or”, as in the “soul or Motown” decisions. (the author's judgement call: “/” = “or”) |
| ✓ | Michael Jackson is a classic soul or Motown singer. one-per-artist/soul/3 · both judges: not supported | Michael Jackson’s only chunk is a release list; nothing mentions soul or Motown. |
| ✓ | JAŸ-Z's album is The Blueprint (2001). top-k-a/hiphop-rnb/6 · both judges: supported | - The Blueprint (2001) (JAŸ-Z, chunk [2]) |
| ✓ | BTS is a South Korean boy group. top-k-a/kpop/3 · both judges: supported | Description: South Korean boy group (BTS, chunk [2]) |
| ✓ | Beyoncé is an artist. top-k-b/hiphop-rnb/9 · both judges: supported | Artist: Beyoncé (continued) (chunk [6]) |
| ✓ | JAŸ-Z is tagged with "hip hop rnb and dance hall". top-k-b/hiphop-rnb/1 · both judges: supported | Tags: …; hip hop rnb and dance hall; … (JAŸ-Z, chunk [3]) |
| ✓ | Arijit Singh is known for Bollywood. top-k-b/bollywood/5 · both judges: supported | No tags. Two titles say Bollywood (“Bollywood Nonstop Dandiya 2023”, “Brand Bollywood Downunder”) and 36 songs are marked “(From <film>)”. (the author's judgement call) |
| ✓ | Fela Kuti's tags include "afro-beat". top-k-b/afrobeat/1 · both judges: supported | Tags: …; afro-beat; … (Fela Kuti, chunk [1]) |
| ✓ | JAŸ-Z's album is Vol. 3… Life and Times of S. Carter (1999). top-k-a/hiphop-rnb/4 · both judges: supported | - Vol. 3… Life and Times of S. Carter (1999) (JAŸ-Z, chunk [2]) |
| ✓ | Stromae is from Belgium. one-per-artist/stromae/0 · both judges: supported | Country: BE (Belgium) (Stromae, chunk [1]) |
| ✓ | Aretha Franklin is known as "the queen of soul". top-k-a/soul/7 · both judges: supported | Tags: …; the queen of soul; … (Aretha Franklin, chunk [5]) |
| ✓ | Fela Kuti is Nigerian. one-per-artist/afrobeat/2 · both judges: supported | Country: NG (Nigeria); Tags: …; nigerian; … (Fela Kuti, chunk [1]) |
| ✓ | J Balvin is a Latin music artist. top-k-a/latin/0 · both judges: supported | Tags: …; latin urban; …; latin pop; …; latin (J Balvin, chunk [1]) |
| ✓ | BTS is from South Korea. top-k-a/kpop/4 · both judges: supported | Country: KR (South Korea) (BTS, chunk [2]) |
| ✓ | JAŸ-Z's album is Reasonable Doubt (1996). top-k-a/hiphop-rnb/1 · both judges: supported | - Reasonable Doubt (1996) (JAŸ-Z, chunk [1]) |
| ✓ | Lata Mangeshkar is referred to as "The Nightingale Of India" in the release titled "The Golden Voices Of Bollywood - Volume 1". top-k-b/bollywood/3 · both judges: supported | - The Golden Voices Of Bollywood - Volume 1 - The Nightingale Of India (2007) (Lata Mangeshkar, chunk [4]) |
| ✓ | Some of Arijit Singh's film songs include "Kesariya (From 'Brahmastra')", "Chaleya (From 'Jawan')", and "What Jhumka? (From 'Rocky Aur Rani Kii Prem Kahaani')". top-k-b/bollywood/7 · both judges: supported | - Kesariya (From “Brahmastra”), - Chaleya (From “Jawan”), - What Jhumka? (From “Rocky Aur Rani Kii Prem Kahaani”) (Arijit Singh, chunks [5] and [6]) |
| ✓ | Arijit Singh is known for Bollywood. one-per-artist/bollywood/4 · both judges: supported | No tags. Two titles say Bollywood and 9 songs are marked “(From <film>)” (Arijit Singh, chunk [2]). (the author's judgement call) |
| ✓ | Bad Bunny is a Latin music artist. top-k-b/latin/1 · both judges: supported | Tags: …; latin; …; latin urban; latin trap; … (Bad Bunny, chunk [2]) |
| ✓ | Fela Kuti makes Afro-beat music. one-per-artist/afrobeat/1 · both judges: supported | Tags: …; afrobeat; …; afro-beat; … (Fela Kuti, chunk [1]) |
Limits
- Gemini wrote the claims. RAGAS’s claim-splitting step was skipped so the verdicts could be compared one to one, so every claim is phrased the way Gemini split the answer.
- One pass of the RAGAS judge. Its verdicts move between runs. The agreement rate is for this pass.
- Agreements were only sampled. 20 of 145 were checked, so the error rate among agreements is a rough estimate. The “checked” scores assume every agreement not in the sample is right.
- The four runs share questions, so the same claim (“Adele released 21 in 2011”) can count up to four times.