Comparing retrieval strategies
The same 13 test questions, run through the pipeline with different ways of picking the top 6 chunks (or none), and scored by the same judge (google/gemini-2.5-flash).
- top-k: the 6 nearest chunks. One artist can fill several slots.
- two-per-artist: the nearest chunks, at most 2 from any artist.
- one-per-artist: the nearest chunk from each of 6 different artists.
- rerank (the ■ squares): the 30 nearest chunks, scored 0–3 for usefulness by an LLM (
gpt-4o-mini); the best 6 are kept. - long-context (the ◆ diamonds): no search at all. Every artist’s whole profile goes into the prompt, about 28k tokens. The baseline that shows what retrieval adds; its retrieval isn’t scored.
top-k-atop-k-btwo-per-artistone-per-artistrerank-arerank-brerank-clong-context-along-context-b
Each run’s full results, with every claim and the judge’s verdict on it: top-k-a · top-k-b · two-per-artist · one-per-artist · rerank-a · rerank-b · rerank-c · long-context-a · long-context-b
Average scores
One row per metric, one dot per run (nudged up or down so equal scores stay visible). The grey band spans the runs that used the same strategy: how much the scores move when nothing changes. Differences inside that band are noise.
Show as a table
| Metric | top-k-a | top-k-b | two-per-artist | one-per-artist | rerank-a | rerank-b | rerank-c | long-context-a | long-context-b |
|---|---|---|---|---|---|---|---|---|---|
| Context recall | 0.90 | 0.90 | 0.96 | 1.00 | 0.93 | 0.93 | 0.93 | – | – |
| Context precision | 1.00 | 1.00 | 1.00 | 0.99 | 1.00 | 1.00 | 1.00 | – | – |
| Faithfulness | 0.97 | 1.00 | 1.00 | 0.90 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| Answer relevancy | 0.63 | 0.66 | 0.69 | 0.66 | 0.67 | 0.63 | 0.70 | 0.58 | 0.78 |
| Correctness | 0.83 | 0.83 | 0.89 | 0.83 | 0.89 | 0.89 | 0.89 | 1.00 | 0.89 |
| Refusal right | 1.00 | 1.00 | 0.92 | 0.92 | 1.00 | 0.85 | 1.00 | 0.92 | 1.00 |
Question by question
score 0–0.2 0.2–0.4 0.4–0.6 0.6–0.8 0.8–1 · “–” doesn’t apply
What each strategy retrieved
The artists behind the 6 retrieved chunks. ×n means n chunks from the same artist, ✓ marks an expected artist, and the line below names expected artists that didn’t come back.
| Question | top-k-a | top-k-b | two-per-artist | one-per-artist | rerank-a | rerank-b | rerank-c | long-context-a | long-context-b |
|---|---|---|---|---|---|---|---|---|---|
| Which artists in the data make Afrobeat or other African music? | ✓ Fela Kuti ×3✓ Burna Boy ×2✓ Wizkid answer: correct | ✓ Fela Kuti ×3✓ Burna Boy ×2✓ Wizkid answer: correct | ✓ Fela Kuti ×2✓ Burna Boy ×2✓ Wizkid ×2 answer: correct | ✓ Fela Kuti✓ Burna Boy✓ WizkidBeyoncéAretha FranklinPrince answer: incorrect | ✓ Fela Kuti ×3✓ Burna Boy ×2✓ Wizkid answer: correct | ✓ Fela Kuti ×2✓ Burna Boy ×2✓ Wizkid ×2 answer: correct | ✓ Fela Kuti ×2✓ Burna Boy ×2✓ Wizkid ×2 answer: correct | all 51 artists, no search answer: correct | all 51 artists, no search answer: partial |
| Which K-pop groups are in the data? | ✓ BLACKPINK ×2✓ BTS ×3Kanye West Tribute Band answer: correct | ✓ BLACKPINK ×2✓ BTS ×3Kanye West Tribute Band answer: correct | ✓ BLACKPINK ×2✓ BTS ×2Kanye West Tribute BandKendrick Lamar answer: correct | ✓ BLACKPINK✓ BTSKanye West Tribute BandKendrick LamarBeyoncéKishore Kumar answer: correct | ✓ BLACKPINK ×2✓ BTS ×3Kanye West Tribute Band answer: correct | ✓ BLACKPINK ×2✓ BTS ×3Kanye West Tribute Band answer: correct | ✓ BLACKPINK ×2✓ BTS ×3Kanye West Tribute Band answer: correct | all 51 artists, no search answer: correct | all 51 artists, no search answer: correct |
| Which Latin music artists are in the data? | ✓ J Balvin ×4✓ Bad Bunny✓ Daddy Yankee missing: Shakira answer: partial | ✓ J Balvin ×4✓ Bad Bunny✓ Daddy Yankee missing: Shakira answer: partial | ✓ J Balvin ×2✓ Bad Bunny✓ Daddy Yankee ×2✓ Shakira answer: correct | ✓ J Balvin✓ Bad Bunny✓ Daddy Yankee✓ ShakiraCardi BBruno Mars answer: correct | ✓ J Balvin ×2✓ Bad Bunny✓ Daddy Yankee✓ Shakira ×2 answer: correct | ✓ J Balvin ×2✓ Bad Bunny✓ Daddy Yankee✓ Shakira ×2 answer: correct | ✓ J Balvin ×2✓ Bad Bunny✓ Daddy Yankee✓ Shakira ×2 answer: correct | all 51 artists, no search answer: correct | all 51 artists, no search answer: correct |
| Which artists are known for Indian film music or Bollywood? | ✓ A. R. Rahman ×2✓ Arijit Singh ×3✓ Lata Mangeshkar missing: Kishore Kumar, Diljit Dosanjh answer: partial | ✓ A. R. Rahman ×2✓ Arijit Singh ×3✓ Lata Mangeshkar missing: Kishore Kumar, Diljit Dosanjh answer: partial | ✓ A. R. Rahman ×2✓ Arijit Singh ×2✓ Lata Mangeshkar ×2 missing: Kishore Kumar, Diljit Dosanjh answer: partial | ✓ A. R. Rahman✓ Arijit Singh✓ Lata Mangeshkar✓ Kishore Kumar✓ Diljit DosanjhPrince answer: correct | ✓ A. R. Rahman ×3✓ Arijit Singh ×2✓ Lata Mangeshkar missing: Kishore Kumar, Diljit Dosanjh answer: partial | ✓ A. R. Rahman ×3✓ Lata Mangeshkar ×2✓ Arijit Singh missing: Kishore Kumar, Diljit Dosanjh answer: partial | ✓ A. R. Rahman ×2✓ Arijit Singh ×3✓ Lata Mangeshkar missing: Kishore Kumar, Diljit Dosanjh answer: partial | all 51 artists, no search answer: correct | all 51 artists, no search answer: partial |
| Which classic soul or Motown singers are in the data? | ✓ Marvin Gaye ×3✓ Aretha Franklin ×3 missing: Stevie Wonder answer: partial | ✓ Marvin Gaye ×3✓ Aretha Franklin ×3 missing: Stevie Wonder answer: partial | ✓ Marvin Gaye ×2✓ Aretha Franklin ×2Frank Sinatra✓ Stevie Wonder answer: partial | ✓ Marvin Gaye✓ Aretha FranklinFrank Sinatra✓ Stevie WonderElvis PresleyMichael Jackson answer: partial | ✓ Marvin Gaye ×3✓ Aretha Franklin ×3 missing: Stevie Wonder answer: partial | ✓ Marvin Gaye✓ Aretha Franklin ×4Whitney Houston missing: Stevie Wonder answer: partial | ✓ Marvin Gaye ×3✓ Aretha Franklin ×3 missing: Stevie Wonder answer: partial | all 51 artists, no search answer: correct | all 51 artists, no search answer: correct |
| Which artists blend hip hop with R&B, and what are some of their albums? | JAŸ-Z ×3Kendrick LamarRihannaBeyoncé | JAŸ-Z ×3Kendrick LamarRihannaBeyoncé | JAŸ-Z ×2Kendrick LamarRihannaBeyoncéDrake | JAŸ-ZKendrick LamarRihannaBeyoncéDrakeBurna Boy | Kendrick Lamar ×2Rihanna ×2BeyoncéDrake | Kendrick Lamar ×2Rihanna ×2BeyoncéDrake | JAŸ-Z ×3Kendrick LamarRihannaBeyoncé | all 51 artists, no search | all 51 artists, no search |
| In what year did Fela Kuti release Zombie? | ✓ Fela Kuti ×3Burna Boy ×2Wizkid answer: correct | ✓ Fela Kuti ×3Burna Boy ×2Wizkid answer: correct | ✓ Fela Kuti ×2Burna Boy ×2Wizkid ×2 answer: correct | ✓ Fela KutiBurna BoyWizkidJAŸ-ZDaft PunkEminem answer: correct | ✓ Fela Kuti ×3Burna Boy ×2Wizkid answer: correct | ✓ Fela Kuti ×3Burna Boy ×2Wizkid answer: correct | ✓ Fela Kuti ×3Burna Boy ×2Wizkid answer: correct | all 51 artists, no search answer: correct | all 51 artists, no search answer: correct |
| Which country is Stromae from? | ✓ Stromae ×2Bruno MarsDaft PunkStevie WonderThe Weeknd answer: correct | ✓ Stromae ×2Bruno MarsDaft PunkStevie WonderThe Weeknd answer: correct | ✓ Stromae ×2Bruno MarsDaft PunkStevie WonderThe Weeknd answer: correct | ✓ StromaeBruno MarsDaft PunkStevie WonderThe WeekndBurna Boy answer: correct | ✓ Stromae ×2Bruno MarsDaft PunkStevie WonderThe Weeknd answer: correct | ✓ Stromae ×2Bruno MarsDaft PunkStevie WonderThe Weeknd answer: correct | ✓ Stromae ×2Bruno MarsDaft PunkStevie WonderThe Weeknd answer: correct | all 51 artists, no search answer: correct | all 51 artists, no search answer: correct |
| When did Adele release 21? | ✓ Adele ×3Ed SheeranBillie EilishOlivia Rodrigo answer: correct | ✓ Adele ×3Ed SheeranBillie EilishOlivia Rodrigo answer: correct | ✓ Adele ×2Ed SheeranBillie EilishOlivia RodrigoSia answer: correct | ✓ AdeleEd SheeranBillie EilishOlivia RodrigoSiaAriana Grande answer: correct | ✓ Adele ×3Ed SheeranBillie EilishOlivia Rodrigo answer: correct | ✓ Adele ×3Ed SheeranBillie EilishOlivia Rodrigo answer: correct | ✓ Adele ×3Ed SheeranBillie EilishOlivia Rodrigo answer: correct | all 51 artists, no search answer: correct | all 51 artists, no search answer: correct |
| Where are Daft Punk from, and since when have they been active? | ✓ Daft Punk ×3StromaePrinceKendrick Lamar answer: correct | ✓ Daft Punk ×3StromaePrinceKendrick Lamar answer: correct | ✓ Daft Punk ×2StromaePrinceKendrick LamarBTS answer: correct | ✓ Daft PunkStromaePrinceKendrick LamarBTSBeyoncé answer: correct | ✓ Daft Punk ×3StromaePrinceKendrick Lamar answer: correct | ✓ Daft Punk ×3StromaePrinceKendrick Lamar answer: correct | ✓ Daft Punk ×3StromaePrinceKendrick Lamar answer: correct | all 51 artists, no search answer: correct | all 51 artists, no search answer: correct |
| How many Grammy awards has Beyoncé won? | ✓ Beyoncé ×4Cardi BNicki Minaj | ✓ Beyoncé ×4Cardi BNicki Minaj | ✓ Beyoncé ×2Cardi B ×2Nicki MinajRihanna | ✓ BeyoncéCardi BNicki MinajRihannaAriana GrandeWizkid | ✓ Beyoncé ×4Cardi BNicki Minaj | ✓ Beyoncé ×4Cardi BNicki Minaj | ✓ Beyoncé ×4Cardi BNicki Minaj | all 51 artists, no search | all 51 artists, no search |
Reading this
- Compare against the noise band first. Two runs with identical settings already differ, because answers and the judge’s verdicts vary.
- Retrieval scores don’t use an LLM, so they only change when the retrieved chunks change. Generation scores can change on any run.
- Each run’s full details (every claim and verdict) are on its own page: top-k-a, top-k-b, two-per-artist, one-per-artist, rerank-a, rerank-b, rerank-c, long-context-a, long-context-b.