Re-ranking: scale, cap, temperature, or the batch?

What it showed

• The batch is the problem. Scored on its own, Stevie Wonder’s profile chunk (tags: soul, motown) gets the top score every time. Scored as candidate #18 in a batch of 30, it gets 0 or 1. Chunks near the top of the search results score the same both ways; chunks further down are underrated in the batch (table below).
• Temperature 1 made the scores noisy. The same chunk got very different scores in different runs, and the best run at temperature 1 won by luck. Temperature 0 isn’t a full fix: on 0–100 the scores were identical every time, but on 0–3 the soul candidates were less repeatable than at temperature 1 (table below). And stable isn’t the same as good: at temperature 0, plain 0–3 lost Shakira for the Latin question.
• At temperature 0, the scale doesn’t matter and the cap does all the work. Both capped variants match two-per-artist exactly on recall and correctness. The re-ranker scored Stevie Wonder 0 or 1 even then; the cap only let his chunks in because it pushed out Marvin Gaye’s and Aretha Franklin’s extra chunks.
• A cap of 2 can’t fit a 5-artist answer into 6 slots. Kishore Kumar and Diljit Dosanjh were missing for Bollywood in every run. Diljit Dosanjh scores low even on his own (his tags are punjabi and bhangra), which is a fair judgement.
• Yes/no probabilities didn’t help. Tried first as a continuous score: gpt-4o-mini answered “true” with near-certainty for every relevant chunk, so they tied too.
• Next: score each candidate in its own call (pointwise), the way cross-encoders and probability-based re-rankers work. That costs 30 small calls per question instead of one large one.