Is it possible to get top N predictions, with scores?

[jkhartshorne]

I'm comparing t5 predictions against some human data. I know that t5's prediction tends to be the same as the top human prediction. What I'd like to compare against is runners-up. That is, is its second choice (or third choice, etc.) the same as for humans, and is the relative probability similar (or at least correlated)? That is, when humans are nearly equally split between two response, is t5 as well? When humans strongly prefer the top answer to the second most likely, is that also the case for t5?

You get the idea.

I see in the Colabs how to get the top prediction and its score (here), but I've been string out in terms of finding a way of getting multiple scores. I think I can set temperature to 1 and then I'll get predictions in proportion to probability, but I'd potentially have to run it a bunch of times to be confident that the top N predictions are the same as the true top N.

(Also, when I tried this, many of the responses were broken, ungrammatical, nonsensical responses. So I think I'm getting stuff that is too low probability.)