Calculate probability of being in the top-N based on pairwise probabilities I have a set of, say, 20 documents and I have calculated all the pairwise probabilities that doc_x is more important than doc_y (e.g. P(doc_1 > doc_2) = 0.7). However, the probabilities may be inconsistent i.e. there may exist three documents where P(doc_1 > doc_2) > P(doc_2 > doc_3) > P(doc_3 > doc_1).
Is there a way to estimate the probability of each document belonging in the top-N most important documents? In other words, how can I say that P(doc_1 belongs in the top-5 documents) = ...%?
EDIT:
The probabilities mentioned above are the output of a machine learning model that is trained to return the 'probability' of doc_1 > doc_2. So, they are the output of a binary classification model. Therefore, they are estimates of a probability but not a 'real' probability of some underlying distribution.
Let's suppose we always have 20 documents (no more, no less) and all their pairwise probability estimates. Is there a way to estimate the probability of doc_x being in the top-N (N = [1, 2, ... 20]) most important ones? For the sake of simplicity, we can ignore that they may be inconsistent.
It's obvious that P(doc_x belongs in the top-20 docs) = 1.
 A: This situation is called the Condorcet Paradox, which is a violation of the assumption of transitivity in social science research. An often-used solution is to either treat the majority decision of a group as a single decision or to dichotomize choices. Treating the majority as the final decision is common in democracies and dichotomous choices is the de facto rule in voting in some democracies. There is a great walk-through on Wikipedia. In general, it is a function of the number of choices.
EDIT:
I'm going to change around your symbols a bit, because $N$ usually means the size of a population (in my world). I'm also introducing your "top" number as the variable $k$. In your post you want to consider the case $N=20$ and $1\leq k\leq20$.
As you said, the probability of a document being in the top for $N=20$ is 1.0. In other words, there are 20 ways of being in the top 20. Flipping this around, how many ways can a document be in the top 1? There are 20 ways that a document can be the top document. Now we have some bounds.
Stretching further, how many ways are there to be in the top-2? Let $d$ be a matrix with $k$ columns and $x$ rows, where x is the number of ways a document can appear in the top $k$.
For $k=2$, $d=[(doc1,doc2),(doc1,doc3),...,(doc1,doc20),(doc2,doc1),(doc3,doc1),(doc20,doc1)]$
This matrix has 190 rows, or $x=190$, which means there are 190 unique combinations. I believe your problem really is based on combinations, where the number of ways to be in the top $k$ is given by $$\frac{n!}{k!(n-k)!}$$
Working off our "bounds" before, we have a distribution of outcomes
The skew and kurtosis of this distribution are within reasonable limits for a normal distribution, so I say you can treat this distribution of combinations as a normal distribution.
Thus, using a z-table or using a function like pnorm(qnorm(1/n-k+1)) in R, you get the probability of such combinations.

Here is the code I used:
require(psych)
outcomes=c(NULL)
for(i in 1:20){outcomes=c(outcomes,choose(20,i))}
barplot(outcomes,main="C(20,k)",ylab="Combinations",xlab="k")
axis(1,at=c(1:20),tick=F)
describe(outcomes)
my=function(k){return(pnorm(qnorm(1/(20-k+1))))}
pr=c(NULL)
for(i in 1:20){pr=c(pr,my(i))}
barplot(pr)

EDIT: I'm not sure if this will serve your purposes, but this is a simulated Bayesian analysis. I'm still very much a learner of Bayesian analysis, so take it with a grain of salt.
Pretend you have 10 observations of a good driver and 10 observations over bad driver and you want to know the posterior probability that they will finish in the top k, $k=10$.
obs.good=c(rep(1,9),2)
obs.bad=c(rep(20,9),19)
topk=10

You need to find the observed effects.
#calcualte effects
sum.good=summary(lm(obs.good~c(1:length(obs.good))))
sum.bad=summary(lm(obs.bad~c(1:length(obs.bad))))
#save effects
b.good=sum.good$coefficients[2,1]
b.bad=sum.bad$coefficients[2,1]
se.good=sum.good$coefficients[2,2]
se.bad=sum.bad$coefficients[2,2]

Then, calcualte the posterior probability, where the priors are determined by the distribution of combinations.
#calcualte posterior probability
require(BayesCombo)
mypph.good=pph(beta=b.good,se.beta=se.good,H.priors=c(pr[topk-1],pr[topk],1-pr[topk-1]))
mypph.bad=pph(beta=b.bad,se.beta=se.bad,H.priors=c(pr[topk-1],pr[topk],1-pr[topk-1]))
mypph.good$pphs
mypph.bad$pphs
par(mfrow=c(1,2))
plot(mypph.good)
plot(mypph.bad)

See that the probability that the good driver will finish below 10th place is 96.1%, that they will finish in exactly 10th place is 3.3%, and that they will finish above 10th place is 0.6%. For the bad driver, the probability they will finish below 10th place is 37%, exactly 10th place 17.3%, and 45.7% above 10th place.
> mypph.good$pphs
         H<          H0          H> 
0.006427993 0.033005494 0.960566514 
> mypph.bad$pphs
       H<        H0        H> 
0.4571048 0.1727696 0.3701256

You can copy the code above and change your topk and obs variables. Like I said though, I am still learning Bayesian analysis, so maybe another user can add to what I've done here.
