Why does the mean AURPC go down the more examples one uses?

Question

In the discussion of this question, the following new one arose:

Why is the mean AURPRC higher the fewer examples are used?

Here is a minimal (Python) code example showing the effect:

import numpy as np
from sklearn.metrics import average_precision_score


def auprc(scores_of_negatives, scores_of_positives):
    y_true = np.concatenate(
        (np.full(scores_of_negatives.size, 0),
         np.full(scores_of_positives.size, 1))
    )
    y_scores = np.concatenate((scores_of_negatives, scores_of_positives))
    return average_precision_score(y_true, y_scores)


a_neg, b_neg = 0.3, 1.3
a_pos, b_pos = 2.0, 0.9

for n in [1, 2, 4, 8, 16, 32, 64, 128, 256]:
    auprcs = []
    for _ in range(10000):
        auprcs.append(auprc(
            np.random.beta(a_neg, b_neg, n),
            np.random.beta(a_pos, b_pos, n)
        ))
    print(n, np.mean(auprcs))

Output:

1 0.95795
2 0.9387583333333334
4 0.9231154166666665
8 0.9127574443265069
16 0.9044083984271667
32 0.8996982152618184
64 0.8967667717732809
128 0.895994038020052
256 0.8949927372666804

Answer 1

I think the answer has to do with the probability of drawing a sample from x1=np.random.beta(a_neg, b_neg, n) being greater than a sample from x2=np.random.beta(a_pos, b_pos, n). The more samples you draw the more likely you will catch the case where x1>x2. If you only draw a single sample (N=1) the probability of that happening is ~0.08673 (simulate it ;)

we can then calculate the PR auc as:

0.08673*0.5+(1-0.08673)*1 ~ 0.956635

which matches your N=1 case. N=2 is a bit more tricky so i will leave that to the avid reader.

About 0.08673 of the time our PR AUC will be 0.5 (the case where the negative sample has higher score than the positive one) and then (1-0.08673) of the time the other case will happen.

Now if you increase the sample size your PR_AUC converges to the true PR AUC given samples ranked with those distributions. (if you run your code with n=1024 or 4096 or any other big value it seems to slowly converge to something like ~0.894).

I apologize for it being not the most mathy answer but I think it catches the right idea.