# Is AUROC sample size-unbiased

Some metrics are sample size-biased, i.e., they will give biased estimates in small sample sizes. Is there a formal proof that AUROC is either sample size-biased or unbiased? I.e., will the expectation over the AUROC computed over test sets of size 100 be the same as the expectation over the AUROC computed over test sets of size 10,000 drawn from the same distribution?

We have thus an unbiased estimator of the empirical AUC-ROC; the recenti(ish) paper Confidence Intervals for the Area Under the Receiver Operating Characteristic Curve in the Presence of Ignorable Missing Data (2019) by Cho et al. looks at this in more details in a modern and approachable manner. To that effect, there has been a constant stream of papers on this since probably... 1970's as the equivalence relation between the AUC-ROC and Wilcoxon’s rank-sum test statistic has been a fertile ground for such work. sklearn uses trapezoid integration of the ROC curve which is equivalent to its AUC because it too uses all the possible threshold values. Note that, unlike other integration tasks, we know that we have a "real" discontinuous step function, so no smoothing is needed; Hand & Till (2001)'s paper A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems on Section 2 comment this further and caution against smoothing. CV.SE has an excellent relevant thread on "How to calculate Area Under the Curve (AUC), or the c-statistic, by hand" that is also relevant (and shows how smoothing also leads to a slightly biased AUC-ROC estimate in one case). Oh and if we want a "within CV.SE" derivation as to why this AUC is the same as this probability of concordance, this is here.
• Thank you for your comment, good clarifications to add. I should have said that the calculation is exact (i.e. we know all the pairs) so there is no "under-" or "over-" estimation in regard to the empirical calculations given a fixed (smaller or larger) sample size in the context of classification. This is what is described as the $c$-index / concordance-index / Wilcoxon’s rank-sum test statistic i.e. the counts of the number of concordant pairs. (Cont.) Jan 25 at 12:33
• sklearn uses trapezoid integration of the ROC curve which is equivalent to its AUC. CV.SE has a relevant thread here. (I will amend the main answer accordingly.) Jan 25 at 12:33