In cross-validation, which is the AUC population parameter I really want to estimate?

Question

In machine learning, AUC is usually used as a performance metric of an algorithm. As one is interested in the performance of the algorithm when applied to new cases beyond those used during the training process, either a independent test-set or a cross-validation procedure is used.

In both cases, the AUC coming from them aims to be an estimate of the general performance in the population of the algorithm. This implies to make an inferece. Thus, the calculated test/cross-validated AUC is used as estimate of the AUC population parameter and several different procedures to calculate the AUC confidence interval exists (e.g. LeDell et al., 2015)

My question may sound quite theoretical but it is not clear to me which population parameter these AUC estimate and CI refers to. I mean which among the following or more possibilities (assuming all cases are sampled by the same population):

• the average test AUC when the current trained model is used to make prediction in infinite samples of new cases as large as the training one.
• the test AUC when the trained model is used to make prediction in all new cases of the population
• the average cross-validated AUC of infinite models trained by infinite samples of size n

Answer 1

It is the first case, i.e. the expected value of the AUC and CI with the same test set size.

We can rule out the third case (infinite models) immediately because the cross-validation is done using only the trained model. Hence, it is not valid for any other model.

While the AUC for the first and second cases would be the same (perhaps given weak assumptions on the model), the CI would be smaller if prediction were performed on the entire population instead of a subset of it (second case).