In machine learning, AUC is usually used as a performance metric of an algorithm. As one is interested in the performance of the algorithm when applied to new cases beyond those used during the training process, either a independent test-set or a cross-validation procedure is used.
In both cases, the AUC coming from them aims to be an estimate of the general performance in the population of the algorithm. This implies to make an inferece. Thus, the calculated test/cross-validated AUC is used as estimate of the AUC population parameter and several different procedures to calculate the AUC confidence interval exists (e.g. LeDell et al., 2015)
My question may sound quite theoretical but it is not clear to me which population parameter these AUC estimate and CI refers to. I mean which among the following or more possibilities (assuming all cases are sampled by the same population):
- the average test AUC when the current trained model is used to make prediction in infinite samples of new cases as large as the training one.
- the test AUC when the trained model is used to make prediction in all new cases of the population
- the average cross-validated AUC of infinite models trained by infinite samples of size n