# In cross-validation, which is the AUC population parameter I really want to estimate?

In machine learning, AUC is usually used as a performance metric of an algorithm. As one is interested in the performance of the algorithm when applied to new cases beyond those used during the training process, either a independent test-set or a cross-validation procedure is used.

In both cases, the AUC coming from them aims to be an estimate of the general performance in the population of the algorithm. This implies to make an inferece. Thus, the calculated test/cross-validated AUC is used as estimate of the AUC population parameter and several different procedures to calculate the AUC confidence interval exists (e.g. LeDell et al., 2015)

My question may sound quite theoretical but it is not clear to me which population parameter these AUC estimate and CI refers to. I mean which among the following or more possibilities (assuming all cases are sampled by the same population):

• the average test AUC when the current trained model is used to make prediction in infinite samples of new cases as large as the training one.
• the test AUC when the trained model is used to make prediction in all new cases of the population
• the average cross-validated AUC of infinite models trained by infinite samples of size n
• "These CIs" is a bit vague. Can you provide some context? What is the procedure for calculating the AUC CI that you are referring to? (Different setups can pertain to different options.) – gung Feb 7 '17 at 15:38
• Thanks @gung, I edited the question so hopefully now it is more clear. – Massimiliano Grassi Feb 7 '17 at 17:23
• The first two papers estimate the CI of one AUC and have no link to cross-validation whatsoever. So I guess your question only refers to LeDell et al., 2015? – Calimo Feb 8 '17 at 8:45
• @Calimo I'm sorry if introducing the references may have led to some misunderstandings. My question wants to be theoretical and not referring to any procedure in particular. – Massimiliano Grassi Feb 8 '17 at 15:14