5
$\begingroup$

In machine learning, AUC is usually used as a performance metric of an algorithm. As one is interested in the performance of the algorithm when applied to new cases beyond those used during the training process, either a independent test-set or a cross-validation procedure is used.

In both cases, the AUC coming from them aims to be an estimate of the general performance in the population of the algorithm. This implies to make an inferece. Thus, the calculated test/cross-validated AUC is used as estimate of the AUC population parameter and several different procedures to calculate the AUC confidence interval exists (e.g. LeDell et al., 2015)

My question may sound quite theoretical but it is not clear to me which population parameter these AUC estimate and CI refers to. I mean which among the following or more possibilities (assuming all cases are sampled by the same population):

  • the average test AUC when the current trained model is used to make prediction in infinite samples of new cases as large as the training one.
  • the test AUC when the trained model is used to make prediction in all new cases of the population
  • the average cross-validated AUC of infinite models trained by infinite samples of size n
$\endgroup$
  • $\begingroup$ "These CIs" is a bit vague. Can you provide some context? What is the procedure for calculating the AUC CI that you are referring to? (Different setups can pertain to different options.) $\endgroup$ – gung Feb 7 '17 at 15:38
  • $\begingroup$ Thanks @gung, I edited the question so hopefully now it is more clear. $\endgroup$ – Massimiliano Grassi Feb 7 '17 at 17:23
  • $\begingroup$ The first two papers estimate the CI of one AUC and have no link to cross-validation whatsoever. So I guess your question only refers to LeDell et al., 2015? $\endgroup$ – Calimo Feb 8 '17 at 8:45
  • $\begingroup$ @Calimo I'm sorry if introducing the references may have led to some misunderstandings. My question wants to be theoretical and not referring to any procedure in particular. $\endgroup$ – Massimiliano Grassi Feb 8 '17 at 15:14
2
+50
$\begingroup$

It is the first case, i.e. the expected value of the AUC and CI with the same test set size.

We can rule out the third case (infinite models) immediately because the cross-validation is done using only the trained model. Hence, it is not valid for any other model.

While the AUC for the first and second cases would be the same (perhaps given weak assumptions on the model), the CI would be smaller if prediction were performed on the entire population instead of a subset of it (second case).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.