Confidence interval vs. cross validation to estimate performance of a machine learning model

Question

When training and testing a machine learning model, if I split the dataset just once, I may end up with the "good" portion so I can have a good performance. I may also end up with the "bad" portion so I can have a poor performance.

To deal with such uncertainty, I usually use 5-fold or 10-fold cross validation to average the performance - usually AUC ROC.

But a colleague of mine, who's from a statistics background, told me that cross validation was not needed. Instead, split the data once, train and test the model, then simply use the confidence interval to estimate the performance. For example, I split my data just once, run the model, my AUC ROC is 0.80 and my 95% confidence interval radius is 0.05. Then the range of AUC ROC is .80+-0.05, which ends up with 0.75 to 0.85. Now I know the range of my model's performance without doing cross validation.

Is it true that the confidence interval can replace cross validation?

Answer 1

Yes, but:

• If the confidence interval you get from the single split test is sufficiently narrow, you're done and don't need cross validation.

• One reason to do cross validation is that it will test all available cases in turn. This means that the confidence interval of the cross validation results will be narrower than that of the single split.

• The confidence intervals above refer to uncertainty due to finite test sample size only. In addition, (repeated) cross validation allows you to estimate uncertainty due to model instability, which cannot be measured by a single split.
  So if that is a concern, go for a repeated cross validation or out-of-bootstrap scheme.