When training and testing a machine learning model, if I split the dataset just once, I may end up with the "good" portion so I can have a good performance. I may also end up with the "bad" portion so I can have a poor performance.
To deal with such uncertainty, I usually use 5-fold or 10-fold cross validation to average the performance - usually AUC ROC.
But a colleague of mine, who's from a statistics background, told me that cross validation was not needed. Instead, split the data once, train and test the model, then simply use the confidence interval to estimate the performance. For example, I split my data just once, run the model, my AUC ROC is 0.80 and my 95% confidence interval radius is 0.05. Then the range of AUC ROC is .80+-0.05, which ends up with 0.75 to 0.85. Now I know the range of my model's performance without doing cross validation.
Is it true that the confidence interval can replace cross validation?