You can construct confidence intervals both on model parameters and on figures of merit for predictions (as well as prediction intervals for new predictions).
Confidence intervals for model parameters are what you need e.g. if the model parameters correspond to some physical value (think of time constant of a chemical reaction, molar absorption coefficient to take examples from my field) that you want to measure. The ultimate outcome that is sought is model coefficient ± error on model coefficient.
While for classical model parameter confidence intervals you basically have one source of random uncertainty only (the training data), prediction intervals cover two sources of uncertainty: the random uncertainty of the model (from the training data) and random uncertainty of the new cases (test data). Confidence intervals for figures of merit such as classification accuracy or $RMSE_p$ again usually cover only one source of uncertainty: the test data as they assume performance of a given model is sought (i.e. they as what performance does this model have rather than what performance can I get with another such training dataset).
When people talk about performance of a model, typically the model coefficients in themselves are not of interest.
Cross valdiation: the observed performance (figure of merit) is subject to (at least) two sources of random uncertainty:
- model instability, i.e. uncertainty of the model parameters/variance due to exchanging a few cases in the training set, and
- variance due to the finite number of test cases.
I suspect that you and your colleague each focuses on one of these two points - and you are both right in that each of "your" points should be covered.
Model (in)stability can be measured/expressed in different ways: either via variance between model parameters or as variance of predictions for the same case but by different (surrogate) models. So with the 20% test set, you won't be able to measure (or separate) it, as only a single final model is tested.
We do (iterated/repeated cross validation to get an estimate of (in)stability):
Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 390, 1261-1271 (2008). DOI: 10.1007/s00216-007-1818-6
Note that I typically report range of observed values rather than constructing confidence intervals as I often don't know the (effective) sample size (and nowadays I wouldn't even try any more to arrive at it in the way we tried in the paper).
Model comparison is yet more complicated.
