standard error for a confidence interval in a held out test set? Assume I am doing a regression problem with a held out test set that contains 20% of the data. On the training data, I perform cross-validation to finalize my model. I can then make predictions on the testing data.
If I want to build a confidence interval to show how accurate and variant my model is, do I use the standard errors that is yielded by the test data, or do I need to record the standard errors when I am doing the cross-validation on my training data and use that to build my final test data confidence interval?
I also thought that building the confidence interval by using the SE of the test data is the way to go, but one of my friends (who is very smart) says that isn't correct. He advises that I record the standard errors when I am doing cross-validation and use that as the SE when I am building a CI on the test data. I don't understand why though.
 A: You can construct confidence intervals both on model parameters and on figures of merit for predictions (as well as prediction intervals for new predictions). 
Confidence intervals for model parameters are what you need e.g. if the model parameters correspond to some physical value (think of time constant of a chemical reaction, molar absorption coefficient to take examples from my field) that you want to measure. The ultimate outcome that is sought is model coefficient ± error on model coefficient. 
While for classical model parameter confidence intervals you basically have one source of random uncertainty only (the training data), prediction intervals cover two sources of uncertainty: the random uncertainty of the model (from the training data) and random uncertainty of the new cases (test data).  Confidence intervals for figures of merit such as classification accuracy or $RMSE_p$ again usually cover only one source of uncertainty: the test data as they assume performance of a given model is sought (i.e. they as what performance does this model have rather than what performance can I get with another such training dataset). 
When people talk about performance of a model, typically the model coefficients in themselves are not of interest.

Cross valdiation: the observed performance (figure of merit) is subject to (at least) two sources of random uncertainty:


*

*model instability, i.e. uncertainty of the model parameters/variance due to exchanging a few cases in the training set, and

*variance due to the finite number of test cases.


I suspect that you and your colleague each focuses on one of these two points - and you are both right in that each of "your" points should be covered. 
Model (in)stability can be measured/expressed in different ways: either via variance between model parameters or as variance of predictions for the same case but by different (surrogate) models. So with the 20% test set, you won't be able to measure (or separate) it, as only a single final model is tested.
We do (iterated/repeated cross validation to get an estimate of (in)stability):
Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 390, 1261-1271 (2008). DOI: 10.1007/s00216-007-1818-6
Note that I typically report range of observed values rather than constructing confidence intervals as I often don't know the (effective) sample size (and nowadays I wouldn't even try any more to arrive at it in the way we tried in the paper). 

Model comparison is yet more complicated. 


*

*Often, not models but algorithms are to be compared. I.e. the results are to be generalized to new data sets of somehow similar characteristics. Then you have an additional source variance which you cannot estimate from within one data set which you have at hand.
See Bengio, Y. and Grandvalet, Y.: No Unbiased Estimator of the Variance of K-Fold Cross-Validation Journal of Machine Learning Research, 2004, 5, 1089-1105 

*It may or may not be possible to do a paired test here. 
