4
$\begingroup$

Assume I am doing a regression problem with a held out test set that contains 20% of the data. On the training data, I perform cross-validation to finalize my model. I can then make predictions on the testing data.

If I want to build a confidence interval to show how accurate and variant my model is, do I use the standard errors that is yielded by the test data, or do I need to record the standard errors when I am doing the cross-validation on my training data and use that to build my final test data confidence interval?

I also thought that building the confidence interval by using the SE of the test data is the way to go, but one of my friends (who is very smart) says that isn't correct. He advises that I record the standard errors when I am doing cross-validation and use that as the SE when I am building a CI on the test data. I don't understand why though.

$\endgroup$
3
  • $\begingroup$ Are you actually interested in confidence intervals (i.e. the model itself or the model parameters themselves are what you are interested in) or do you look for prediction intervals (i.e. uncertainty on predictions [for new cases])? $\endgroup$ Jun 25, 2017 at 14:00
  • $\begingroup$ I am interested in confidence intervals (the model itself) because I want a way to compare the performance of different models other than just using accuracy-based metrics since I am also interested in the variance of these models. The method that I described was a prediction interval and not a confidence interval? $\endgroup$
    – Kevin Sun
    Jun 27, 2017 at 5:51
  • $\begingroup$ no, you were talking about confidence interval. It's just often when people ask me about confidence intervals they actually need prediction or tolerance intervals (particularly if it is about prediction) but don't know that - so I tend to ask back whether c.i. is really what is needed. $\endgroup$ Jun 30, 2017 at 7:17

1 Answer 1

2
$\begingroup$

You can construct confidence intervals both on model parameters and on figures of merit for predictions (as well as prediction intervals for new predictions).

Confidence intervals for model parameters are what you need e.g. if the model parameters correspond to some physical value (think of time constant of a chemical reaction, molar absorption coefficient to take examples from my field) that you want to measure. The ultimate outcome that is sought is model coefficient ± error on model coefficient.

While for classical model parameter confidence intervals you basically have one source of random uncertainty only (the training data), prediction intervals cover two sources of uncertainty: the random uncertainty of the model (from the training data) and random uncertainty of the new cases (test data). Confidence intervals for figures of merit such as classification accuracy or $RMSE_p$ again usually cover only one source of uncertainty: the test data as they assume performance of a given model is sought (i.e. they as what performance does this model have rather than what performance can I get with another such training dataset).

When people talk about performance of a model, typically the model coefficients in themselves are not of interest.


Cross valdiation: the observed performance (figure of merit) is subject to (at least) two sources of random uncertainty:

  • model instability, i.e. uncertainty of the model parameters/variance due to exchanging a few cases in the training set, and
  • variance due to the finite number of test cases.

I suspect that you and your colleague each focuses on one of these two points - and you are both right in that each of "your" points should be covered.

Model (in)stability can be measured/expressed in different ways: either via variance between model parameters or as variance of predictions for the same case but by different (surrogate) models. So with the 20% test set, you won't be able to measure (or separate) it, as only a single final model is tested. We do (iterated/repeated cross validation to get an estimate of (in)stability): Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 390, 1261-1271 (2008). DOI: 10.1007/s00216-007-1818-6
Note that I typically report range of observed values rather than constructing confidence intervals as I often don't know the (effective) sample size (and nowadays I wouldn't even try any more to arrive at it in the way we tried in the paper).


Model comparison is yet more complicated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.