# Confidence and Prediction Intervals and Cross-Validation

So suppose, I have 5000 points of data. I hold out 1000 for validation testing, and conduct simple linear regression on the first 4000 points.

How can I determine if the linear model: y = a + bx + e, reasonably explains the relationship between x and y using this technique?

I've heard that prediction and confidence intervals are very useful in these situations, but am not too sure how to apply them.

A quite simple approach is to use the training partition (your 4000 samples) for model tuning and model evaluation (to decide on your single best model), then to use the test partition (your 1000 held-out samples) once for obtaining an estimate on how well your single best model performs on new, unseen data.

Besides others, this can be done using repeated cross validation on the training set. With e.g. using 10 partitions and 20 repeats, you will train each model type and parameter set 200 times and obtain 200 performance estimates for each. The spread of those estimates gives you a hint on how likely it is to obtain a model that performs badly using this model type and parametrization. This information can be used to decide upon the best suited model type and parametrization then. After doing so, the chosen model is trained using all data in the training partition (usually done automatically by your ML software), then used to predict the target variable for the held-out test partition for a final performance estimate.

Here's a simple example on how this could look like (using R caret):

> model <- train(x = iris[,1:3], y = iris[,4], method = 'lm',
trControl = trainControl(method = 'repeatedcv', number = 10, repeats = 20))
> print(model$resample) RMSE Rsquared Resample 1 0.1485536 0.9667557 Fold01.Rep01 2 0.2404998 0.9097242 Fold02.Rep01 3 0.2173052 0.9494554 Fold03.Rep01 4 0.1733321 0.9486735 Fold04.Rep01 5 0.1871449 0.9429690 Fold05.Rep01 6 0.2347253 0.9206204 Fold06.Rep01 7 0.2419445 0.8938819 Fold07.Rep01 8 0.1856019 0.9603457 Fold08.Rep01 9 0.1195387 0.9785385 Fold09.Rep01 ... > boxplot(model$resample\$RMSE, main = 'RMSE')


PS: this result is not a classic confidence interval (which can be calculated as well), but still employed quite frequently as it gives you a good hint about what is going on with your partitions and resamples.

"How can I determine whether a linear regression is good" is quite a long question. You also mention training and test sets which is in many ways a separate question.

When checking a linear regression you typically do what we call "regression diagnostics". You can look up a lot more online about this and probably find some straight-forward videos. And example link that is a bit dense is: http://people.duke.edu/~rnau/testing.htm. Different people have a different art to determining whether the model meets the assumptions. R-squared is often looked at, normality plots are often looked at (and non-normality may drive larger confidence intervals), etc. Your goal here is to determine whether the requirements of linear regression are met and fix them if they aren't.

Confidence intervals and prediction intervals will give you an idea of how far off your model will be from reality. How close you need your confidence bands depends on your requirements. If you're building the space shuttle you probably need to have a tighter tolerance on component manufacturing than you do if you're building a child's toy firetruck.