So suppose, I have 5000 points of data. I hold out 1000 for validation testing, and conduct simple linear regression on the first 4000 points.

How can I determine if the linear model: y = a + bx + e, reasonably explains the relationship between x and y using this technique?

I've heard that prediction and confidence intervals are very useful in these situations, but am not too sure how to apply them.


A quite simple approach is to use the training partition (your 4000 samples) for model tuning and model evaluation (to decide on your single best model), then to use the test partition (your 1000 held-out samples) once for obtaining an estimate on how well your single best model performs on new, unseen data.

Besides others, this can be done using repeated cross validation on the training set. With e.g. using 10 partitions and 20 repeats, you will train each model type and parameter set 200 times and obtain 200 performance estimates for each. The spread of those estimates gives you a hint on how likely it is to obtain a model that performs badly using this model type and parametrization. This information can be used to decide upon the best suited model type and parametrization then. After doing so, the chosen model is trained using all data in the training partition (usually done automatically by your ML software), then used to predict the target variable for the held-out test partition for a final performance estimate.

Here's a simple example on how this could look like (using R caret):

> model <- train(x = iris[,1:3], y = iris[,4], method = 'lm', 
  trControl = trainControl(method = 'repeatedcv', number = 10, repeats = 20))
> print(model$resample)
         RMSE  Rsquared     Resample
1   0.1485536 0.9667557 Fold01.Rep01
2   0.2404998 0.9097242 Fold02.Rep01
3   0.2173052 0.9494554 Fold03.Rep01
4   0.1733321 0.9486735 Fold04.Rep01
5   0.1871449 0.9429690 Fold05.Rep01
6   0.2347253 0.9206204 Fold06.Rep01
7   0.2419445 0.8938819 Fold07.Rep01
8   0.1856019 0.9603457 Fold08.Rep01
9   0.1195387 0.9785385 Fold09.Rep01
> boxplot(model$resample$RMSE, main = 'RMSE')

enter image description here

PS: this result is not a classic confidence interval (which can be calculated as well), but still employed quite frequently as it gives you a good hint about what is going on with your partitions and resamples.

| cite | improve this answer | |

"How can I determine whether a linear regression is good" is quite a long question. You also mention training and test sets which is in many ways a separate question.

When checking a linear regression you typically do what we call "regression diagnostics". You can look up a lot more online about this and probably find some straight-forward videos. And example link that is a bit dense is: http://people.duke.edu/~rnau/testing.htm. Different people have a different art to determining whether the model meets the assumptions. R-squared is often looked at, normality plots are often looked at (and non-normality may drive larger confidence intervals), etc. Your goal here is to determine whether the requirements of linear regression are met and fix them if they aren't.

Confidence intervals and prediction intervals will give you an idea of how far off your model will be from reality. How close you need your confidence bands depends on your requirements. If you're building the space shuttle you probably need to have a tighter tolerance on component manufacturing than you do if you're building a child's toy firetruck.

If your linear regression matches all the assumptions required for linear regression, and your diagnostic shows a good regression, then you can get ready to use your test set. Your test set is used to validate whether your model is over-fit to the training set. If your training set doesn't represent the total population then your test set will have significant differences in performance than your training set did. This shows you that you have a problem with the model or that you don't have enough data about the total population being input into your model (or some other problem).

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.