When fitting a linear regression model, is it always recommended to plot the residuals?

Question

If my linear regression model provides good results on the test set, and my main goal is to predict correctly, is there still a reason to plot the residuals and check the residual assumptions? (for example, to check if they are independent)

Let's assume the regression model has good results for a relevant metric for my goal. When I plot the residuals I see that the residuals are not independent and they do not have a constant variance. When I plot them using QQ-PLOT I see that they are not normally distributed. O.K, many important assumptions are violated. Should I care about it if the model's predictions are accurate?

Answer 1

Another good reason to plot residuals is to check the linearity assumptions. If the residuals are similar whatever the predicted value, then your model seems good.

If the residuals are small for small predicted values, and large for large predicted values, then the assumption of linearity does not seem good. In that case, I would try predicting the log of the Y value instead.

Another thing to look at is whether the residuals are normally distributed - if not, then again there might be ways to make a better model.

Answer 2

Recommendations are for free, so people are always happy to recommend doing more work when they are not the ones who have to do it. That being said, plotting data, including your predictions, is a good practice that will help you notice problems with the data that you have no way to know based on numbers alone (see the famous Anscombe's quartet, for example). Typing plot(resid(model)) or plot(predicted, observed) or plot(observed-predicted, observed) is really not that much work.

Of course, if you have 100 000 models, one for each gene, or each voxel in a brain, then nobody is checking residuals. (but maybe they should)

Edit, answer to comment: Many of the assumptions are important for inference but not for predictions, so if assumptions are violated but predictions are good, then it's not really a big deal in a sense that it won't make predictions invalid.

However, if residuals are not independent, that means that there is an effect in the data that you can model better, hence improving your predictions. If they are not normal, then you might consider using a more robust model., which would again improve your predictions (according to some metrics). Finally, if they are not homoskedastic, then you can sometimes improve model fit by modeling variance, or you can realize that you are using the wrong model e.g., Gaussian, instead of Poisson/logistic, and you can fix it and again improve your predictions.

I am not saying it will always make a large difference or that you must satisfy your assumptions, but sometimes it helps.

Answer 3

"The model's predictions are accurate" - If by "predictions", you mean "predictions from real-world data", then this is something that you almost certainly do not know for sure. Why? Almost always by "test set", we mean a set of data that have been bootstrapped out of our training data. So, if our training data aren't perfectly representative of data that your model will encounter in the wild (which is what you're going to be applying your model to, if prediction is your goal), then neither is your test data. Almost surely your training data are not perfectly representative of data you will see in the wild. Is it good enough? You have to check for these sorts of things.

Your overall perspective on the model evaluation process is backward. We need to be scientists about the matter. Being a scientist about the matter means that you're not looking for evidence that your model is right... it means meticulously looking for evidence that your model could be wrong. The depth that you have delved for ways that you could be wrong is the true test of any hypothesis. Your hypothesis is "this linear regression model will do a good job predicting future Y from future X." You want to look in every corner for things that might suggest that this hypothesis is wrong. Obviously, you cannot check every corner, because there are infinitely many corners, many of which are so dark that you cannot see into them. Still, you should check the corners that you can check. One of those corners you check by holding out a test set. This is usually to check the "overfitting" corner - Is my model just regurgitating the training data, or is it actually doing some kind of generalization? However, there are many other corners. One of these that is extremely common to crop up in real life is the "data isn't representative" corner. This means that you trained your model on the data that you had, but the data that you had isn't a very good picture of all of the data that the model sees in real life. Checking out the residuals is one way to convince yourself that this might be a way for your model to fail (and remember, you are meticulously looking for ways that your model can fail).

So, when you see this sort of thing:

You should be thinking to yourself "okay, well what happens when my model is confronted with $x=2.5$?" Why? Because the data "curves", and the model doesn't (i.e., my residuals aren't independent of X). If your model might well be confronted with $x=2.5$ in the real world, then your training data isn't very representative of the real world, is it? You can see this because we don't have anything like $x=2.5$ in our training data. If you don't have anything like $x=2.5$ in your training data, then you don't have anything like it in your test data either. Bootstrapping test data out of training data doesn't fix problems with your training data!

Anyway, the point here isn't that you should or shouldn't look at the residuals. The point here is that you are taking entirely the wrong perspective on how to evaluate your model. You shouldn't be focused on finding evidence that your model is good. You should be focused on finding evidence that your model is BAD! If you look really really hard for evidence that your model is bad, and don't find any, then and only then should you begin having confidence that your model is "good". Checking out the regression coefficient is one way to do this. Checking the residuals is another one of many ways to check for "common" pitfalls where models turn out to be "bad".

Answer 4

Accuracy is Everything

As others have noted, you want your model to be accurate before you start screaming from the rooftops that it is useful. If your regression says that candy consumption predicts weight loss, you better hope that is actually true by checking whether or not that is valid. To give a visual example, we can plot a typical regression in R with the iris dataset, predicting petal dimensions for each flower. After we can plot the residuals:

#### Fit Model ####
fit <- lm(Petal.Length ~ Petal.Width,
          iris)

#### Plot Residuals ####
plot(density(resid(fit)))

Which look fairly normal:

Bad Fit and Consequences

If we fit a really bad model, it can have grave consequences even if the predictors are significant. Consider if we decided to just add zero to the model this time and plot the residuals:

#### Misspecify Model ####
fit.bad <- lm(Petal.Length ~ 0,
              iris)

#### Plot Again ####
plot(density(resid(fit.bad)))

Then residuals are now strangely bimodal:

There is probably a clear reason why. Since the model actually isn't modeling anything (the syntax reads "average petal length is disaggregated by zero"), we are actually just getting the density of the raw values of petal length. As proof, we can run the following code:

#### Plot Density of Variable ####
plot(density(iris$Petal.Length))

And sure enough, the plot is exactly the same:

Takeaway Message

To ensure your model is behaving, always check performance. The results may come off as exciting until you find out your model isn't a model at all (or at least a very poor one).