8
$\begingroup$

If my linear regression model provides good results on the test set, and my main goal is to predict correctly, is there still a reason to plot the residuals and check the residual assumptions? (for example, to check if they are independent)

Let's assume the regression model has good results for a relevant metric for my goal. When I plot the residuals I see that the residuals are not independent and they do not have a constant variance. When I plot them using QQ-PLOT I see that they are not normally distributed. O.K, many important assumptions are violated. Should I care about it if the model's predictions are accurate?

$\endgroup$
3
  • 7
    $\begingroup$ is there a reason not to plot the residuals? $\endgroup$
    – rep_ho
    Oct 18, 2022 at 11:33
  • 3
    $\begingroup$ Yes, because looking at the residuals will help us improve predictive accuracy even more. Any pattern in the residuals can be exploited by a model which incorporates these patterns. $\endgroup$ Oct 18, 2022 at 21:13
  • $\begingroup$ @rep_ho i guess when you have to draw the plots by hand...with a stick...in a desert? $\endgroup$ Oct 21, 2022 at 10:38

4 Answers 4

9
$\begingroup$

Another good reason to plot residuals is to check the linearity assumptions. If the residuals are similar whatever the predicted value, then your model seems good.

If the residuals are small for small predicted values, and large for large predicted values, then the assumption of linearity does not seem good. In that case, I would try predicting the log of the Y value instead.

Another thing to look at is whether the residuals are normally distributed - if not, then again there might be ways to make a better model.

$\endgroup$
1
  • 12
    $\begingroup$ What you describe sounds more like heteroskedasticity than nonlinearity. $\endgroup$ Oct 19, 2022 at 8:04
19
$\begingroup$

Recommendations are for free, so people are always happy to recommend doing more work when they are not the ones who have to do it. That being said, plotting data, including your predictions, is a good practice that will help you notice problems with the data that you have no way to know based on numbers alone (see the famous Anscombe's quartet, for example). Typing plot(resid(model)) or plot(predicted, observed) or plot(observed-predicted, observed) is really not that much work.

Of course, if you have 100 000 models, one for each gene, or each voxel in a brain, then nobody is checking residuals. (but maybe they should)

Edit, answer to comment: Many of the assumptions are important for inference but not for predictions, so if assumptions are violated but predictions are good, then it's not really a big deal in a sense that it won't make predictions invalid.

However, if residuals are not independent, that means that there is an effect in the data that you can model better, hence improving your predictions. If they are not normal, then you might consider using a more robust model., which would again improve your predictions (according to some metrics). Finally, if they are not homoskedastic, then you can sometimes improve model fit by modeling variance, or you can realize that you are using the wrong model e.g., Gaussian, instead of Poisson/logistic, and you can fix it and again improve your predictions.

I am not saying it will always make a large difference or that you must satisfy your assumptions, but sometimes it helps.

$\endgroup$
3
  • $\begingroup$ also added in the main post: Let's assume that the regression model has good results on a relevant metric for my goal . When I plot the residuals, I see that the residuals are not independent, have not a constant variance, and when I plot them using QQ-PLOT, I see that they are not normal distributed. O.K, many important assumptions are violated. Should I care about it if the model's predictions are accurate? $\endgroup$
    – Amit S
    Oct 18, 2022 at 12:09
  • $\begingroup$ @AmitS I expanded my answer $\endgroup$
    – rep_ho
    Oct 18, 2022 at 13:22
  • 5
    $\begingroup$ @AmitS If the model makes you rich, and you are content to be rich instead of putting in some additional work to be really rich, you have a valid argument for using your existing model, despite its potential to be improved. (You might consider posting your question as its own question. I rather like my response as an answer, and I would be curious to read what others think.) $\endgroup$
    – Dave
    Oct 18, 2022 at 13:34
5
$\begingroup$

"The model's predictions are accurate" - If by "predictions", you mean "predictions from real-world data", then this is something that you almost certainly do not know for sure. Why? Almost always by "test set", we mean a set of data that have been bootstrapped out of our training data. So, if our training data aren't perfectly representative of data that your model will encounter in the wild (which is what you're going to be applying your model to, if prediction is your goal), then neither is your test data. Almost surely your training data are not perfectly representative of data you will see in the wild. Is it good enough? You have to check for these sorts of things.

Your overall perspective on the model evaluation process is backward. We need to be scientists about the matter. Being a scientist about the matter means that you're not looking for evidence that your model is right... it means meticulously looking for evidence that your model could be wrong. The depth that you have delved for ways that you could be wrong is the true test of any hypothesis. Your hypothesis is "this linear regression model will do a good job predicting future Y from future X." You want to look in every corner for things that might suggest that this hypothesis is wrong. Obviously, you cannot check every corner, because there are infinitely many corners, many of which are so dark that you cannot see into them. Still, you should check the corners that you can check. One of those corners you check by holding out a test set. This is usually to check the "overfitting" corner - Is my model just regurgitating the training data, or is it actually doing some kind of generalization? However, there are many other corners. One of these that is extremely common to crop up in real life is the "data isn't representative" corner. This means that you trained your model on the data that you had, but the data that you had isn't a very good picture of all of the data that the model sees in real life. Checking out the residuals is one way to convince yourself that this might be a way for your model to fail (and remember, you are meticulously looking for ways that your model can fail).

So, when you see this sort of thing:

enter image description here

You should be thinking to yourself "okay, well what happens when my model is confronted with $x=2.5$?" Why? Because the data "curves", and the model doesn't (i.e., my residuals aren't independent of X). If your model might well be confronted with $x=2.5$ in the real world, then your training data isn't very representative of the real world, is it? You can see this because we don't have anything like $x=2.5$ in our training data. If you don't have anything like $x=2.5$ in your training data, then you don't have anything like it in your test data either. Bootstrapping test data out of training data doesn't fix problems with your training data!

Anyway, the point here isn't that you should or shouldn't look at the residuals. The point here is that you are taking entirely the wrong perspective on how to evaluate your model. You shouldn't be focused on finding evidence that your model is good. You should be focused on finding evidence that your model is BAD! If you look really really hard for evidence that your model is bad, and don't find any, then and only then should you begin having confidence that your model is "good". Checking out the regression coefficient is one way to do this. Checking the residuals is another one of many ways to check for "common" pitfalls where models turn out to be "bad".

$\endgroup$
4
$\begingroup$

Accuracy is Everything

As others have noted, you want your model to be accurate before you start screaming from the rooftops that it is useful. If your regression says that candy consumption predicts weight loss, you better hope that is actually true by checking whether or not that is valid. To give a visual example, we can plot a typical regression in R with the iris dataset, predicting petal dimensions for each flower. After we can plot the residuals:

#### Fit Model ####
fit <- lm(Petal.Length ~ Petal.Width,
          iris)

#### Plot Residuals ####
plot(density(resid(fit)))

Which look fairly normal:

enter image description here

Bad Fit and Consequences

If we fit a really bad model, it can have grave consequences even if the predictors are significant. Consider if we decided to just add zero to the model this time and plot the residuals:

#### Misspecify Model ####
fit.bad <- lm(Petal.Length ~ 0,
              iris)

#### Plot Again ####
plot(density(resid(fit.bad)))

Then residuals are now strangely bimodal:

enter image description here

There is probably a clear reason why. Since the model actually isn't modeling anything (the syntax reads "average petal length is disaggregated by zero"), we are actually just getting the density of the raw values of petal length. As proof, we can run the following code:

#### Plot Density of Variable ####
plot(density(iris$Petal.Length))

And sure enough, the plot is exactly the same:

enter image description here

Takeaway Message

To ensure your model is behaving, always check performance. The results may come off as exciting until you find out your model isn't a model at all (or at least a very poor one).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.