2
$\begingroup$

I have a multiple linear regression with about 20 significant predictors - some categorical and come continuous. I ran the model in Statsmodel in Python.

I get a high adj R^2 of approximately 0.95 which suggests good fit. I ran a predicted vs. actual plot (shown below) and have good linearity.

enter image description here

However, I'm having problems when I check assumptions. My residuals to not appear to be normally distributed.

My residuals vs predicted values plot looks like this:

enter image description here

I look at this and depending on the scale, conclude that the residuals MIGHT be randomly distributed around a mean of zero if the scale were changed, that there is "minimal" hetroscedacity, and there are some outliers.

However, if I plot a residuals histogram I get this:

enter image description here

Which indicates the that the residuals may be distributed symmetrically around a mean but not normally distributed.

If I plot a qq of the residuals I get this:

enter image description here

Which I understand to be a fat-tailed distribution.

So my questions are:

  1. The linearity suggests the model is strong but the residual plots suggest the model is unstable. How do I reconcile? Is this a good model or an unstable one?

  2. If the model is unstable, how can I transform the variables (independent, dependent, both) to get my residuals normally distributed while maintaining strong linearity. I've tried various transformations (log, ln, box cox, etc) on the dependent variable, all independent variables, and some independent variables and all it does is destroy the linearity while not fixing the residual distribution.

Am I missing something obvious?

Thanks in advance for help and suggestions.

Improve this question
$\endgroup$
6
  • $\begingroup$ In what sense is your model "unstable"? Have you looked at diagnostic measures of influence and leverage? $\endgroup$
    – whuber ♦
    Oct 26, 2016 at 14:08
  • $\begingroup$ Not yet. But the assumptions when doing MLR are 1) linear relationship; 2) error between observed and predicted values normally distributed 3) little or no multicollinearity; 3) homoscedacity; 4) no autocorrellation. I'm violating #2. $\endgroup$
    – Windstorm1981
    Oct 26, 2016 at 14:32
  • $\begingroup$ #2 is an unusually restrictive assumption. It is needed (in general) only for F tests with relatively small amounts of data. It has little bearing on "stability" of the model. $\endgroup$
    – whuber ♦
    Oct 26, 2016 at 14:34
  • $\begingroup$ @whuber thank you for the perspective. So in terms of importance (I get that there is always interpretation) is the importance of the assumptions in order of decreasing considerations something like this: 1) linear relationship; 2) homoscedacticty and 3) little or no multi collinearity and 4) no autorcorrelation, and then 5) normally distributed error? $\endgroup$
    – Windstorm1981
    Oct 26, 2016 at 14:43
  • $\begingroup$ It depends on what you are using the model to do. For more perspective please see stats.stackexchange.com/questions/32600 $\endgroup$
    – whuber ♦
    Oct 26, 2016 at 14:47

1 Answer 1

Reset to default
2
$\begingroup$

I have run into this kind of situation many a time myself. Here are a few comments from my experience. Rarely is it the case that you see a QQ plot that lines up along a straight line.

  1. The linearity suggests the model is strong but the residual plots suggest the model is unstable. How do I reconcile? Is this a good model or an unstable one?

Response: The curvy QQ plot does not invalidate your model. But, there seems to be way too many variables (20) in your model. Are the variables chosen after variable selection such as AIC, BIC, lasso, etc? Have you tried cross-validation to guard against overfitting? Even after all this, your QQ plot may look curvy. You can explore by including interaction terms and polynomial terms in your regression, but a QQ plot that does not line up nicely in a straight line is a not a substantial issue in practical terms.

Say you are comfortable with retaining all 20 predictors. You can, at a minimum, report White or Newey-West standard errors to adjust for collinearity among the 20 predictors as well as autocorrelation and heteroskedasticity. Your residual plots indicate few clear outliers. You can drop these observations and your QQ plot will look less curvy.

  1. If the model is unstable, how can I transform the variables (independent, dependent, both) to get my residuals normally distributed while maintaining strong linearity. I've tried various transformations (log, ln, box cox, etc) on the dependent variable, all independent variables, and some independent variables and all it does is destroy the linearity while not fixing the residual distribution.

Response: The transformations you tried are all good to try. You need not be fixated on fixing the residuals plot. Even if the QQ plot does not line up on a straight line, your estimated OLS coefficients are still unbiased and consistent. What is impacted is your standard errors of those OLS coefficients, and you can apply common fixes such as White, Newey-West, or boostrapping to get a conservative estimate of the standard errors so that you do not conclude a coefficient is significant when it is not.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks very much for the comments and perspective. I was taught that satisfying the assumptions are "very important - otherwise your model is unstable" which is why I'm so fixated. I should add that I'm really only at the outset here. I agree that 20 predictors is too many. Have not gone through any kind of feature selection yet. Just approaching this one step at a time and since I got such good fit (.95 R^2) I just wanted to resolve what I thought to be the key issue with the normality of residuals before I moved on to refine the model. $\endgroup$
    – Windstorm1981
    Oct 26, 2016 at 14:38
  • $\begingroup$ ...I thought what I had here (non-normal distribution or residuals) invalidated the model at the outset but was confused since I had such great fit before I did any more work - but if I understand you correctly @David the model is ok despite the funky qq? $\endgroup$
    – Windstorm1981
    Oct 26, 2016 at 14:39
  • $\begingroup$ could you explain this part a little more simply? TY "What is impacted is your standard errors of those OLS coefficients, and you can apply common fixes such as White, Newey-West, or boostrapping to get a conservative estimate of the standard errors so that you do not conclude a coefficient is significant when it is not." $\endgroup$
    – Windstorm1981
    Oct 26, 2016 at 14:45
  • $\begingroup$ If you can justify retaining a large number of predictors, and mitigate the risk of overfitting by, say, cross valiation, your model seems OK. If the QQ plot lines up nicely, then great, and even if not, it's not a deal breaker. $\endgroup$
    – DavidH
    Oct 26, 2016 at 16:21
  • 1
    $\begingroup$ Normality assumption is not needed for OLS coefficients to be BLUE (BestLinearUnbiasedEstimator). The formula for deriving coefficients doesn't use nor need normality. However, when you want to make inferences about your OLS coefficients, then normality assumption becomes material. Rarely will all the OLS assumptions be met in practice. Faced with curvy QQ plot (i,e. some evidence of non-normal error terms), you can use bootstrap to construct an empirical confidence intervals (CI's) of your OLS estimates based on the realized residuals from your model, and use those CI's to perform inference. $\endgroup$
    – DavidH
    Oct 26, 2016 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.