28
$\begingroup$

When fitting a regression model, what happens if the assumptions of the outputs are not met, specifically:

  1. What happens if the residuals are not homoscedastic? If the residuals show an increasing or decreasing pattern in Residuals vs. Fitted plot.
  2. What happens if the residuals are not normally distributed, and fail the Shapiro-Wilk test? Shapiro-Wilk test of normality is a very strict test, and sometimes even if the Normal-QQ plot looks somewhat reasonable, the data fails the test.
  3. What happens if one or more predictors are not normally distributed, do not look right on the Normal-QQ plot or if the data fails the Shapiro-Wilk test?

I understand that there is no hard black and white division, that 0.94 is right and 0.95 is wrong, and in the question, I want to know:

  1. What does failing the normality means for a model that is a good fit according to the R-Squared value. Does it become less reliable, or completely useless?
  2. To what extent, the deviation is acceptable, or is it acceptable at all?
  3. When applying transformations on the data to meet the normality criteria, does the model gets better if the data is more normal (higher P-value on Shapiro-Wilk test, better looking on normal Q-Q plot), or it is useless (equally good or bad compared to the original) until the data passes normality test?
$\endgroup$
  • $\begingroup$ I think the answer to the title only is "Yes". $\endgroup$ – Thomas Cleberg Dec 29 '15 at 23:54
  • $\begingroup$ @ThomasCleberg Interesting answer. Is that what you also say when people ask you "How are you?" :) $\endgroup$ – JohnK Dec 29 '15 at 23:59
  • $\begingroup$ No, but it is if they ask me if I'm alive. :) $\endgroup$ – Thomas Cleberg Dec 30 '15 at 0:00
  • $\begingroup$ A basic question to ask yourself: "What do you want to use the regression model for?" $\endgroup$ – Floris Dec 31 '15 at 13:27
31
$\begingroup$

What happens if the residuals are not homoscedastic? If the residuals show an increasing or decreasing pattern in Residuals vs. Fitted plot.

If the error term is not homoscedastic (we use the residuals as a proxy for the unobservable error term), the OLS estimator is still consistent and unbiased but is no longer the most efficient in the class of linear estimators. It is the GLS estimator now that enjoys this property.

What happens if the residuals are not normally distributed, and fail the Shapiro-Wilk test? Shapiro-Wilk test of normality is a very strict test, and sometimes even if the Normal-QQ plot looks somewhat reasonable, the data fails the test.

Normality is not required by the Gauss-Markov theorem. The OLS estimator is still BLUE but without normality you will have difficulty doing inference, i.e. hypothesis testing and confidence intervals, at least for finite sample sizes. There is still the bootstrap, however.

Asymptotically this is less of a problem since the OLS estimator has a limiting normal distribution under mild regularity conditions.

What happens if one or more predictors are not normally distributed, do not look right on the Normal-QQ plot or if the data fails the Shapiro-Wilk test?

As far as I know the predictors are either considered fixed or the regression is conditional on them. This limits the effect of non-normality.

What does failing the normality means for a model that is a good fit according to the R-Squared value. Does it become less reliable, or completely useless?

The R-squared is the proportion of the variance explained by the model. It does not require the normality assumption and it's a measure of goodness of fit regardless. If you want to use it for a partial F-test though, that is quite another story.

To what extent, the deviation is acceptable, or is it acceptable at all?

Deviation from normality you mean, right? It really depends on your purposes because as I said, inference becomes hard in the absence of normality but is not impossible (bootstrap!).

When applying transformations on the data to meet the normality criteria, does the model gets better if the data is more normal (higher P-value on Shapiro-Wilk test, better looking on normal Q-Q plot), or it is useless (equally good or bad compared to the original) until the data passes normality test?

In short, if you have all the Gauss-Markov assumptions plus normality then the OLS estimator is Best Unbiased (BUE), i.e. the most efficient in all classes of estimators - the Cramer-Rao Lower Bound is attained. This is desirable of course but it's not the end of world if it does not happen. The above remarks apply.

Regarding transformations, bear in mind that while the distribution of the response might be brought closer to normality, interpretation might not be straightforward afterwards.

These are just some short answers to your questions. You seem to be particularly concerned with the implications of non-normality. Overall, I would say that it is not as catastrophic as people (have been made to?) believe and there are workarounds. The two references I have included are a good starting point for further reading, the first one being of theoretical nature.

References:

Hayashi, Fumio. : "Econometrics.", Princeton University Press, 2000

Kutner, Michael H., et al. "Applied linear statistical models.", McGraw-Hill Irwin, 2005.

$\endgroup$
  • $\begingroup$ Regading the point that Gauss-Markov assumptions plus normality imply that OLS is the most efficient of all estimators (not just linears), I'd stress the fact that one of said assumptions is that the conditional mean of $Y$ w.r.t the $X_i$ is linear in the parameters $\beta_i$. If you're assuming that the underlying model is linear, then it's not surprising that a linear estimator (OLS) turns out to beat all others estimator (whether linear or not). $\endgroup$ – DeltaIV Dec 30 '15 at 13:47
  • 2
    $\begingroup$ @DeltaIV I think you are confused since we are talking about linear estimators with respect to the response, $\mathbf{y}$. $\endgroup$ – JohnK Dec 30 '15 at 13:56
  • $\begingroup$ Well, both points are true, actually. One of the hypotheses of G-M is indeed that the ideal model is linear in the $\beta_i$, see: en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem At the same time, it's true what you say: OLS is an estimator of the $\beta_i$, not of $Y$. Thus, when we say it's linear, we mean that it's linear in the i.i.d. random variables $Y_1,\ldots,Y_n$. $\endgroup$ – DeltaIV Dec 30 '15 at 14:44
  • $\begingroup$ @DeltaIV What do you mean by "ideal model"? It's the true model that is linear in the parameters. That does not restrict us however in considering as estimators only linear functions of the response. The G-M states that if we restrict our attention in linear functions of the response, then the OLS is BLUE under some additional assumptions. Now, if we assume normality too then no matter what function of the response you are considering, you simply cannot do better than the OLS, provided of course that the estimator is unbiased. $\endgroup$ – JohnK Dec 30 '15 at 14:49
  • $\begingroup$ ideal model = real model. Sure, we could consider nonlinear functions of the $Y_i$ to estimate the $\beta_i$. I tried to explain that in my second comment, I think we agree. $\endgroup$ – DeltaIV Dec 30 '15 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.