Assumptions of linear models and what to do if the residuals are not normally distributed

Question

I am a little bit confused on what the assumptions of linear regression are.

So far I checked whether:

• all of the explanatory variables correlated linearly with the response variable. (This was the case)
• there was any collinearity among the explanatory variables. (there was little collinearity).
• the Cook's distances of the datapoints of my model are below 1 (this is the case, all distances are below 0.4, so no influence points).
• the residuals are normally distributed. (this may not be the case)

But I then read the following:

violations of normality often arise either because (a) the distributions of the dependent and/or independent variables are themselves significantly non-normal, and/or (b) the linearity assumption is violated.

Question 1 This makes it sound as if the independent and depend variables need to be normally distributed, but as far as I know this is not the case. My dependent variable as well as one of my independent variables are not normally distributed. Should they be?

Question 2 My QQnormal plot of the residuals look like this:

That slightly differs from a normal distribution and the shapiro.test also rejects the null hypothesis that the residuals are from a normal distribution:

> shapiro.test(residuals(lmresult))
W = 0.9171, p-value = 3.618e-06

The residuals vs fitted values look like:

What can I do if my residuals are not normally distributed? Does it mean the linear model is entirely useless?

Answer 1

First off, I would get yourself a copy of this classic and approachable article and read it: Anscombe, F. J. (1973) Graphs in statistical analysis. The American Statistician. 27:17–21.

On to your questions:

Answer 1: Neither the dependent nor independent variable needs to be normally distributed. In fact they can have all kinds of loopy distributions. The normality assumption applies to the distribution of the errors ($Y_{i} - \widehat{Y}_{i}$).

Answer 2: You are actually asking about two separate assumptions of ordinary least squares (OLS) regression:

1. One is the assumption of linearity. This means that the trend in $\overline{Y}$ across $X$ is expressed by a straight line (Right? Straight back to algebra: $y = a +bx$, where $a$ is the $y$-intercept, and $b$ is the slope of the line.) A violation of this assumption simply means that the relationship is not well described by a straight line (e.g., $\overline{Y}$ is a sinusoidal function of $X$, or a quadratic function, or even a straight line that changes slope at some point). My own preferred two-step approach to address non-linearity is to (1) perform some kind of non-parametric smoothing regression to suggest specific nonlinear functional relationships between $Y$ and $X$ (e.g., using LOWESS, or GAMs, etc.), and (2) to specify a functional relationship using either a multiple regression that includes nonlinearities in $X$, (e.g., $Y \sim X + X^{2}$), or a nonlinear least squares regression model that includes nonlinearities in parameters of $X$ (e.g., $Y \sim X + \max{(X-\theta,0)}$, where $\theta$ represents the point where the regression line of $\overline{Y}$ on $X$ changes slope).

2. Another is the assumption of normally distributed residuals. Sometimes one can validly get away with non-normal residuals in an OLS context; see for example, Lumley T, Emerson S. (2002) The Importance of the Normality Assumption in Large Public Health Data Sets. Annual Review of Public Health. 23:151–69. Sometimes, one cannot (again, see the Anscombe article).

However, I would recommend thinking about the assumptions in OLS not so much as desired properties of your data, but rather as interesting points of departure for describing nature. After all, most of what we care about in the world is more interesting than $y$-intercept and slope. Creatively violating OLS assumptions (with the appropriate methods) allows us to ask and answer more interesting questions.

Answer 2

Your first problems are

• in spite of your assurances, the residual plot shows that the conditional expected response isn't linear in the fitted values; the model for the mean is wrong.

• you don't have constant variance. The model for the variance is wrong.

you can't even assess normality with those problems there.

Answer 3

The most accessible exploration of the impact of non-normal errors that I have found is this paper by Schmidt and Finan.

Here is the summary of the results in the abstract:

Although outcome transformations bias point estimates, violations of the normality assumption in linear regression analyses do not. The normality assumption is necessary to unbiasedly estimate standard errors, and hence confidence intervals and P-values. However, in large sample sizes (e.g., where the number of observations per variable is >10) violations of this normality assumption often do not noticeably impact results. Contrary to this, assumptions on, the parametric model, absence of extreme observations, homoscedasticity, and independency of the errors, remain influential even in large sample size settings.

Answer 4

In addition to the previous answer, I would like to add some points to improve your model:

1. Sometimes non-normality of residuals indicates the presence of outliers. If this is the case, handle the outliers first.

2. Maybe using some transformations solve the purpose however, it has consequences. Like the interpretation of coefficients changes if we transform variables.

3. Additionally, to deal with multi-colinearity, you can refer https://www.researchgate.net/post/My_data_has_the_problem_of_multicolinearity_Removing_unique_variables_using_variance_inflation_factor_VIF_didnt_work_Any_solution

Answer 5

I wouldn't say the linear model is completely useless. However, this means that your model doesn't correctly/fully explain your data. There is a part where you have to decide whether the model is "good enough" or not.

For your first question, I don't think that a linear regression model assumes that your dependent and independent variables have to be normal. However, there is an assumption about the normality of the residuals.

For your second question, there is two different things you could consider :

1. Check different kind of models. Another model might be better to explain your data (for example, non-linear regression, etc). You would still have to check that the assumptions of this "new model" are not violated.
2. Your data may not contain enough covariates (dependent variables) to explain the response (outcome). In this case, you cannot do anything else. Sometimes, we may accept to check if the residuals follow a different distributions (e.g. t-distribution) but it doesn't seem to be the case for you.

In addition to your question, I see that your QQPlot is not "normalized". Usually it is easier to look at the plot when your residuals are standardised, see stdres.

stdres(lmobject)

I hope it helps you, maybe someone else will explain this better than me.

Answer 6

For your second question,

Something that happened to me in practice was that I was overfitting my response with many independent variables. In the overfitted model I had non normal residuals. Even though, the results stablished that there wasn´t enought evidence to discart the posibility that some coeficients were zero (with p-values grater than 0.2). So in a second model, dismissing variables following a backward selection procedure I got normal residuals validated both graphically with a qqplot and by hypotesis testing with a Shapiro-Wilk test. Check if this could be your case.