19
$\begingroup$

I am a little bit confused on what the assumptions of linear regression are.

So far I checked whether:

  • all of the explanatory variables correlated linearly with the response variable. (This was the case)
  • there was any collinearity among the explanatory variables. (there was little collinearity).
  • the Cook's distances of the datapoints of my model are below 1 (this is the case, all distances are below 0.4, so no influence points).
  • the residuals are normally distributed. (this may not be the case)

But I then read the following:

violations of normality often arise either because (a) the distributions of the dependent and/or independent variables are themselves significantly non-normal, and/or (b) the linearity assumption is violated.

Question 1 This makes it sound as if the independent and depend variables need to be normally distributed, but as far as I know this is not the case. My dependent variable as well as one of my independent variables are not normally distributed. Should they be?

Question 2 My QQnormal plot of the residuals look like this:

normality check of residuals

That slightly differs from a normal distribution and the shapiro.test also rejects the null hypothesis that the residuals are from a normal distribution:

> shapiro.test(residuals(lmresult))
W = 0.9171, p-value = 3.618e-06

The residuals vs fitted values look like:

residuals vs fitted

What can I do if my residuals are not normally distributed? Does it mean the linear model is entirely useless?

$\endgroup$
  • 3
    $\begingroup$ Your residuals versus fitted plot suggests that your dependent variable has a lower bound. This could drive the patterns you see. This could give you an indications for alternative models you could consider. $\endgroup$ – Maarten Buis May 28 '14 at 19:21
22
$\begingroup$

First off, I would get yourself a copy of this classic and approachable article and read it: Anscombe FJ. (1973) Graphs in statistical analysis The American Statistician.27:17–21.

On to your questions:

Answer 1: Neither the dependent nor independent variable needs to be normally distributed. In fact they can have all kinds of loopy distributions. The normality assumption applies to the distribution of the errors ($Y_{i} - \hat{Y}_{i}$).

Answer 2: You are actually asking about two separate assumptions of ordinary least squares (OLS) regression:

  1. One is the assumption of linearity. This means that the relationship between $Y$ and $X$ is expressed by a straight line (Right? Straight back to algebra: $y = a +bx$, where $a$ is the $y$-intercept, and $b$ is the slope of the line.) A violation of this assumption simply means that the relationship is not well described by a straight line (e.g. $Y$ is a sinusoidal function of $X$, or a quadratic function, or even a straight line that changes slope at some point). My own preferred two-step approach to address nonlinearity is to (1) perform some kind of nonparametric smoothing regression to suggest specific nonlinear functional relationships between $Y$ and $X$ (e.g. using lowess, or GAMs, etc.), and (2) to specify a functional relationship using either a multiple regression that includes nonlinearities in $X$, (e.g. $Y \sim X + X^{2}$), or a nonlinear least squares regression model that includes nonlinearities in parameters of X (e.g. $Y \sim X + \max{(X-\theta,0)}$, where $\theta$ represents the point where the regression line of $Y$ on $X$ changes slope).

  2. Another is the assumption of normally distributed residuals. Sometimes one can validly get away with non-normal residuals in an OLS context; see for example, Lumley T, Emerson S. (2002) The Importance of the Normality Assumption in Large Public Health Data Sets. Annual Review of Public Health. 23:151–69. Sometimes, one cannot (again, see the Anscombe article).

However, I would recommend thinking about the assumptions in OLS not so much as desired properties of your data, but rather as interesting points of departure for describing nature. After all, most of what we care about in the world is more interesting than $y$-intercept and slope. Creatively violating OLS assumptions (with the appropriate methods) allows us to ask and answer more interesting questions.

$\endgroup$
  • 2
    $\begingroup$ Thanks! In the slides of some statistics course it says that if the assumptions fail you can try to transform Y or transform the explanatory variables. When I transform the Y by doing for example lm(Y^0.3~+X1+X2+...) then my residuals do become normally distributed. Is this a valid thing to do? $\endgroup$ – Stefan May 27 '14 at 18:16
  • $\begingroup$ @Stefan Yes! Transforming a response is often a good thing to do, log, and simple power transforms are common. $\endgroup$ – Gregor May 27 '14 at 22:00
  • $\begingroup$ @Stefan Maybe, maybe not. If you transform your outcome, then your inferences based on the transformed relationships do not necessarily apply to the inverse transformations after you have performed your analysis; this is because $\text{Var}(f(x) \ne f(\text{Var}(x))$. So if you analyze $\ln Y =\beta_{0} + \beta_{X}X + \varepsilon$, finding a significant $\beta_{X}$ does not necessarily translate into a significant $e^{\beta_{X}}$, nor does CI$\beta_{X}$ necessarily correspond to $e^{\text{CI}\beta_{X}}$. $\endgroup$ – Alexis May 27 '14 at 22:18
  • $\begingroup$ @Alexis: Why do these pages say that the variables have to be normally distributed? (1) pareonline.net/getvn.asp?n=2&v=8 (2) statisticssolutions.com/… $\endgroup$ – stackoverflowuser2010 Mar 7 '16 at 23:26
  • 5
    $\begingroup$ @stackoverflowuser2010 Because they do not know what they are talking about? The assumption is built right into the mathematical formalism: $Y = \beta_{0} + \beta_{X}X + \varepsilon$ where $\varepsilon \sim \mathcal{N}(0,\sigma)$. Note that last part: it's the residuals not the variables that are distributed normally. Look: (1) simulate X using a uniform distribution from, oh, say 0 to 100; (2) simulate $Y = 3 + 0.5\times X + \mathcal{N}(0,1)$; (3) regress $Y$ on $X$ and recover $\beta_{0}\approx 3, \beta_{X}\approx 0.5$. Then look at the histograms of $X$ and $Y$. $\endgroup$ – Alexis Mar 8 '16 at 0:20
9
$\begingroup$

Your first problems are

  • in spite of your assurances, the residual plot shows that the conditional expected response isn't linear in the fitted values; the model for the mean is wrong.

  • you don't have constant variance. The model for the variance is wrong.

you can't even assess normality with those problems there.

$\endgroup$
  • $\begingroup$ Please elaborate on how you've concluded about linearity by looking at the plots? I understand that the homoskedasticity assumption is not met here. $\endgroup$ – Dr Nisha Arora May 18 at 15:01
  • $\begingroup$ The conditional mean of residuals is changing as $\hat{y}$ changes; there's a clear down trend then a distinct jump up as we move right. If you can't see it, cut the plot into say 4 slices. I'd put the middle of the range of predicted values about $\hat{y}=30$, so cut it there, and then cut each half in half - say at $0$ and $60$. Now looking at the points within each of those slices ( $<0$,$0-30$, $30-60$, $>60$), draw your best estimate of a straight line. For me the middle two are nearly coincident, so I combined their lines, giving something like this $\endgroup$ – Glen_b May 19 at 5:40
  • $\begingroup$ In the middle half, nearly all the residuals are negative, in the outer parts nearly all the residuals are positive. These are not how random residuals look. $\endgroup$ – Glen_b May 19 at 5:44
  • $\begingroup$ Thanks, @Glen_b. After a long gap, I'm revisiting my concepts so couldn't visualize at first place. $\endgroup$ – Dr Nisha Arora May 19 at 8:46
3
$\begingroup$

I wouldn't say the linear model is completely useless. However, this means that your model doesn't correctly/fully explain your data. There is a part where you have to decide whether the model is "good enough" or not.

For your first question, I don't think that a linear regression model assumes that your dependent and independent variables have to be normal. However, there is an assumption about the normality of the residuals.

For your second question, there is two different things you could consider :

  1. Check different kind of models. Another model might be better to explain your data (for example, non-linear regression, etc). You would still have to check that the assumptions of this "new model" are not violated.
  2. Your data may not contain enough covariates (dependent variables) to explain the response (outcome). In this case, you cannot do anything else. Sometimes, we may accept to check if the residuals follow a different distributions (e.g. t-distribution) but it doesn't seem to be the case for you.

In addition to your question, I see that your QQPlot is not "normalized". Usually it is easier to look at the plot when your residuals are standardised, see stdres.

stdres(lmobject)

I hope it helps you, maybe someone else will explain this better than me.

$\endgroup$
0
$\begingroup$

In addition to previous answer, I would like to add some points to improve your model:

  1. Sometimes non-normality of residuals indicates presence of outliers. If this is the case, handle the outliers first.

  2. May be using some transformations solve the purpose.

  3. Additionally, to deal with multi-colinearity, you can refer https://www.researchgate.net/post/My_data_has_the_problem_of_multicolinearity_Removing_unique_variables_using_variance_inflation_factor_VIF_didnt_work_Any_solution

$\endgroup$
0
$\begingroup$

For your second question,

Something that happened to me in practice was that I was overfitting my response with many independent variables. In the overfitted model I had non normal residuals. Even though, the results stablished that there wasn´t enought evidence to discart the posibility that some coeficients were zero (with p-values grater than 0.2). So in a second model, dismissing variables following a backward selection procedure I got normal residuals validated both graphically with a qqplot and by hypotesis testing with a Shapiro-Wilk test. Check if this could be your case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.