31
$\begingroup$

I have been encountering many assumptions associated with linear regression (especially ordinary least squares regression) which are untrue or unnecessary. For example:

  • independent variables must have a Gaussian distribution
  • outliers are the points either above or below the upper or lower whiskers correspondingly (employing Boxplot terminology)
  • and that the sole purpose of transformations is to bring a distribution close to normal in order to suit the model.

I would like to know what are the myths that are generally taken for facts/assumptions about linear regression, especially concerning associated nonlinear transformations and distributional assumptions. How did such myths come about?

Improve this question
$\endgroup$
9
  • 4
    $\begingroup$ Linear regression refers more specifically to ordinary least squares regression? (or is this another misconception/assumption that regression equals minimizing the squared error?) $\endgroup$
    – Sextus Empiricus
    Feb 5, 2022 at 13:56
  • 14
    $\begingroup$ BTW, please keep each answer to one myth (or set of closely related ones). Otherwise the voting will be ambiguous and the answers will be more difficult to curate. $\endgroup$
    – whuber ♦
    Feb 5, 2022 at 16:01
  • 6
    $\begingroup$ Incidentally, this question is too broad: it asks about a large set of things. Could we please focus answers on issues directly related to ordinary least squares regression? Otherwise this thread will quickly become a mess and have to be closed. $\endgroup$
    – whuber ♦
    Feb 5, 2022 at 16:05
  • 3
    $\begingroup$ The question is closed because it needs more focus, but are there suggestions to bring more focus? Or is the closing about the unclear sentence 'the purpose of transformations, linear regression, and different kinds of distributions' which is very broad when taken literally. $\endgroup$
    – Sextus Empiricus
    Feb 6, 2022 at 12:28
  • 2
    $\begingroup$ Do you consider differencing to be included as a data transformation? If so, there are endless questions on this site where people are confused about whether and how stationarity is required for linear regression, if differencing should be applied first, etc. $\endgroup$
    – Chris Haug
    Feb 6, 2022 at 19:43

10 Answers 10

Reset to default
33
$\begingroup$

There are three myths that bother me.

  1. Predictor variables need to be normal.

  2. The pooled/marginal distribution of $Y$ has to be normal.

  3. Predictor variables should not be correlated, and if they are, some should be removed.

I believe that the first two come from misunderstanding the standard assumption about normality in an OLS linear regression, which assumes that the error terms, which are estimated by the residuals, are normal. It seems that people have misinterpreted this to mean that the pooled/marginal distribution of all $Y$ values has to be normal.

Indeed, as is mentioned in a comment, we still get a lot of what we like about OLS linear regression without having normal error terms, though we do not need a normal marginal distribution of $Y$ or normal features in order to have the OLS estimator coincide with maximum likelihood estimation.

For the myth about correlated predictors, I have two hypotheses.

  1. People misinterpret the Gauss-Markov assumption about error term independence to mean that the predictors are independent.

  2. People think they can eliminate features to get strong performance with fewer variables, decreasing the overfitting.

I understand the idea of dropping predictors in order to have less overfitting risk without sacrificing much of the information in your feature space, but that seems not to work out. My post here gets into why and links to further reading.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you @Dave for such a riveting demonstration of the facts. But on your 3rd point, isn't the first half of it; "Predictor variables should not be correlated" a norm? Isn't it true that multicollinearity is a matter of concern for inference when it comes to making sense of the parameters associated with the the predictors? Does multicollinearity not lead to vague CIs of the parameters? In fact, when making dummy variables of categorical data, I would drop one of the dummies in order to avoid multicollinearity. How do you defend your third argument then? $\endgroup$
    – Ritik P. Nayak
    Feb 6, 2022 at 19:46
  • 2
    $\begingroup$ @RitikP.Nayak Yes, confidence intervals are wider when features are correlated. If you start screening features for correlation, however, you’re influencing downstream inference. If you want uncorrelated features in order to have the narrowest confidence intervals you can have, that might call for a designed experiment. $\endgroup$
    – Dave
    Feb 6, 2022 at 19:59
  • $\begingroup$ Thank you @Dave . I'm with you on it. But I request a comment on the second point too that I made in the earlier comment; "when making dummy variables of categorical data, I would drop one of the dummies in order to avoid multicollinearity". To add to this, does multicollinearity doesn't at all matter for prediction purposes? $\endgroup$
    – Ritik P. Nayak
    Feb 6, 2022 at 20:07
  • $\begingroup$ You’re not dropping a category; you’re letting it be subsumed by the intercept. $\endgroup$
    – Dave
    Feb 6, 2022 at 20:23
  • 1
    $\begingroup$ Even normality of the error terms isn't needed for most things. The only thing that will buy you is efficiency. OLS will be unbiased and/or consistent even if errors aren't normally distributed. $\endgroup$
    – RoyalTS
    Feb 7, 2022 at 9:35
24
$\begingroup$

Myth

A linear regression model can only model linear relationships between the outcome $y$ and the explanatory variables.

Fact

Despite the name, linear regression models can easily accomodate nonlinear relationships using polynomials, fractional polynomials, splines and other methods. The term "linear" in linear regression pertains to the fact that the model is linear in the parameters $\beta_0, \beta_1, \ldots$. For an in-depth explanation about the term "linear" with regards to models, I highly recommend this post.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I'm confused by the statement of the "myth" -- it's true that linear-regression models can only model linear relationships (pretty much by-definition). I think you're kinda mixing two different points together: (1) that the term "linear" can mean different things (which seems to be the main focus of the linked answer); (2) that non-linear problems can be approximated as linear problems to enable linear-regression (e.g., doing a linear-regression on $y=mz+b$ where $z\equiv x^2$ as an approximation of doing a non-linear regression on $y=mx^2+b ).$ $\endgroup$
    – Nat
    Feb 5, 2022 at 20:03
  • 4
    $\begingroup$ @Nat I encourage you to read through the linked post. The word "linear" in linear regression means linear in the parameters, not linear in $x$. The model $y=mx^2+b$ would therefore not be called nonlinear regression, because it's still linear in its parameters. It's a linear model of a nonlinear relationship. $\endgroup$
    – COOLSerdash
    Feb 5, 2022 at 20:09
  • 2
    $\begingroup$ An example of a nonlinear problem would be $y= b_0+ x^{b_1} + \text{error}$, where you cannot describe $y$ in terms of a linear sum of the $b$ coefficients you are solving for. An example of how this model would show up is if you wanted to measure the decay rate of something (e.g., heart rate during exercise), but the rate and the asymptotic bounds both depend on independent variables (e.g., initial measurement, time passed, temperature, and subject ID). $\endgroup$
    – Max Candocia
    Feb 5, 2022 at 20:30
  • 1
    $\begingroup$ @Max If you intend $x$ to be an explanatory variable and $b_1$ to be a parameter, that is not the kind of "nonlinear relationship" described in this answer, because $b_1$ does not enter the model in a linear fashion (nor can the model be transformed to make it so). $\endgroup$
    – whuber ♦
    Feb 5, 2022 at 21:56
20
$\begingroup$

@Dave's answers are excellent. Here are some more myths.

  1. The original scale/transformation for Y is one you should use in the model.
  2. The central limit theorem means you don't have to worry about any of this if N is moderately large.
  3. Trying different transformations for Y does not distort standard errors, p-values, or confidence interval widths.
Improve this answer
$\endgroup$
5
  • 5
    $\begingroup$ If I could, I would give a separate upvote for each of these points because they address issues that appear to be particularly poorly understood by people who visit this site. $\endgroup$
    – whuber ♦
    Feb 5, 2022 at 14:18
  • 1
    $\begingroup$ I need an explanation about the second point about the central limit theorem, because I just posted an answer that is about the opposite statement 'That you need to worry about it' although I have tried to make it more nuanced and make it not about the CLT and instead phrased it to be about the 'principle behind the CLT' and tried to focus on the idea of the limiting behaviour... $\endgroup$
    – Sextus Empiricus
    Feb 5, 2022 at 15:37
  • $\begingroup$ .. is that the idea behind the second point, the myth of the CLT? That people make the CLT, which is about the behaviour of a limit at infinity, equal to the the behaviour of statistics that relate to samples with a finite size? $\endgroup$
    – Sextus Empiricus
    Feb 5, 2022 at 15:40
  • 1
    $\begingroup$ Yes and many, many people misunderstand the fact that if you should have taken log(Y) instead of Y (or square root or some other nonlinear transformation) the CLT will not rescue you. $\endgroup$
    – Frank Harrell
    Feb 5, 2022 at 16:30
  • 5
    $\begingroup$ This answer could be greatly improved by adding some more context (like most answers here have done) by e.g. clearly stating what the truth is that stands opposed to the myths, explaining why the myths are wrong and/or trying to explain what might be causing people to believe the myths. $\endgroup$
    – NotThatGuy
    Feb 6, 2022 at 4:02
15
$\begingroup$

Myth: Variables that are not "significant" should be removed from a multiple regression.

See When should one include a variable in a regression despite it not being statistically significant? for a discussion. Then search our site for "model identification," "regularization," "Lasso," etc.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you @whuber . I have upvoted and accepted your answer. But let's for moment not look it from the 'statistical significance' lens. Let's talk about the correlation between the IV and DV. Let's say for instance, that I have 3 predictors in a dataset (say, x1, x2, and x3). There's no linear relationship between the three. But, x1 and x2 are highly correlated with the DV (say, y) whereas x3 has a very weak correlation with the DV, say a correlation coefficient of 0.2. Does this then become a suitable reason for one to exclude x3 from making predictions? $\endgroup$
    – Ritik P. Nayak
    Feb 8, 2022 at 10:09
  • $\begingroup$ The answer is a very strong, resounding, no. The whole basket of variables matters in multiple regression: removing or adding just one can completely change the picture. The raw (bivariate) correlation tells you even less than the multiple regression p-value might. For insight into what goes on in multiple regression, see stats.stackexchange.com/a/28493/919. For additional examples and discussion see stats.stackexchange.com/a/34813/919, stats.stackexchange.com/a/372416/919, etc. $\endgroup$
    – whuber ♦
    Feb 8, 2022 at 15:15
12
$\begingroup$

Myths:

  • The normality of residuals (and possibly other assumptions of the model) should be tested with a formal hypothesis test, such as the Shapiro-Wilk test.
  • A small $p$-value of such tests indicates that the model is invalid.

Facts:

  • Formal test of normality (and of other assumptions such as homoskedasticity) do not answer the relevant questions and if they are used to guide subsequent actions, can distort the operating characteristic of the models (e.g. inflating type 1 errors, change distribution of $p$-values under the null etc.).
  • A "significant" Shapiro-Wilk test of the residuals just indicates some degree of incompatibiltiy with a normal distribution. It does not say that the (inevitable) deviation from a normal distribution is meaningful or impactful concerning the operating characteristics of the model. Some aspects - e.g. prediction intervals - are more sensitive with regards to the distribution of the errors than others. The $t$-test of the coefficients are reasonably robust (with regards to type 1 errors), for example. Whether or not the deviation of the residuals from a normal distribution is worrysome depends on a number of things: the goal of the analysis, the sample size, the degree of deviation, and more.
Improve this answer
$\endgroup$
0
11
$\begingroup$

Myth: You should always standardize (or somehow "normalize") variables for the purpose of fitting regression models.

Usually not: software will either do this automatically (under the hood, as it were) or uses algorithms that accommodate huge ranges of values among the variables without losing numerical precision.

When the order of magnitude of one explanatory variable is more than about eight times greater than of another variable, though, then watch out: even preliminary standardization can run into trouble. ("Eight" orders of magnitude is the square root of double precision, which is about 15.6 orders of magnitude.) The commonest example is when a date is used along with other variables, because some dates are represented as the number of seconds elapsed since approximately 1970, which is on the order of $10^9$ seconds.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ @Max I cannot discern anything in your comment that appears relevant to this issue. Perhaps you have a different conception of what "standardize" and "normalize" mean in statistics. The former means to recenter and rescale a variable to achieve zero mean and unit variance while the latter applies to a comparable affine transformation to place the variable's range at $0$ to $1.$ $\endgroup$
    – whuber ♦
    Feb 5, 2022 at 21:13
7
$\begingroup$

Where do these ideas come from?

Poor texts (correction: very poor texts) after treating descriptive statistics often include some more or less mangled version of an idea that (1) you ideally need normally distributed variables to do anything inferential, or else (2) you need non-parametric tests. Then they may or may not mention that transformations could get you nearer (1).

The first context for writing like ,this is often Student $t$ tests for comparing means and (Pearson) correlations. There is some historical context for this, for example in treatments that focused on a reference case of a bivariate normal distribution with a correlation as one parameter.

So then writers start talking about regression.

These texts are usually innocent of any formal specification of a data generation process.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Regarding sources, do not forget tons of machine learning tutorials, blog posts, YouTube videos and such the authors of which have never understood (or even studied) statistics at an appropriate level. $\endgroup$
    – Richard Hardy
    Feb 5, 2022 at 18:39
5
$\begingroup$

"Also, how did such myths come about?"

One common assumption in regression is homoscedasticity (and a myth is that this is also necessary). Transformations are used to bring the data closer to this assumption.

The violation of the assumption doesn't make the fitting method bad, least squares regression is the best unbiased linear estimator (in terms of lowest variance of the estimates) no matter what the underlying distribution is.

But, the violation of the assumptions may cause wrong inferences when we express the observed effects in terms of significance/p-values.

There is a difference between the assumptions that are necessary for least squares regression to work, and the assumptions that are necessary for the significance and hypothesis tests based on least squares regression to work.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Is OLS still blue under heteroskedasticity? Would that not be GLS instead? $\endgroup$
    – Richard Hardy
    Feb 5, 2022 at 18:44
  • 4
    $\begingroup$ A hobby-horse is that we would be better off not talking about assumptions but more about ideal conditions. Blame is shared by those don't teach the difference between (1) assumptions that imply theorems (2) ideal conditions that are often Utopian on the one hand and often not crucial on the other. But statistics (like many others) is a spiral subject that requires many cycles of learning and unlearning. Much still boils down to assertions from experience with flavour like "With this kind of data, logarithms are always a good idea" or "That assumption doesn't usually bite hard". $\endgroup$
    – Nick Cox
    Feb 5, 2022 at 18:50
2
$\begingroup$

Myth: The error/deviation of the observations needs to be normally distributed.

No, it doesn't.

It is not just about the distribution of the errors of the observations, Instead what often matters is the distribution of the error of the estimates.

These estimates are computed as a weighted sum of the observations $$\hat\beta = M \cdot y$$ with $$M = (X^TX)^{-1}X^T$$

If we want to estimate the error or significance of the estimates $\hat\beta$, then it is sufficient if those estimates follow approximately a normal distribution. This can happen also when the sampled error of observations $y$ do not follow a normal distribution.

Due to the same principle of the central limit theorem, a statistic that is a weighted sum of variables or some sort of mean of variables will approach a normal distribution.

So even if the distribution of the error/deviation of the observations is not normally distributed, the error/deviation estimates might still be approximately normally distributed.

Improve this answer
$\endgroup$
8
  • $\begingroup$ This sounds like a common misconception about the central limit theorem. $\endgroup$
    – Dave
    Feb 5, 2022 at 15:36
  • $\begingroup$ @Dave I mentioned on purpose the 'principle of the central limit theorem' and not plainly 'the central limit theorem'. $\endgroup$
    – Sextus Empiricus
    Feb 5, 2022 at 15:44
  • 1
    $\begingroup$ The fact is that the estimates from OLS are a linear combination of observations $$\hat\beta = M \cdot y$$ with $$M = (X^TX)^{-1}X^T$$ For the distribution of sums we know that they approach normal distributions. The CLT is the statement about the limit for a sample that becomes infinitely large, but the principle that "sums of random variables can be approximated by a normal distribution" is not less true because of that. Sure, it doesn't work for all types of variables. The point is that the OLS estimate is about the distribution of the estimate and not the distribution of the errors. $\endgroup$
    – Sextus Empiricus
    Feb 5, 2022 at 15:51
  • 2
    $\begingroup$ The CLT helps here far less than you think. Often when the residuals are skewed the wrong transformation of Y was used. This can ruin all the $\beta$s. $\endgroup$
    – Frank Harrell
    Feb 5, 2022 at 16:29
  • 1
    $\begingroup$ @RichardHardy I have edited the post. Feel free to improve it further. I believe you get the point that I wish to make. $\endgroup$
    – Sextus Empiricus
    Feb 6, 2022 at 9:03
-3
$\begingroup$

Myth: If the histogram of the residuals is nicely bell-shaped, and if the normal q-q plot of the residuals is very close to a straight line (and the sample size is reasonably large so that sampling error is minor), then the normality assumption is reasonable.

Improve this answer
$\endgroup$
5
  • 3
    $\begingroup$ This needs elaboration: which "normality assumption" are you referring to and why do you call this a "myth"? One overarching problem is that your assertion is completely qualitative and subjective: how are we intended to understand "nicely bell-shaped," "very close to .. straight," and "reasonable"? Readers might be surprised but not enlightened. $\endgroup$
    – whuber ♦
    Feb 5, 2022 at 21:53
  • $\begingroup$ It's all just textbook stuff. $\endgroup$
    – BigBendRegion
    Feb 5, 2022 at 22:41
  • 7
    $\begingroup$ That doesn't clarify anything. Also, since many sources state the opposite of this answer, some explanation is needed! $\endgroup$
    – whuber ♦
    Feb 5, 2022 at 22:53
  • 3
    $\begingroup$ Without further elaboration, this one seems to be wrong. $\endgroup$
    – Dave
    Feb 6, 2022 at 19:13
  • $\begingroup$ Elaboration: the behaviour I indicated easily occurs when $Y$ is highly discrete. $\endgroup$
    – BigBendRegion
    Aug 15, 2022 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.