What are some of the most common misconceptions about linear regression?

Question

I'm curious, for those of you who have extensive experience collaborating with other researchers, what are some of the most common misconceptions about linear regression that you encounter?

I think can be a useful exercise to think about common misconceptions ahead of time in order to

1. Anticipate people's mistakes and be able to successful articulate why some misconception is incorrect

2. Realize if I am harboring some misconceptions myself!

A couple of basic ones I can think of:

Independent/Dependent variables must be normally distributed

Variables must be standardized for accurate interpretation

Any others?

All responses are welcome.

Answer 1

False premise: A $\boldsymbol{\hat{\beta} \approx 0}$ means that there is no strong relationship between DV and IV.
Non-linear functional relationships abound, and yet data produced by many such relationships would often produce nearly zero slopes if one assumes the relationship must be linear, or even approximately linear.

Relatedly, in another false premise researchers often assume—possibly because many introductory regression textbooks teach—that one "tests for non-linearity" by building a series of regressions of the DV onto polynomial expansions of the IV (e.g., $Y \sim \beta_{0} + \beta_{X}X + \varepsilon$, followed by $Y \sim \beta_{0} + \beta_{X}X + \beta_{X^{2}}X^{2} + \varepsilon$, followed by $Y \sim \beta_{0} + \beta_{X}X + \beta_{X^{2}}X^{2} + \beta_{X^{3}}X^{3} + \varepsilon$, etc.). Just as straight line cannot well represent a non-linear functional relationship between DV and IV, a parabola cannot well represent literally an infinite number of nonlinear relationships (e.g., sinusoids, cycloids, step functions, saturation effects, s-curves, etc. ad infinitum). One may instead take a regression approach that does not assume any particular functional form (e.g., running line smoothers, GAMs, etc.).

A third false premise is that increasing the number of estimated parameters necessarily results in a loss of statistical power. This may be false when the true relationship is non-linear and requires multiple parameters to estimate (e.g., a "broken stick" function requires not only the intercept and slope terms of a straight line, but requires point at which slope changes and a how much slope changes by estimates also): the residuals of a misspecified model (e.g., a straight line) may grow quite large (relative to a properly specified functional relation) resulting in a lower rejection probability and wider confidence intervals and prediction intervals (in addition to estimates being biased).

Answer 2

It's very common to assume that only $y$ data are subject to measurement error (or at least, that this is the only error that we shall consider). But this ignores the possibility - and consequences - of error in the $x$ measurements. This might be particularly acute in observational studies where the $x$ variables are not under experimental control.

Regression dilution or regression attenuation is the phenomenon recognised by Spearman (1904) whereby the estimated regression slope in simple linear regression is biased towards zero by the presence of measurement error in the independent variable. Suppose the true slope is positive — the effect of jittering the points' $x$ co-ordinates (perhaps most easily visualised as "smudging" the points horizontally) is to render the regression line less steep. Intuitively, points with a large $x$ are now more likely to be so because of positive measurement error, while the $y$ value is more likely to reflect the true (error-free) value of $x$, and hence be lower than the true line would be for the observed $x$.

In more complex models, measurement error in $x$ variables can produce more complicated effects on the parameter estimates. There are errors in variables models that take such error into account. Spearman suggested a correction factor for disattenuating bivariate correlation coefficients and other correction factors have been developed for more sophisticated situations. However, such corrections can be difficult — particularly in the multivariate case and in the presence of confounders — and it may be controversial whether the correction is a genuine improvement, see e.g. Smith and Phillips (1996).

So I suppose this is two misconceptions for the price of one — on the one hand it is a mistake to think that the way we write $y = X\beta + \varepsilon$ means "all the error is in the $y$" and ignore the very physically real possibility of measurement errors in the independent variables. On the other hand, it may be inadvisable to blindly apply "corrections" for measurement error in all such situations as a knee-jerk response (though it may well be a good idea to take steps to reduce the measurement error in the first place).

(I should probably also link to some other common error-in-variables models, in increasingly general order: orthogonal regression, Deming regression, and total least squares.)

References

• Smith, G. D., & Phillips, A. N. (1996). "Inflation in epidemiology: 'the proof and measurement of association between two things' revisited". British Medical Journal, 312(7047), 1659–1661.

• Spearman, C. (1904). "The proof and measurement of association between two things." American Journal of Psychology 15: 72–101.

Answer 3

There are some standard misunderstandings that apply in this context as well as other statistical contexts: e.g., the meaning of $p$-values, incorrectly inferring causality, etc.

A couple of misunderstandings that I think are specific to multiple regression are:

1. Thinking that the variable with the larger estimated coefficient and/or lower $p$-value is 'more important'.
2. Thinking that adding more variables to the model gets you 'closer to the truth'. For example, the slope from a simple regression of $Y$ on $X$ may not be the true direct relationship between $X$ and $Y$, but if I add variables $Z_1, \ldots, Z_5$, that coefficient will be a better representation of the true relationship, and if I add $Z_6, \ldots, Z_{20}$, it will be even better than that.

Answer 4

I'd say the first one you list is probably the most common -- and perhaps the most widely taught that way -- of the things that are plainly seen to be wrong, but here are some others that are less clear in some situations (whether they really apply) but may impact even more analyses, and perhaps more seriously. These are often simply never mentioned when the subject of regression is introduced.

• Treating as random samples from the population of interest sets of observations that cannot possibly be close to representative (let alone randomly sampled). [Some studies could instead be seen as something nearer to convenience samples]

• With observational data, simply ignoring the consequences of leaving out important drivers of the process that would certainly bias the estimates of the coefficients of the included variables (in many cases, even to likely changing their sign), with no attempt to consider ways of dealing with them (whether out of ignorance of the problem or simply being unaware that anything can be done). [Some research areas have this problem more than others, whether because of the kinds of data that are collected or because people in some application areas are more likely to have been taught about the issue.]

• Spurious regression (mostly with data collected over time). [Even when people are aware it happens, there's another common misconception that simply differencing to supposed stationary is sufficient to completely avoid the problem.]

There are many others one could mention of course (treating as independent data that will almost certainly be serially correlated or even integrated may be about as common, for example).

You may notice that observational studies of data collected over time may be hit by all of these at once... yet that kind of study is very common in many areas of research where regression is a standard tool. How they can get to publication without a single reviewer or editor knowing about at least one of them and at least requiring some level of disclaimer in the conclusions continues to worry me.

Statistics is fraught with problems of irreproducible results when dealing with fairly carefully controlled experiments (when combined with perhaps not so carefully controlled analyses), so as soon as one steps outside those bounds, how much worse must the reproducibility situation be?

Answer 5

I probably wouldn't call these misconceptions, but maybe common points of confusion/hang-ups and, in some cases, issues that researchers may not be aware of.

• Multicollinearity (including the case of more variables than data points)
• Heteroskedasticity
• Whether values of the independent variables are subject to noise
• How scaling (or not scaling) affects interpretation of the coefficients
• How to treat data from multiple subjects
• How to deal with serial correlations (e.g. time series)

On the misconception side of things:

• What linearity means (e.g. $y = ax^2 + bx + c$ is nonlinear w.r.t. $x$, but linear w.r.t. the weights).
• That 'regression' means ordinary least squares or linear regression
• That low/high weights necessarily imply weak/strong relationships with the dependent variable
• That dependence between the dependent and independent variables can necessarily be reduced to pairwise dependencies.
• That high goodness-of fit on the training set implies a good model (i.e. neglecting overfitting)

Answer 6

In my experience, students frequently adopt the view the that squared errors (or OLS regression) are an inherently appropriate, accurate, and overall good thing to use, or are even without alternative. I have frequently seen OLS advertised along with remarks that it "gives greater weight to more extreme/deviant observations", and most of the time it is at least implied that this is a desirable property. This notion may be modified later, when the treatment of outliers and robust approaches are introduced, but at that point the damage is done. Arguably, the widespread use of squared errors has historically more to do with their mathematical convenience than with some natural law of real-world error costs.

Overall, greater emphasis could be placed on the understanding that the choice of error function is somewhat arbitrary. Ideally, any choice of penalty within an algorithm should be guided by the corresponding real-world cost function associated with potential error (i.e., using a decision-making framework). Why not establish this principle first, and then see how well we can do?

Answer 7

Another common misconception is that the error term (or disturbance in econometrics parlance) and the residuals are the same thing.

The error term is a random variable in the true model or data generating process, and is often assumed to follow a certain distribution, whereas the residuals are the deviations of the observed data from the fitted model. As such, the residuals can be considered to be estimates of the errors.

Answer 8

The most common misconception I encounter is that linear regression assumes normality of errors. It doesn't. Normality is useful in connection with some aspects of linear regression e.g. small sample properties such as confidence limits of coefficients. Even for these things there are asymptotic values available for non-normal distributions.

The second most common is a cluster of confusion with regards to endogeneity, e.g. not being careful with feedback loops. If there's a feedback loop from Y back to X it's an issue.

Answer 9

The one I've often seen is a misconception on applicability of linear regression in certain use cases, in practice.

For example, let us say that the variable that we are interested in is count of something (example: visitors on website) or ratio of something (example: conversion rates). In such cases, the variable can be better modeled by using link functions like Poisson (counts), Beta (ratios) etc. So using generalized model with more appropriate link function is more suitable. But just because the variable is not categorical, I've seen people starting with simple linear regression (link function = identity). Even if we disregard the accuracy implications, the modeling assumptions are a problem here.

Answer 10

An error that I made is to assume a symmetry of X and Y in the OLS. For instance, if I assume a linear relation $$ Y = a \, X + b$$ with a and b given by my software using OLS, then I believe that assuming X as a function of Y will give using OLS the coefficients: $$ X = \frac{1}{a} \, Y - \frac{b}{a}$$ that is wrong.

Maybe this is also related to the difference between OLS and total least square or first principal component.

Answer 11

Here is one I think is frequently overlooked by researchers:

• Variable interaction: researchers often look at isolated betas of individual predictors, and often don't even specify interaction terms. But in real world things interact. Without proper specification of all possible interaction terms, you don't know how your "predictors" engage together into forming an outcome. And if you want to be diligent and specify all interactions, the number of predictors will explode. From my calculations you can investigate only 4 variables and their interactions with 100 subjects. If you add one more variable you can overfit very easily.

Answer 12

Another common misconception is that the estimates (fitted values) are not invariant to transformations, e.g.

$$f(\hat{y}_i) \neq \widehat{f(y_i)}$$ in general, where $\hat{y}_i = \vec{x}_i ^T \hat{\beta}$, the fitted regression value based on your estimated regression coefficients.

If this is what you want for monotonic functions $f(\cdot)$ not necessarily linear, then what you want is a quantile regression.

The equality above holds in linear regression for linear functions but non-linear functions (e.g. $log(\cdot)$) this will not hold. However, this will hold for any monotonic function in quantile regression.

This comes up all the time when you do a log transform of your data, fit a linear regression, then exponentiate the fitted value and people read that as the regression. This isn't the mean, this is the median (if things are truly log-normally distributed).