69
$\begingroup$

I'm curious, for those of you who have extensive experience collaborating with other researchers, what are some of the most common misconceptions about linear regression that you encounter?

I think can be a useful exercise to think about common misconceptions ahead of time in order to

  1. Anticipate people's mistakes and be able to successful articulate why some misconception is incorrect

  2. Realize if I am harboring some misconceptions myself!

A couple of basic ones I can think of:

Independent/Dependent variables must be normally distributed

Variables must be standardized for accurate interpretation

Any others?

All responses are welcome.

$\endgroup$
  • 5
    $\begingroup$ This should probably be make CW, since it invites a list of possibilities & it will be difficult to say that one is objectively the 'correct answer'. $\endgroup$ – gung Jun 9 '16 at 19:54
  • $\begingroup$ A lot of people I know still insist on performing linearizations on their data and leaving it at that, even when the computing environment they use has good support for nonlinear regression. (The linearizations are of course useful as starting points for the nonlinear fits, but these people don't even realize that.) $\endgroup$ – J. M. is not a statistician Jun 9 '16 at 21:35
  • 1
    $\begingroup$ @gung: Is Community Wiki still a thing? Mostly deprecated network-wide, CW was never really about providing marginal, broadly-asked big list questions a get out of jail free card, or robbing people of reputation they might have otherwise earned had the question been on-topic in the first place. The only way you can even put it on a question anymore is if you ask a moderator to do it. $\endgroup$ – Robert Harvey Jun 13 '16 at 5:05
  • 1
    $\begingroup$ If God had made the world linear, there wouldn't be nonlinear regression. $\endgroup$ – Mark L. Stone Jun 14 '16 at 20:26
  • 1
    $\begingroup$ @RobertHarvey: Yes, it is still very much a thing on CrossValidated (in my opinion, unfortunately). We have had some heated Meta discussions about it (e.g. this one), but the current status quo is that CW status is being enforced on all opinion-based or big-list questions that are considered on topic enough to stay open. $\endgroup$ – amoeba Jun 14 '16 at 21:54

12 Answers 12

38
$\begingroup$

False premise: A $\hat{\beta} \approx 0$ means that there is no strong relationship between DV and IV.
Non-linear functional relationships abound, and yet data produced by many such relationships would often produce nearly zero slopes if one assumes the relationship must be linear, or even approximately linear.

Relatedly, in another false premise researchers often assume—possibly because many introductory regression textbooks teach—that one "tests for non-linearity" by building a series of regressions of the DV onto polynomial expansions of the IV (e.g. $Y \sim \beta_{0} + \beta_{X}X + \varepsilon$, followed by $Y \sim \beta_{0} + \beta_{X}X + \beta_{X^{2}}X^{2} + \varepsilon$, followed by $Y \sim \beta_{0} + \beta_{X}X + \beta_{X^{2}}X^{2} + \beta_{X^{3}}X^{3} + \varepsilon$, etc.). Just as straight line cannot well represent a non-linear functional relationship between DV and IV, a parabola cannot well represent literally an infinite number of nonlinear relationships (e.g. sinusoids, cycloids, step functions, saturation effects, s-curves, etc. ad infinitum). One may instead take a regression approach that does not assume any particular functional form (e.g. running line smoothers, GAMs, etc.).

A third false premise is that increasing the number of estimated parameters necessarily results in a loss of statistical power. This may be false when the true relationship is non-linear and requires multiple parameters to estimate (e.g. a "broken stick" function requires not only the intercept and slope terms of a straight line, but requires point at which slope changes and a how much slope changes by estimates also): the residuals of a misspecified model (e.g. a straight line) may grow quite large (relative to a properly specified functional relation) resulting in a lower rejection probability and wider confidence intervals and prediction intervals (in addition to estimates being biased).

$\endgroup$
  • 4
    $\begingroup$ (+1) Quibbles: (1) I don't think even introductory texts imply that all curves are polynomial functions, rather that they can be approximated well enough over a given range by polynomial functions. So they fall into the class of "regression approaches that do not assume any particular functional form", governed by a "hyperparameter" specifying wiggliness: the span for loess, the no. knots for regression on a spline basis, the degree for regression on a polynomial basis. (I'm not waving a flag for polynomials - it's well known they tend to flail around at the ends more than we'd like -, ... $\endgroup$ – Scortchi Jun 10 '16 at 9:49
  • 2
    $\begingroup$ ... just giving them their due.) (2) A sinusoid might well be fit as such, within the linear model framework; a saturation effect using a non-linear model (a rectangular hyperbola, say); &c. Of course you didn't say otherwise, but it's perhaps worth pointing out that if you know there's a cycle, or an asymptote, applying those constraints in your model will be helpful. $\endgroup$ – Scortchi Jun 10 '16 at 9:49
  • 2
    $\begingroup$ @Scortchi I could not agree more! (Indeed, given an infinite number of polynomials, any function can be perfectly represented.) Was aiming at concise. :) $\endgroup$ – Alexis Jun 10 '16 at 17:13
  • 2
    $\begingroup$ @Alexis Try approximating Conway's base 13 function by polynomials. :) $\endgroup$ – Solomonoff's Secret Jun 12 '16 at 2:05
  • 1
    $\begingroup$ Or $\chi_{\mathbb{Q}}$... $\endgroup$ – Stephan Kolassa Jun 12 '16 at 13:20
21
$\begingroup$

There are some standard misunderstandings that apply in this context as well as other statistical contexts: e.g., the meaning of $p$-values, incorrectly inferring causality, etc.

A couple of misunderstandings that I think are specific to multiple regression are:

  1. Thinking that the variable with the larger estimated coefficient and/or lower $p$-value is 'more important'.
  2. Thinking that adding more variables to the model gets you 'closer to the truth'. For example, the slope from a simple regression of $Y$ on $X$ may not be the true direct relationship between $X$ and $Y$, but if I add variables $Z_1, \ldots, Z_5$, that coefficient will be a better representation of the true relationship, and if I add $Z_6, \ldots, Z_{20}$, it will be even better than that.
$\endgroup$
  • 12
    $\begingroup$ Good stuff. This answer might be even more useful if it explained why the two are wrong and what one should do instead? $\endgroup$ – D.W. Jun 10 '16 at 1:11
21
+100
$\begingroup$

It's very common to assume that only $y$ data are subject to measurement error (or at least, that this is the only error that we shall consider). But this ignores the possibility - and consequences - of error in the $x$ measurements. This might be particularly acute in observational studies where the $x$ variables are not under experimental control.

Regression dilution or regression attenuation is the phenomenon recognised by Spearman (1904) whereby the estimated regression slope in simple linear regression is biased towards zero by the presence of measurement error in the independent variable. Suppose the true slope is positive — the effect of jittering the points' $x$ co-ordinates (perhaps most easily visualised as "smudging" the points horizontally) is to render the regression line less steep. Intuitively, points with a large $x$ are now more likely to be so because of positive measurement error, while the $y$ value is more likely to reflect the true (error-free) value of $x$, and hence be lower than the true line would be for the observed $x$.

In more complex models, measurement error in $x$ variables can produce more complicated effects on the parameter estimates. There are errors in variables models that take such error into account. Spearman suggested a correction factor for disattenuating bivariate correlation coefficients and other correction factors have been developed for more sophisticated situations. However, such corrections can be difficult — particularly in the multivariate case and in the presence of confounders — and it may be controversial whether the correction is a genuine improvement, see e.g. Smith and Phillips (1996).

So I suppose this is two misconceptions for the price of one — on the one hand it is a mistake to think that the way we write $y = X\beta + \varepsilon$ means "all the error is in the $y$" and ignore the very physically real possibility of measurement errors in the independent variables. On the other hand, it may be inadvisable to blindly apply "corrections" for measurement error in all such situations as a knee-jerk response (though it may well be a good idea to take steps to reduce the measurement error in the first place).

(I should probably also link to some other common error-in-variables models, in increasingly general order: orthogonal regression, Deming regression, and total least squares.)

References

$\endgroup$
  • $\begingroup$ On that note: this is one reason for the use of the technique that is called either "total least squares" or "orthogonal regression" (depending on the reference you are reading); it is significantly more complicated than plain least squares, but is worth doing if all your points are contaminated with error. $\endgroup$ – J. M. is not a statistician Jun 10 '16 at 3:45
  • $\begingroup$ @J.M. Thanks - yes, in fact I'd originally meant to put in a link to TLS, but got distracted by the Smith and Phillips article! $\endgroup$ – Silverfish Jun 10 '16 at 12:30
  • 2
    $\begingroup$ +1 Great addition to this topic. I've often considered EIV models in my work. However, apart from their complexity or reliance on knowledge of "error ratios", there is a more conceptual issue to consider: Many regressions, especially in supervised learning or prediction, want to relate observed predictors to observed outcomes. EIV models, on the other hand, attempt to identify the underlying relationship between the mean predictor and mean response...a slightly different question. $\endgroup$ – user75138 Jun 12 '16 at 13:07
  • 2
    $\begingroup$ So, what one would call "dilution" of the "true" regression (in a scientific context) would be called "absence of predictive utility" or something like that in a prediction context. $\endgroup$ – user75138 Jun 12 '16 at 13:11
14
$\begingroup$

I'd say the first one you list is probably the most common -- and perhaps the most widely taught that way -- of the things that are plainly seen to be wrong, but here are some others that are less clear in some situations (whether they really apply) but may impact even more analyses, and perhaps more seriously. These are often simply never mentioned when the subject of regression is introduced.

  • Treating as random samples from the population of interest sets of observations that cannot possibly be close to representative (let alone randomly sampled). [Some studies could instead be seen as something nearer to convenience samples]

  • With observational data, simply ignoring the consequences of leaving out important drivers of the process that would certainly bias the estimates of the coefficients of the included variables (in many cases, even to likely changing their sign), with no attempt to consider ways of dealing with them (whether out of ignorance of the problem or simply being unaware that anything can be done). [Some research areas have this problem more than others, whether because of the kinds of data that are collected or because people in some application areas are more likely to have been taught about the issue.]

  • Spurious regression (mostly with data collected over time). [Even when people are aware it happens, there's another common misconception that simply differencing to supposed stationary is sufficient to completely avoid the problem.]

There are many others one could mention of course (treating as independent data that will almost certainly be serially correlated or even integrated may be about as common, for example).

You may notice that observational studies of data collected over time may be hit by all of these at once... yet that kind of study is very common in many areas of research where regression is a standard tool. How they can get to publication without a single reviewer or editor knowing about at least one of them and at least requiring some level of disclaimer in the conclusions continues to worry me.

Statistics is fraught with problems of irreproducible results when dealing with fairly carefully controlled experiments (when combined with perhaps not so carefully controlled analyses), so as soon as one steps outside those bounds, how much worse must the reproducibility situation be?

$\endgroup$
  • 6
    $\begingroup$ Closely related to some of your points might be the idea that "only $y$ data are subject to measurement error" (or at least, "this is the only error that we shall consider"). Not sure if that deserves shoe-horning in here, but it is certainly very common to ignore the possibility - and consequences - of random error in the $x$ variables. $\endgroup$ – Silverfish Jun 10 '16 at 0:55
  • 2
    $\begingroup$ @Silverfish I totally agree with you. $\endgroup$ – Mark L. Stone Jun 10 '16 at 1:08
  • $\begingroup$ @Silverfish it's CW so you should feel extra-free to edit in a suitable addition like that. $\endgroup$ – Glen_b Jun 10 '16 at 1:28
  • $\begingroup$ @Silverfish there's a reason I didn't already add it myself when you mentioned it... I think it probably is worth a new answer $\endgroup$ – Glen_b Jun 10 '16 at 1:49
12
$\begingroup$

I probably wouldn't call these misconceptions, but maybe common points of confusion/hang-ups and, in some cases, issues that researchers may not be aware of.

  • Multicollinearity (including the case of more variables than data points)
  • Heteroskedasticity
  • Whether values of the independent variables are subject to noise
  • How scaling (or not scaling) affects interpretation of the coefficients
  • How to treat data from multiple subjects
  • How to deal with serial correlations (e.g. time series)

On the misconception side of things:

  • What linearity means (e.g. $y = ax^2 + bx + c$ is nonlinear w.r.t. $x$, but linear w.r.t. the weights).
  • That 'regression' means ordinary least squares or linear regression
  • That low/high weights necessarily imply weak/strong relationships with the dependent variable
  • That dependence between the dependent and independent variables can necessarily be reduced to pairwise dependencies.
  • That high goodness-of fit on the training set implies a good model (i.e. neglecting overfitting)
$\endgroup$
7
$\begingroup$

In my experience, students frequently adopt the view the that squared errors (or OLS regression) are an inherently appropriate, accurate, and overall good thing to use, or are even without alternative. I have frequently seen OLS advertised along with remarks that it "gives greater weight to more extreme/deviant observations", and most of the time it is at least implied that this is a desirable property. This notion may be modified later, when the treatment of outliers and robust approaches are introduced, but at that point the damage is done. Arguably, the widespread use of squared errors has historically more to do with their mathematical convenience than with some natural law of real-world error costs.

Overall, greater emphasis could be placed on the understanding that the choice of error function is somewhat arbitrary. Ideally, any choice of penalty within an algorithm should be guided by the corresponding real-world cost function associated with potential error (i.e., using a decision-making framework). Why not establish this principle first, and then see how well we can do?

$\endgroup$
  • 2
    $\begingroup$ The choice is also application-dependent. OLS is useful for algebraic, y-axis fits but less so for geometric applications, where total least squares (or some other cost function based on orthogonal distance) makes more sense. $\endgroup$ – Willie Wheeler Jun 10 '16 at 23:25
4
$\begingroup$

Another common misconception is that the error term (or disturbance in econometrics parlance) and the residuals are the same thing.

The error term is a random variable in the true model or data generating process, and is often assumed to follow a certain distribution, whereas the residuals are the deviations of the observed data from the fitted model. As such, the residuals can be considered to be estimates of the errors.

$\endgroup$
  • $\begingroup$ I bet people would be interested in explanation as to why this matters, or in what sorts of cases. $\endgroup$ – rolando2 Mar 27 '18 at 21:41
4
$\begingroup$

The most common misconception I encounter is that linear regression assumes normality of errors. It doesn't. Normality is useful in connection with some aspects of linear regression e.g. small sample properties such as confidence limits of coefficients. Even for these things there are asymptotic values available for non-normal distributions.

The second most common is a cluster of confusion with regards to endogeneity, e.g. not being careful with feedback loops. If there's a feedback loop from Y back to X it's an issue.

$\endgroup$
4
$\begingroup$

An error that I made is to assume a symmetry of X and Y in the OLS. For instance, if I assume a linear relation $$ Y = a \, X + b$$ with a and b given by my software using OLS, then I believe that assuming X as a function of Y will give using OLS the coefficients: $$ X = \frac{1}{a} \, Y - \frac{b}{a}$$ that is wrong.

Maybe this is also related to the difference between OLS and total least square or first principal component.

$\endgroup$
3
$\begingroup$

The one I've often seen is a misconception on applicability of linear regression in certain use cases, in practice.

For example, let us say that the variable that we are interested in is count of something (example: visitors on website) or ratio of something (example: conversion rates). In such cases, the variable can be better modeled by using link functions like Poisson (counts), Beta (ratios) etc. So using generalized model with more appropriate link function is more suitable. But just because the variable is not categorical, I've seen people starting with simple linear regression (link function = identity). Even if we disregard the accuracy implications, the modeling assumptions are a problem here.

$\endgroup$
2
$\begingroup$

Here is one I think is frequently overlooked by researchers:

  • Variable interaction: researchers often look at isolated betas of individual predictors, and often don't even specify interaction terms. But in real world things interact. Without proper specification of all possible interaction terms, you don't know how your "predictors" engage together into forming an outcome. And if you want to be diligent and specify all interactions, the number of predictors will explode. From my calculations you can investigate only 4 variables and their interactions with 100 subjects. If you add one more variable you can overfit very easily.
$\endgroup$
0
$\begingroup$

Another common misconception is that the estimates (fitted values) are not invariant to transformations, e.g.

$$f(\hat{y}_i) \neq \widehat{f(y_i)}$$ in general, where $\hat{y}_i = \vec{x}_i ^T \hat{\beta}$, the fitted regression value based on your estimated regression coefficients.

If this is what you want for monotonic functions $f(\cdot)$ not necessarily linear, then what you want is a quantile regression.

The equality above holds in linear regression for linear functions but non-linear functions (e.g. $log(\cdot)$) this will not hold. However, this will hold for any monotonic function in quantile regression.

This comes up all the time when you do a log transform of your data, fit a linear regression, then exponentiate the fitted value and people read that as the regression. This isn't the mean, this is the median (if things are truly log-normally distributed).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.