3
$\begingroup$

As far as I am concerned, "multicollinearity" referers to the presence of collinearity between two or more variables, even if there is no pair of variables that have a particularly high correlation.

In simple linear regression, we only have one independent variable, but can multicollinearity affect the model in some way?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
9
$\begingroup$

If there is only a single IV, multicollinearity is not a concern/issue and a simple linear regression model with a single IV is analogous to a correlation.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ This is for those who might struggle with undefined acronyms: I suppose IV stands for independent variable rather than, say, instrumental variable (which would be my first reaction when seeing the acronym). $\endgroup$
    – Richard Hardy
    Nov 20 at 8:39
  • 2
    $\begingroup$ What if the patient has only one IV but there are several different drugs administered through that IV? $\endgroup$
    – Stef
    Nov 20 at 12:40
7
$\begingroup$

IMO, it does make sense, to some extent. Although it depends on what you mean by "multicollinearity" and "simple linear regression". Many people have distinct definitions for the above terms, and I might not be on the same page with you or others.

Case 1: Simple Linear Regression is a model with one explanatory variable and an intercept.

In this case you actually have two explanatory variables, it's just that one of them is constant (the intercept). If your other explanatory variable is also constant - you'll have the same problems as multivariable linear model having multicollinear variables. In particular - your coefficients could not be interpretable or make any sense.

Case 2: Simple Linear Regression is a model with one explanatory variable and no intercept.

In this case multicollinearity means your training data or data matrix is a vector of zeroes. It might not fit with some definitions for "multicollinearity", but eventually you're gonna have the same problems - in a nutshell, your coefficient(s) will be uninterpretable.

Improve this answer
$\endgroup$
5
  • $\begingroup$ It's OK, I'm here to learn. If I'd reworded my comment, saying that a simple linear model can't suffer from multicollinearity per-se, but it can have same problems multicollinearity causes, would that be alright with you? Why's the case of an explanatory variable having a constant value nonsense? It does happen. $\endgroup$
    – Alex Teush
    Nov 19 at 13:39
  • 2
    $\begingroup$ +1. Contra @PeterFlom, I would indeed consider the intercept an explanatory variable: it encodes, or "explains", the overall mean. If we have 1000 observations of something, 999 of them belonging to group A and 1 to group B, we would fit a regression where the group membership variable has 999 zeros and a single 1. The corresponding regression parameter estimate is highly variable. Which I would indeed see as a case of "collinearity with the intercept". $\endgroup$
    – Stephan Kolassa
    Nov 19 at 14:37
  • 1
    $\begingroup$ (As an aside, one issue why we at my place of work use the Belsley et al. approach to collinearity is that the VIF inherently can't deal with this "collinearity with the intercept".) $\endgroup$
    – Stephan Kolassa
    Nov 19 at 14:38
  • 1
    $\begingroup$ @StephanKolassa Good points. And I also prefer the Belsley approach (I wrote my dissertation on that). $\endgroup$
    – Peter Flom
    Nov 19 at 14:46
  • $\begingroup$ On your "it depends on what you mean by ... simple linear regression", I am not sure I personally would regard "a model with one explanatory variable and no intercept" as simple linear regression unless the data had been pre-processed to have its mean at the origin. $\endgroup$
    – Henry
    Nov 21 at 0:34
5
$\begingroup$

Collinearity is a relation among two or more independent variables. In simple linear regression you have one independent variable. 1 < 2.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.