3
$\begingroup$

As far as I am concerned, "multicollinearity" referers to the presence of collinearity between two or more variables, even if there is no pair of variables that have a particularly high correlation.

In simple linear regression, we only have one independent variable, but can multicollinearity affect the model in some way?

$\endgroup$

3 Answers 3

9
$\begingroup$

If there is only a single IV, multicollinearity is not a concern/issue and a simple linear regression model with a single IV is analogous to a correlation.

$\endgroup$
2
  • 2
    $\begingroup$ This is for those who might struggle with undefined acronyms: I suppose IV stands for independent variable rather than, say, instrumental variable (which would be my first reaction when seeing the acronym). $\endgroup$ Commented Nov 20, 2023 at 8:39
  • 2
    $\begingroup$ What if the patient has only one IV but there are several different drugs administered through that IV? $\endgroup$
    – Stef
    Commented Nov 20, 2023 at 12:40
7
$\begingroup$

IMO, it does make sense, to some extent. Although it depends on what you mean by "multicollinearity" and "simple linear regression". Many people have distinct definitions for the above terms, and I might not be on the same page with you or others.

Case 1: Simple Linear Regression is a model with one explanatory variable and an intercept.

In this case you actually have two explanatory variables, it's just that one of them is constant (the intercept). If your other explanatory variable is also constant - you'll have the same problems as multivariable linear model having multicollinear variables. In particular - your coefficients could not be interpretable or make any sense.

Case 2: Simple Linear Regression is a model with one explanatory variable and no intercept.

In this case multicollinearity means your training data or data matrix is a vector of zeroes. It might not fit with some definitions for "multicollinearity", but eventually you're gonna have the same problems - in a nutshell, your coefficient(s) will be uninterpretable.

$\endgroup$
5
  • $\begingroup$ It's OK, I'm here to learn. If I'd reworded my comment, saying that a simple linear model can't suffer from multicollinearity per-se, but it can have same problems multicollinearity causes, would that be alright with you? Why's the case of an explanatory variable having a constant value nonsense? It does happen. $\endgroup$
    – Alex Teush
    Commented Nov 19, 2023 at 13:39
  • 2
    $\begingroup$ +1. Contra @PeterFlom, I would indeed consider the intercept an explanatory variable: it encodes, or "explains", the overall mean. If we have 1000 observations of something, 999 of them belonging to group A and 1 to group B, we would fit a regression where the group membership variable has 999 zeros and a single 1. The corresponding regression parameter estimate is highly variable. Which I would indeed see as a case of "collinearity with the intercept". $\endgroup$ Commented Nov 19, 2023 at 14:37
  • 1
    $\begingroup$ (As an aside, one issue why we at my place of work use the Belsley et al. approach to collinearity is that the VIF inherently can't deal with this "collinearity with the intercept".) $\endgroup$ Commented Nov 19, 2023 at 14:38
  • 1
    $\begingroup$ @StephanKolassa Good points. And I also prefer the Belsley approach (I wrote my dissertation on that). $\endgroup$
    – Peter Flom
    Commented Nov 19, 2023 at 14:46
  • $\begingroup$ On your "it depends on what you mean by ... simple linear regression", I am not sure I personally would regard "a model with one explanatory variable and no intercept" as simple linear regression unless the data had been pre-processed to have its mean at the origin. $\endgroup$
    – Henry
    Commented Nov 21, 2023 at 0:34
5
$\begingroup$

Collinearity is a relation among two or more independent variables. In simple linear regression you have one independent variable. 1 < 2.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.