0
$\begingroup$

1)I saw many posts online and on cross-validated itself, that the predictive power of a model is not influenced by multicollinearity. I would like to know what is meant by this statement, does it mean that the model, even if I omitted one out of the 2 correlated variables would still have the same predictive power as before?

2)What is meant by

coefficient estimates of the multiple regression may change erratically in response to small changes in the model or the data.

Does this mean that if I only introduce one more observation into the training dataset, then the coefficient estimates would change drastically? If this is the case, what is the intuitive reason behind it?  

Improve this question
$\endgroup$
6
  • 5
    $\begingroup$ Does this answer your question? Why doesn't collinearity affect the predictions? $\endgroup$
    – Sextus Empiricus
    Commented Apr 30, 2020 at 13:04
  • $\begingroup$ See related stats.stackexchange.com/a/70910/3277. $\endgroup$
    – ttnphns
    Commented Apr 30, 2020 at 13:21
  • $\begingroup$ In linear regression, "predictive power" of a model can be understood as its R-square. R-square is (see the above link) the angle between the subspace of predictors and the predictand vector (and the error is the nadir from the vector's head onto the subspace). Multicollinearity mean that the p predictors span not p dim. subspace but a lesser dim. subspace - for example, 3 vectors lie on a plane or 2 vectors forming a line. Which means that (at least) one of the predictors, any, is unnecessary in defining the subspace the predictors define. $\endgroup$
    – ttnphns
    Commented Apr 30, 2020 at 13:45
  • $\begingroup$ (cont.) You can remove some redundant predictor(s), while the remaining ones will still support that same subspace from which the R-square is gauged off. So, removing the predictor only removes multicollinearity condition and changes not the R-square: because the former subspace (with the unchanged dimensionality) persists in this case. $\endgroup$
    – ttnphns
    Commented Apr 30, 2020 at 13:46
  • $\begingroup$ You can have p predictors with collinearity and they still spam p dim. As long as one vector is close to a linear combination of the others there is collinearity. In that case of not being exactly linear dependent they still span p dim. $\endgroup$
    – javierazcoiti
    Commented Apr 30, 2020 at 13:50

4 Answers 4

Reset to default
3
$\begingroup$

A comment relating to your concerns, to quote:

Moderate multicollinearity may not be problematic. However, severe multicollinearity is a problem because it can increase the variance of the coefficient estimates and make the estimates very sensitive to minor changes in the model. The result is that the coefficient estimates are unstable and difficult to interpret. Multicollinearity saps the statistical power of the analysis, can cause the coefficients to switch signs, and makes it more difficult to specify the correct model...

And also:

In short, multicollinearity:

  • can make choosing the correct predictors to include more difficult.

  • interferes in determining the precise effect of each predictor, but...

  • doesn’t affect the overall fit of the model or produce bad predictions.

An interesting effect of the variance inflation factors (VIF), to quote:

In this model, the VIFs are high because of the interaction term. Interaction terms and higher-order terms (e.g., squared and cubed predictors) are correlated with main effect terms because they include the main effects terms. To reduce high VIFs produced by interaction and higher-order terms, you can standardize the continuous predictor variables...

...we’ll choose the Subtract the mean method, which is also known as centering the variables. This method removes the multicollinearity produced by interaction and higher-order terms as effectively as the other standardization methods, but it has the added benefit of not changing the interpretation of the coefficients.

The author then gives a demonstration of the effect upon re-running a model after standardization:

Compare the Summary of Model statistics between the two models and you’ll notice that S, R-squared, adjusted R-squared, and the others are all identical. Multicollinearity doesn’t affect how well the model fits. In fact, if you want to use the model to make predictions, both models produce identical results for fitted values and prediction intervals!

So, multicollinearity does not apparently impact predictions, however, do note if the error terms are autocorrelated, for example, one can produce a next period forecast that benefits from applying a weighting of prior known residual error terms.

Improve this answer
$\endgroup$
1
$\begingroup$

Applied to linear regression case.

Question 1

Multicollinearity happens when your predictors are linearly dependent (or close to be). This means, some of your N predictors can be obtained (or nearly) by linear combinations of the others. If predictor A is linearly dependent, you can remove it, and the ability to fit of your system remains the same. If it is not exactly linearly dependent, your ability to fit data will be lower generally.

Related to "the predictive power of a model is not influenced by multicollinearity". You can fit your response variable Y and have the same error by using new orthogonal predictors as long as your collinear predictors define the same vector space as the new orthogonal predictors.

Question 2

With an example. Imagine we are in 2 dimensional space.

We have two points: Y1 is (40,50), y2 is (39,50).

We want to approximate Y1 and Y2 (with error 0) using two bases.

Base E: e1, e2 are (1,0) and (0,1)

Base A: a1, a2 are (1,-1) and (1,-0.99)

e1 and e2 are orthogonal and are base vectors of the plane. a1 and a2 are almost linear dependent, they have a strong collinearity, but they are also base vectors of the plane because they are not the same vector.

We observe the results of the coefficient estimates with Y1 and Y2 (small changes in data)

Y1 is fitted as 40*e1 + 50*e2 and Y2 is predicted as 39*e1 + 50*e2

Y1 is fitted as -8960*a1 + 9000*a2 and Y2 is predicted as -8861*a1 + 8900*a2

You can observe the size and variance of the coefficients when using predictors with strong collinearity. By the way, other choices of a1 and a2 may show more variance in this example.

Improve this answer
$\endgroup$
4
  • $\begingroup$ regarding your answer to q1, you mean that if the correlated variables have a r=1, then the predictive power of the model, won't go down(adjusted r-square), but if the r is not equal to 1, then the predictive power(adjusted r-squared) will go down as compared to a model with one of the 2 correlated variables omitted? $\endgroup$
    – divyam sureka
    Commented Apr 30, 2020 at 9:19
  • $\begingroup$ regarding your answer to q2, I'm sorry but I couldn't understand your explanation with the matrix, vectors and orthogonal stuff, as I'm a beginner in stats and regressions. Could you if possible please provide me with an answer explaining only the intuitive reason of q2? Thanks a lot! $\endgroup$
    – divyam sureka
    Commented Apr 30, 2020 at 9:25
  • $\begingroup$ Hi. In the example to question 2, I am showing what happens to coefficients when we have a little change in Y (from Y1 to Y2). When using orthogonal predictors like e1 and e2, coefficients change slightly. When we use predictors with strong collinearity like a1 and a2, the small change from Y1 to Y2 causes big changes in the coefficients. In both cases, the error is zero, because in both cases we can create Y1 and Y2 by linear combinations of e1 and e2 or a1 and a2. I am not talking here about the errors, I am talking about the variance of the coefficients. I hope it helps. $\endgroup$
    – javierazcoiti
    Commented Apr 30, 2020 at 9:52
  • $\begingroup$ About question 1. If you have a predictor that is a linear combination of the others, you can remove it. You remove it, r squared doesn't change. If it's not a perfect linear combination and you remove it, r -squared will decrease in most of the cases and r-adjusted should improve. $\endgroup$
    – javierazcoiti
    Commented Apr 30, 2020 at 14:16
0
$\begingroup$

Multicollinearity makes the (observation)×(feature) matrix singular or near-singular.

This is why it reduces the predictive power of the model.

For more clear explanation follow the given link:

https://stats.stackexchange.com/a/70910/283279

Improve this answer
$\endgroup$
4
  • $\begingroup$ The question is about the statement that multicollinearity does not influence the predictive power of the model, not why it does. $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 30, 2020 at 6:11
  • $\begingroup$ Question is why multicollinearity does not influence the predictive power of the model, but as per me it's not always true. If you add correlated variables then it will increase the variance. The reason for same is mentioned in the given link by me. $\endgroup$
    – VIJAY
    Commented Apr 30, 2020 at 6:27
  • $\begingroup$ Fair enough, but this does not answer the question. You could say, "I don't know why someone would say that, I think the opposite is true..." or something. $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 30, 2020 at 6:38
  • $\begingroup$ Thanks for pointing it out. I'll update my answer accordingly. $\endgroup$
    – VIJAY
    Commented May 1, 2020 at 13:16
0
$\begingroup$

You have asked two questions (as an aside, it is better, on this site, to ask one question at a time).

The first question concerned the predictive power of a regression and you then asked if removing one variable would not affect the predictive power. I think you may have misread some statements. It is commonly said that collinearity does not affect the predictions of the model. That is, the predicted values of the dependent variable are valid, even if there is a lot of collinearity among the independent variables.

But perhaps you mean something else. If so, please clarify what you mean by this question, or cite an example of someone saying it.

Your second question has a simple answer: Yes. In fact, with severe collinearity you can make tiny changes in the existing data and get completely different results - e.g. the signs of parameter estimates can change and be significant in both directions. David Belsley (one of the real collinearity mavens) gives an example where changing the 4th significant digit reverses all the signs of the parameter estimates.

Intuitively, and using the case of collinearity between two variables (rather than any set of variables) this is because the two variables are very nearly the same. This means that $$ b_1 x_1 - b_2x_2$$ is going to be very close to $$ -b_1x_1 + b_2x_2$$

If you want the math, I recommend looking at either of Belsley's two books (if you can get them from a library) or any of the many papers on colinearity.

Improve this answer
$\endgroup$
6
  • $\begingroup$ The final part about flipping the signs is not so intuitive to me. This only makes sense when the value is close to zero. Maybe you meant $$b_1 x_1 + b_2 x_2 \approx b_2 x_1 + b_1 x_2$$ and when $b_1$ and $b_2$ have a different sign then switching them means that the signs of the coefficients have changed while the value is still much the same. $\endgroup$
    – Sextus Empiricus
    Commented May 1, 2020 at 16:27
  • $\begingroup$ @PeterFlom, you said that any tiny changes in the existing data will produce completely different results, so suppose i am estimating price of a house with dependent variables size, high school in the neighborhood, So in my training data if my size changes from 1000 to 1001, it would produce completely different results? $\endgroup$
    – divyam sureka
    Commented May 1, 2020 at 20:58
  • $\begingroup$ @divyamsureka Not necessarily, but it might. $\endgroup$
    – Peter Flom
    Commented May 2, 2020 at 12:08
  • $\begingroup$ @SextusEmpiricus I'm not sure how what you wrote differs from what I wrote. $\endgroup$
    – Peter Flom
    Commented May 2, 2020 at 12:10
  • $\begingroup$ @PeterFlom the signs change in the sense that negative values become positive values and vice versa, but you are not getting the strong sense of sign changes, where the coefficients are 'flipped' and multiplied with '-1'. $\endgroup$
    – Sextus Empiricus
    Commented May 2, 2020 at 12:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.