I am working on some predictive modeling project these days: trying to learn a model and make real-time predictions based on the model I learned offline.
I started using ridge regression recently, because I read that regularization can help to reduce the effect of multicollinearity.
However, I read this blog today. I am totally confused now. According to this blog, multicollinearity does NOT hurt a model's predictive power that much.
So, in the end, is multicollinearity a problem or not?
It's a problem for causal inference - or rather, it indicates difficulties in causal inference - but it's not a particular problem for prediction/forecasting (unless it's so extreme that it prevents model convergence or results in singular matrices, and then you won't get predictions anyway). This, I think, is the meaning of that blog post, as well. It sounds like you may be insisting on a yes-or-no answer when the answer is that it depends. Here's what it depends on, and why it can at least be said that (non-perfect) multicollinearity is never a reason to drop a variable from a model - any problems that multicollinearity indicate won't go away because you dropped a variable and stopped seeing the collinearity.
Predictors that are highly correlated with each other just don't do as good a job of improving your predictions as they would if they were not collinear, but still separately correlated with the outcome variable; neither one is doing much more work than the other one is already doing and would do on its own anyway. Maybe they're so strongly related to each other because they are capturing basically the same underlying construct, in which case neither one is adding much more on top of the other for good reason, and it would be impossible to separate them out ontologically for predictive purposes anyway, by manipulating the units of observation to have different values on each of the two predictor variables so that they work better as predictors. But that doesn't mean that including both of them in your model as-is is bad or wrong. Throw them both in, why not - any random measurement error that's involved will also be partly addressed by the fact that you basically have two separate measurements of the same thing, so you might have a marginal increase in predictive power for that good reason (and in this sense I think I disagree with Kjetil's comment above).
When it comes to causal inference, it's a problem simply because it prevents us from being able to tell, confidently at least, which of the collinear predictors is doing the predicting, and therefore the explaining and, presumably, causing. With enough observations, you will eventually be able to identify the separate effects of even highly collinear (but never perfectly collinear) variables. This is why Rob Franzese and UMich likes to call multicollinearity "micronumerosity." There's always some collinearity between predictors. That's one of the reasons why we generally just need lots of observations. Sometimes an impossible amount, for our causal-inference needs. But the problem is the complexity of the world and the unfortunate circumstances that prevent us from observing a wider variety of situations where different factors vary more in relation to each other. Multicollinearity is the symptom of that lack of useful data, and multivariate regression is the (imperfect) cure. Yet so many people seem to think of multicollinearity as something they're doing wrong with their model, and as if it's a reason to doubt what findings they do have.
It's not an issue for predictive modeling when all you care about is the forecast and nothing else.
Consider this simple model: $$y=\beta+\beta_xx+\beta_zz+\varepsilon$$ Suppose that $z=\alpha x$
We have perfectly collinear regressors, and a typical OLS solution will not exist because $(X^TX)^{-1}$ has a singularity.
However, let's plug one equation into another: $$y=\beta+\beta_xx+\beta_z\alpha x+\varepsilon= \beta+\beta_2 x+\varepsilon,$$ where $\beta_2\equiv \beta_x+\beta_z\alpha$
So, clearly, we can estimate $\hat\beta_2$ by usual OLS methods, i.e. there is a solution. The only problem's that it's not unique!
We can choose any $\hat\beta_z$, which would give us $\hat\beta_x=\beta_2-\alpha\hat\beta_x$: we have infinite number of pairs $(\hat\beta_x,\hat\beta_z)$ that correspond to a unique solution $\hat\beta_2$. Obviously, any of these pairs is as good as any other for prediction of $\hat y$. Moreover, all these pairs are as good as the unique $\hat\beta_2$ coefficient for the purpose of forecasting.
The only problem is the inference. If you want to know how $x$ impacts $y$ your typical analysis of $\hat\beta_x$ coefficient and its variance will be pointless.
Multicollinearity is generally not the best scenario for regression analysis. Our life would be much easier if all predictors are orthogonal.
It's a problem for model interpretation (trying to understand the data):
Imagine if you have to write a report to your boss about your data. You build a near-perfect multicollinearity model, and tell your boss about the model. You might say "my first predictor is positively correlated with the response... I'm going to tell you more why.. Your boss is happy, but askes you to try again without a few data points. Your coefficients in your new model is now ... very different, the coefficient for your first predictor is now negative! Your boss won't trust you anymore! Your model is not robust.
Multicollinearity is still a problem for predictive power. Your model will overfit and less likely to generalize to out-of-sample data. Fortunately, your $R^2$ will be unaffected and your coefficients will still be unbiased.
I'd argue that if the correlation between a variable and another variable (or linear combination of variables) changes between the in-sample and out-of-sample data, you can start to see multicollinearity affecting the accuracy of out-of-sample predictions. Multicollinearity just adds another assumption (consistent correlation) that must be reasonably met for your model to keep performing well.
