In a multiple linear regression with highly correlated regressors, what is the best strategy to use? Is it a legitimate approach to add the product of all the correlated regressors?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ I'm sorry see @Suncoolsu's answer was deleted. It and the comments that followed clarified a difference between multicollinearity and ill conditioning. Also, in a comment Suncoolsu pointed out how preliminary standardization can help with polynomial regression. If it happened to reappear I would vote it up ;-). $\endgroup$– whuber ♦Commented Oct 13, 2010 at 19:49
-
$\begingroup$ @Ηλίας : The product is likely to be unstable in many applications. It can be plagued by many zeros if the individual regressors have some zeros; its absolute value is likely to have strong positive skew, giving rise to some high-leverage points; it might amplify outlying data, especially simultaneous outliers, further adding to their leverage. It might be rather difficult to interpret, too, especially if the regressors already are re-expressions of the original variables (like logs or roots). $\endgroup$– whuber ♦Commented Oct 13, 2010 at 20:47
Add a comment
|
6 Answers
$\begingroup$
$\endgroup$
2
Principal components make a lot of sense... mathematically. However, I'd be wary of simply using some mathematical trick in this case and hoping that I don't need to think about my problem.
I'd recommend thinking a little about what kind of predictors I have, what the independent variable is, why my predictors are correlated, whether some of my predictors are actually measuring the same underlying reality (if so, whether I can just work with a single measurement and which of my predictors would be best for this), what I am doing the analysis for - if I'm not interested in inference, only in prediction, then I could actually leave things just as they are, as long as future predictor values are similar to past ones.
answered Oct 13, 2010 at 19:25
-
4$\begingroup$ Completely agreed, +1. But the characterization of PCA as a "mathematical trick" unfairly disparages it, IMHO. If you agree (I'm not sure you do) that summing or averaging groups of regressors, as Srikant suggests, would be acceptable, then PCA should be just as acceptable and it usually improves the fit. Moreover, the principal components can provide insight into which groups of predictors are correlated and how they correlate: that's an excellent tool for the thinking you are advocating. $\endgroup$– whuber ♦Commented Oct 13, 2010 at 20:40
-
4$\begingroup$ @whuber, I see and agree with your point, and I don't want to disparage PCA, so definitely +1. I just wanted to point out that blindly using PCA without looking at and thinking about the underlying problem (which no one here is advocating) would leave me with a bad feeling... $\endgroup$ Commented Oct 13, 2010 at 21:22
$\begingroup$
$\endgroup$
1
You can use principal components or ridge regression to deal with this problem. On the other hand, if you have two variables that are correlated highly enough to cause problems with parameter estimation, then you could almost certainly drop either one of the two without losing much in terms of prediction--because the two variables carry the same information. Of course, that only works when the problem is due to two highly correlated independents. When the problem involves more than two variables that are together nearly collinear (any two of which may have only moderate correlations), you'll probably need one of the other methods.
-
2$\begingroup$ (+1) Now, the problem is that the OP didn't indicate how many variables enter the model, because in case they are numerous it might be better to do both shrinkage and variable selection, through e.g. the elasticnet criterion (which is combination of Lasso and Ridge penalties). $\endgroup$– chlCommented Oct 13, 2010 at 18:22
$\begingroup$
$\endgroup$
1
Here is another thought that is inspired by Stephan's answer:
If some of your correlated regressors are meaningfully related (e.g., they are different measures of intelligence i.e., verbal, math etc) then you can create a single variable that measures the same variable using one of the following techniques:
Sum the regressors (appropriate if the regressors are components of a whole, e.g., verbal IQ + math IQ = Overall IQ)
Average of the regressors (appropriate if the regressors are measuring the same underlying construct e.g., size of left shoe, size of right shoe to measure length of feet)
Factor analysis (to account for errors in measurements and to extract a latent factor)
You can then drop all the correlated regressors and replace them with the one variable that emerges from the above analysis.
-
1$\begingroup$ This makes sense if the regressors are all measured on the same scale. In psychology, various subscales are often measured on different scales (and still correlated), so a weighted sum or average (which is really the same here) would be appropriate. And of course, one could view PCA as providing just this kind of weighting by calculating axes of maximum variance. $\endgroup$ Commented Oct 13, 2010 at 21:24
$\begingroup$
$\endgroup$
2
I was about to say much the same thing as Stephan Kolassa above (so have upvoted his answer). I'd only add that sometimes multicollinearity can be due to using extensive variables which are all highly correlated with some measure of size, and things can be improved by using intensive variables, i.e. dividing everything through by some measure of size. E.g. if your units are countries, you might divide by population, area, or GNP, depending on context.
Oh - and to answer the second part of the original question: I can't think of any situation when adding the product of all the correlated regressors would be a good idea. How would it help? What would it mean?
-
$\begingroup$ My initial idea was to add take into account the pairwise interaction of the regressors $\endgroup$– ΗλίαςCommented Oct 13, 2010 at 22:17
-
$\begingroup$ It is often a good idea to take into account pairwise interaction. But not all lof them: You need to think trough which makes sense! $\endgroup$ Commented Jul 9, 2014 at 0:20
$\begingroup$
$\endgroup$
4
I'm no expert on this, but my first thought would be to run a principal component analysis on the predictor variables, then use the resulting principal components to predict your dependent variable.
answered Oct 13, 2010 at 17:48
-
$\begingroup$ Nice idea. (It doesn't generalize well to categorical predictors, though.) I suspect that many other strategies could be interpreted from this point of view, too. For example, selecting a subset of $k$ of the predictors could be interpreted as approximating a basis for the span of the $k$ largest eigenvectors in a PCA. $\endgroup$– whuber ♦Commented Oct 13, 2010 at 17:59
-
$\begingroup$ In an explanatory approach, then you have to interpret how your linear combination(s) of the $p$ variables relate to the outcome, and this might sometimes be tricky. $\endgroup$– chlCommented Oct 13, 2010 at 18:05
-
$\begingroup$ @chl Good point. But since the principal components are linear combinations, it's straightforward (although sometimes a bit of a pain) to compose the fitted regression model (=one linear transformation) with the projection onto the components (=another linear transformation) to obtain an interpretable linear model involving all the original variables. This is somewhat akin to orthogonalization techniques. Note, too, that Srikant's latest proposals (sum or average the regressors) essentially approximate the principal eigenvector yet induce similar explanatory difficulties. $\endgroup$– whuber ♦Commented Oct 13, 2010 at 20:37
-
$\begingroup$ @whuber Yes, I agree with both of your points. I extensively used PLS regression and CCA, so in this case we have to deal with linear combinations on both side (st. a max. covariance or correlation criteria); with a large number of predictors, interpreting the canonical vectors is painful, so we merely look at the most contributing variables. Now, I can imagine that there is not so much predictors so that all of your arguments (@Stephan, @Mike) make sense. $\endgroup$– chlCommented Oct 14, 2010 at 5:56
$\begingroup$
$\endgroup$
7
One of the ways to reduce the effects of correlation is to standardize the regressors. In standardizing, all the regressors are subtracted by their respective means and divided by their respective standard deviations. Specifically, if $X$ is the regression matrix:
$$x_{ij}^{standardized}=\frac {x_{ij}-\overline{x_{.j}}} {s_{j}}$$
This is not a remedy, but definitely a step in the right direction.
-
8$\begingroup$ Linear transformations (like these) never change correlation coefficients. The point to standardization is to improve the conditioning of the normal matrix. $\endgroup$– whuber ♦Commented Oct 13, 2010 at 17:57
-
1$\begingroup$ Standardizing the variables will not affect the correlations among the independent variables and will not "reduce the effect of correlation" in any way that I can think of with respect to this problem. $\endgroup$– BrettCommented Oct 13, 2010 at 18:15
-
2$\begingroup$ @Brett, a typical example where standardization helps is Polynomial Regression. It is always recommended to standardize the regressors. Standardizing doesn't change the correlation matrix, but makes the var cov matrix (which is now the correl matrix) well behaved (called conditioning by @whuber pointing to the condition number of the matrix, IMHO). $\endgroup$ Commented Oct 13, 2010 at 18:23
-
$\begingroup$ Agreed. Centering is useful when entering higher order terms, like polynomial or interaction terms. That doesn't seem to be the case here and will not otherwise help with the problem of correlated predictors. $\endgroup$– BrettCommented Oct 13, 2010 at 18:28
-
$\begingroup$ I deleted it because I didn't want to confuse people with wrong answer. Probably the moderators brought it up again. $\endgroup$ Commented Oct 14, 2010 at 0:14