21
$\begingroup$

Using Pearson's Correlation Coefficient, I have several variables that are highly correlated ($\rho = 0.978$ and $\rho = 0.989$ for 2 pairs of variables that are in my model).

The reason some of the variables are highly correlated is because one variable is used in the calculation for another variable.

Example:

$B = V / 3000$ and $E = V * D$

$B$ and $E$ have $\rho = 0.989$

Is it possible for me to just "throw away" one of the variables?

Improve this question
$\endgroup$

4 Answers 4

Reset to default
32
$\begingroup$

Both B and E are derived from V. B and E are clearly not truly "independent" variables from each other. The underlying variable that really matters here is V. You should probably disgard both B and E in this case and keep V only.

In a more general situation, when you have two independent variables that are very highly correlated, you definitely should remove one of them because you run into the multicollinearity conundrum and your regression model's regression coefficients related to the two highly correlated variables will be unreliable. Also, in plain English if two variables are so highly correlated they will obviously impart nearly exactly the same information to your regression model. But, by including both you are actually weakening the model. You are not adding incremental information. Instead, you are infusing your model with noise. Not a good thing.

One way you could keep highly correlated variables within your model is to use instead of regression a Principal Component Analysis (PCA) model. PCA models are made to get rid off multicollinearity. The trade off is that you end up with two or three principal components within your model that are often just mathematical constructs and are pretty much incomprehensible in logical terms. PCA is therefore frequently abandoned as a method whenever you have to present your results to an outside audience such as management, regulators, etc... PCA models create cryptic black boxes that are very challenging to explain.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ (+1) for the explanation of PCA. $\endgroup$
    – steffen
    Commented Nov 26, 2010 at 8:16
  • 1
    $\begingroup$ Thanks, this was a great explanation. I've heard and read about PCA, but this is for a final project for a "regression" graduate course I'm taking, and the professor just wants us to use LR. Regardless, I really appreciate the explanation of PCA and will probably use it myself for fun. $\endgroup$
    – TheCloudlessSky
    Commented Nov 26, 2010 at 11:38
  • 4
    $\begingroup$ In certain circumstances the recommendations in this answer would not work. For example, what if the true relationship is Y = B + E = V/3000 + V*D? Then the variables happen to have high correlation due to the ranges of V and D in the dataset--which is (or can be) pure accident--while throwing away either one of B or E will result in the wrong model. In short, "dependence" is not in general a valid reason for removing some variables from a model; including strongly dependent variables does not necessarily "weaken" a model; PCA is not always the way out. $\endgroup$
    – whuber ♦
    Commented Nov 26, 2010 at 17:49
  • $\begingroup$ @whuber, I am not sure I agree with your comments. I would think "dependence" is in general a pretty valid reason for removing some variables from a regression model. Otherwise, your regression coefficients can't be reliable. In the example you use that would be problematic for regression, one simple solution is to use the entire expression (V/3000 + V*D) as a single variable. $\endgroup$
    – Sympa
    Commented Nov 26, 2010 at 18:35
  • 5
    $\begingroup$ More generally, if the model is beta1*(V/3000) + beta2*(VD) you can't do this: in other words, your suggestion presumes you know a linear constraint among the coefficients. It is true that the regression coefficients can have *relatively large VIFs or standard errors, but with sufficient amounts of data--or with well chosen observations--the estimates will be reliable enough. So, we agree there is a problem and indeed I agree with your solution as one of several alternatives to consider. I disagree that it is as general and necessary as you make it out to be. $\endgroup$
    – whuber ♦
    Commented Nov 26, 2010 at 19:00
7
$\begingroup$

Here is an answer from the point of view of a machine learner, although I am afraid that I'll be beaten by real statisticians for it.

Is it possible for me to just "throw away" one of the variables?

Well, the question is what type of model you want to use for prediction. It depends e.g. on ...

  • can the model with correlated predictors ? E.g. although NaiveBayes theoretically has problems with correlated variables, experiments have shown that it still can perform well.
  • how does the model process the predictor variables ? E.g. the difference between B and V will be normalized out in a probability density estimation, maybe the same for E and V depending on the variance of D (as euphoria already said)
  • which usage combination of B and E (one, none, both) delivers the best result, estimated by a mindful crossvalidation + a test on a holdout set ?

Sometimes we machine learners even perform genetic optimization to find the best arithmetic combination of a set of predictors.

Improve this answer
$\endgroup$
0
7
$\begingroup$

B is a linear transform of V. E represents an interaction between V and D. Have you considered specifying a model that is Y = Intercept + V + D + V:D? As @euphoria83 suggests, it seems likely that there is little variation in D, so it may not solve your problem; however it should at least make the independent contributions of V and D clear. Be sure to center both V and D beforehand.

Improve this answer
$\endgroup$
1
  • 4
    $\begingroup$ +1: Not only is this suggestion a good approach to the problem in question, it shows that throwing away variables is not always the right (or even a good) approach to solving collinearity problems. $\endgroup$
    – whuber ♦
    Commented Nov 26, 2010 at 17:52
0
$\begingroup$

If D is not a constant, then B and E are effectively two different variables because of the variations in D. The high correlation indicates that D is practically constant throughout the training data. If that is the case, then you can discard either B or E.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Well D itself is a another equation that is calculated by other variables: $D = \frac{n_1}{2} * \frac{N_2}{n_2}$. Does this still apply? $\endgroup$
    – TheCloudlessSky
    Commented Nov 26, 2010 at 1:25
  • $\begingroup$ If you discard B or E and treat them as equivalent then you are implicitly asserting that V is all that really matters. If that is the case, you would be better off retaining B in the model as its interpretation is clear. Further, if you retain E, but D actually has limited variance, the validity of the interpretation of your results would even more suspect (than usual) for different values of D. $\endgroup$
    – russellpierce
    Commented Nov 26, 2010 at 3:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.