I am working in a project using linear regression with a data set that has a lot of multicollinearity. From what I could understand from my research about the topic, I can separate the issues caused by multicollinearity in:

  1. Problems related to the modelling itself, like the instability of the regression coefficients.
  2. Interpretability problems

I found mainly 3 methods that can help, but I don't think they can solve the intepretability issue:

  1. Using shrinkage methods such as Ridge and Elastic Net
  2. Eliminating features with high VIF
  3. Combining features suffering from multicollinearity

For example, let's say I have a dataset where my dependent variable is "health" and my independent variables are different types of medicines.

My questions are: 1. What if I want to compare the impact of each medicine to the health? 2. What if I get a dataset in which the intakes of medicines are differently correlated?

Thanks a lot!

  • 1
    $\begingroup$ The three methods that you mention are effective ways to produce regression models that are more stable than models that include all the predictors. Prediction results however do not change much whichever choice you make. $\endgroup$ Commented Apr 7, 2017 at 16:36

1 Answer 1


Let's say, I am using X brand antidepressant and my doctor also prescribed me Y brand simulant to avoid my excess sleepyness caused by the antidepressant and lets say this is a common way of how doctors prescribe that particular antidepressant.

Here, I am making up an imaginary list of patient's drug usage and their health scores:

  X(mg/day) Y(mg/day)   Heart Health Score
    100       50             1000
    125       60             1200
    90        50             1800
    300       200            4000
    80        45             800
    10        0              100
    ...       ...

See the pattern? More drugs -> more healthy heart. Yet can you tell is it X or Y helping? No. This is, for me, is another definition of multicollinearity in the first place. No matter which regularization method you use, there is simply no way to know.

The methods for regularization (rigde, elastic net etc..) exploits this. In this case, I can simply eliminate one of them. In other words if I apply PCA(as a regularization step) to this data, one eigenvalue would be near zero leaving me with no collinearity in the resulting scores.

In a more complicated real life case with more variables and these variables being somewhat linearly dependent to each other in varoius degrees systematically or by chance there is nothing to do except to increase the number of observations to catch useful variaties of cases and to eliminate "by chance" part. To sum up, the interpretation problem is an unavoidable result of collinearity.

The answer to your second question is, if the intakes of medicines are differently correlated in another data set then the training set is not complete to cover these cases. That means that is not the data your model is trained for. It would be similar to using these medicine -> health model in another countries drug prescription data where the avoiding Y is promoted to doctors.

  • $\begingroup$ It makes sense! But I was hoping that two things would help in trying to better understand the differences in the relation of the two drugs and the health: 1. The small differences between them and how the dependent variable responds. 2. The observations that don't follow the same relation. For example, let's say we have a smaller number of patients that only took one of the medications. Maybe somehow we can "force" the algorithm to learn more from these observations. Maybe the information is there, but I can't see it because I am only looking at the big picture. Does it make sense? $\endgroup$
    – jcp
    Commented Apr 10, 2017 at 13:08
  • $\begingroup$ It does make sense. In that cases, however, there would be less or no collinearity. Maybe you should simulate the cases and apply PCA to those data. I am suggesting PCA because it is, in my opinion, somewhat easier to interpret by looking at number of relatively big eigenvalues etc.. $\endgroup$
    – gunakkoc
    Commented Apr 10, 2017 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.