Multicollinearity in linear regression vs interpretability in new data

Question

I am working in a project using linear regression with a data set that has a lot of multicollinearity. From what I could understand from my research about the topic, I can separate the issues caused by multicollinearity in:

1. Problems related to the modelling itself, like the instability of the regression coefficients.
2. Interpretability problems

I found mainly 3 methods that can help, but I don't think they can solve the intepretability issue:

1. Using shrinkage methods such as Ridge and Elastic Net
2. Eliminating features with high VIF
3. Combining features suffering from multicollinearity

For example, let's say I have a dataset where my dependent variable is "health" and my independent variables are different types of medicines.

My questions are: 1. What if I want to compare the impact of each medicine to the health? 2. What if I get a dataset in which the intakes of medicines are differently correlated?

Thanks a lot!

Answer 1

Let's say, I am using X brand antidepressant and my doctor also prescribed me Y brand simulant to avoid my excess sleepyness caused by the antidepressant and lets say this is a common way of how doctors prescribe that particular antidepressant.

Here, I am making up an imaginary list of patient's drug usage and their health scores:

  X(mg/day) Y(mg/day)   Heart Health Score
    100       50             1000
    125       60             1200
    90        50             1800
    300       200            4000
    80        45             800
    10        0              100
    ...       ...

See the pattern? More drugs -> more healthy heart. Yet can you tell is it X or Y helping? No. This is, for me, is another definition of multicollinearity in the first place. No matter which regularization method you use, there is simply no way to know.

The methods for regularization (rigde, elastic net etc..) exploits this. In this case, I can simply eliminate one of them. In other words if I apply PCA(as a regularization step) to this data, one eigenvalue would be near zero leaving me with no collinearity in the resulting scores.

In a more complicated real life case with more variables and these variables being somewhat linearly dependent to each other in varoius degrees systematically or by chance there is nothing to do except to increase the number of observations to catch useful variaties of cases and to eliminate "by chance" part. To sum up, the interpretation problem is an unavoidable result of collinearity.

The answer to your second question is, if the intakes of medicines are differently correlated in another data set then the training set is not complete to cover these cases. That means that is not the data your model is trained for. It would be similar to using these medicine -> health model in another countries drug prescription data where the avoiding Y is promoted to doctors.