1
$\begingroup$

I have three independent continuous variables which are highly correlated. The dependent variable is a continuous one.

In this case, should I apply three separate regression between each independent variable and the dependent variable since the independent variables are highly correlated

The correlation matrix (Pearson) shows that the correlations are 0.64, 0.8, 0.63. The aim of using the regression is finding the effect of the independent variables. Also in this field of medical research, we prefer to apply linear regression since interpreting the coefficients is more understandable.

Improve this question
$\endgroup$
4
  • 5
    $\begingroup$ What question are you trying to answer with the regression? The correct thing to do depends on your intent. $\endgroup$
    – Matthew Drury
    Commented Sep 9, 2017 at 2:15
  • $\begingroup$ @ Matthew Drury, Thanks for the reply. I edited the question. $\endgroup$
    – joe
    Commented Sep 9, 2017 at 14:02
  • $\begingroup$ Quick comments. (1) What's sensible depends how much independent data you have and the magnitude of the error terms. If the collinearity isn't 100%, then you'd recover the true coefficients with enough data. But with small samples, you could have the usual multicollinearity problems and symptoms such as large standard errors on individual coefficients. In that case, you're right that distinguishing $b_1$ from $b_2$ may not be possible. $\endgroup$
    – Matthew Gunn
    Commented Sep 9, 2017 at 18:35
  • $\begingroup$ (2) Is this an observational study? Or is the variation in $x$ from an RCT? (With ordinary least squares (OLS), you're estimating a linear conditional expectation function, and you need a specialized setting, additional assumptions for those coefficients to be causal effects. $\endgroup$
    – Matthew Gunn
    Commented Sep 9, 2017 at 18:35

5 Answers 5

Reset to default
2
$\begingroup$

Collinearities are not a problem in all regression techniques, you know... You may want to check out component-based options (e.g. principle component regression or PLS regression) where you can reduce your original independent data to a low rank estimation which can be used for a regression with your dependent data.

EDIT:

Supposing that you have covariation among your IVs you may get a better regression using a latent variable, which will have higher signal/noise ratio. Moreover, if you perform regression individually you risk losing out on potentially important information on more complex interaction patterns between IV. There may e.g. be systematic differences between something which is captured in one of your IVs, such as men/women disease severity score or number of employees in a company.

If the overall variability in your IVs are expected to covary with your DV, then I would just go for a PC-regression. However, if you think that there is greater need to tease out the relevant contributions from the variables to your DV, then I would do a PLS regression.

Anyway, when you make these kinds of regressions, make sure to combine modelling with proper validation (hard splits/external validation if you have enough samples, nested cross-validation otherwise. Single n-fold cross-validation as a last option.)

When, later, you want to check the contributions from your variables to the regression model, you can examine the variable loadings in the PCA or PLS to draw some inference...

Improve this answer
$\endgroup$
2
  • $\begingroup$ thanks for the reply. Since the aim of the research is finding the effect of the IV's on the dependent variable and interpreting the coefficient of a linear regression is easier to understand, do you still recommend using PCA or PLS regression? Do you believe applying three different regressions including each IV and the Dependent variable is a wrong idea? $\endgroup$
    – joe
    Commented Sep 9, 2017 at 14:01
  • $\begingroup$ No worries ;) See edit in my previous response. $\endgroup$
    – CarlBrunius
    Commented Sep 9, 2017 at 14:23
2
$\begingroup$

The appropriate course of action may depend on what the three variables are.

(1) they are effectively measuring the same thing in three slightly different ways. You mention that you are working in health and there are often variables which might be measured in different ways. In that case you might want to combine them into a single variable either by using PCA or simply by summing them if they are measured on commensurate scales.

(2) they are measuring three things which are not correlated by design but just as a matter of fact and you do really want to look at each separately. In that case your multiple regression is telling you the effect of each over and above the effect of the others. So if they are, say, systolic arterial tension, diastolic, and pulse rate, you get the effect of each over and above the others.

With modern software and the moderate correlations you mention there should be no problem actually fitting the model.

Improve this answer
$\endgroup$
0
1
$\begingroup$

In general, multicollinearity is not a serious issue compared to other issues such as endogeniety, heteroscedasticity, etc.

First, perform the multicollinearity test. If it is not the case of perfect multicollinearity (i.e., r = 1), the problem is not that serious. You could perform regression analysis with all three variables if it is not perfectly collinear.

If the variables are perfectly, or nearly perfectly multicollinear, you could drop the variables and leave only one.

This is just a basic and general recommendation. As mentioned in the comment, it depends on your intent and objective of your analysis. Next time, be sure to clarify and specify your question.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the comment. The aim of the research is finding the effect of the IV's. The Pearson correlation matrix is 0.64,0.88,0.63. How can I know they are perfectly multicollinear? Also, do you think the idea of applying three separate regressions between each independent and dependent is a wrong idea? $\endgroup$
    – joe
    Commented Sep 9, 2017 at 13:57
  • $\begingroup$ @joe No they are not perfectly colinear. Perfectly colinear means that the correlation coefficient equals 1. $\endgroup$
    – Buomsoo Kim
    Commented Sep 9, 2017 at 14:22
  • $\begingroup$ @joe Rather than doing separate regressions for each variable, try applying partial least squares. They are one of the common measures for multicollinearity issue $\endgroup$
    – Buomsoo Kim
    Commented Sep 9, 2017 at 14:24
  • $\begingroup$ @mdewey, thanks so much for your time. I really appreciate that. $\endgroup$
    – joe
    Commented Sep 10, 2017 at 22:36
1
$\begingroup$

There are many ways to deal with collinearity but, given your concerns and desires, I am very surprised that no one has mentioned ridge regression.

In addition, be aware that the correlation matrix is not the best way to evaluate colinearity. You should look at condition indexes and proportion of variance explained.

Improve this answer
$\endgroup$
1
  • $\begingroup$ thanks so much for your reply, I was wondering if ridge regression is a linear model? $\endgroup$
    – joe
    Commented Sep 9, 2017 at 20:44
-1
$\begingroup$

Well there are many ways to tackle multicollinearity. Since you want to interpret the coefficients of the regression line, using the principal component method won't probably be very helpful. Since the new variables obtained by the principal component method are a linear combination the original variables. You could try the backward elimination method. Fit the regression line with all the variables and drop the variable with the highest p-value. Continue this until you get all the predictors as significant at your desired level. The final model which you obtain will give you which predictor(s) influence your response the most. Its a crude way, but it's worth a try.

Improve this answer
$\endgroup$
1
  • $\begingroup$ You're welcome, hope it helps $\endgroup$
    – Dante
    Commented Sep 24, 2017 at 13:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.