0
$\begingroup$

In my research, I have four independent variables (X1, X2, X3 and X4) and one response variable (Y). Upon checking the VIF values of the explanatory variables, I noticed that they have high multicollinearity, so I decided to use a Ridge Regression to make predictions about the variables, without, of course, using their P-values. When I plotted the Ridge Trace Plot and the chart with the regression coefficients from the Ridge Regression outcome, I noticed that X2 and X4 had negative effects on the model.

I wasn't surprised that this would happen with X4, but I didn't expect the same to happen with X2. I was even more puzzled when I performed a simple linear regression comparing Y and X2, and noticed a positive correlation.

So I decided to remove X3 from my study, since this variable had the highest VIF value (about 13), and I realized that everything made more sense, with X2 presenting a positive influence on the model.

Should I then delete X3 permanently? If I can include this variable in the model, how would I explain the fact that X2 has a negative effect on the model, and yet it has a positive correlation with Y?

$\endgroup$
3
  • 1
    $\begingroup$ Simpson's paradox? en.wikipedia.org/wiki/Simpson%27s_paradox $\endgroup$
    – psboonstra
    Aug 13, 2021 at 1:29
  • $\begingroup$ @psboonstra, but would Simpson's paradox apply to numerical variables? $\endgroup$
    – Filipe
    Aug 13, 2021 at 1:47
  • 1
    $\begingroup$ yes, I believe it applies. its just a bit harder to visualize. but see my answer which I believe captures simpsons paradox for your case $\endgroup$
    – psboonstra
    Aug 13, 2021 at 2:11

1 Answer 1

1
$\begingroup$

We can ignore the ridge regression bit, and we can also suppose you have just two variables, $X_1$ and $X_3$ (these are the two most relevant to your question). Suppose that $E[Y|X_1,X_3] = \beta_0+\beta_1X_1+\beta_3X_3$. Then, iterating the expectation, $E[Y|X_1] = E_{X_3|X_1} E[Y|X_1,X_3] = \beta_0 + \beta_1X_1 + \beta_3 E[X_3|X_1]$. Now, further suppose that $E[X_3|X_1] = \gamma_0+\gamma_1X_1$. Then, $$E[Y|X_1] = \beta_0 + \beta_1X_1 + \beta_3 (\gamma_0+\gamma_1X_1)\\ = \beta_0 + \beta_3 \gamma_0+(\beta_1+\gamma_1)X_1$$

So, if $\beta_1+\gamma_1 > 0$ and $\beta_1<0$, then you would expect the sort of findings that you are encountering in your data analysis. Moreover, the regression coefficients are functions of correlations: if the assumed model is correct, then $\gamma_1= \mathrm{cor}(X_1,X_3)\sqrt{\dfrac{\mathrm{var}(X_3)}{\mathrm{var}(X_1)}}$. So, the sign of $\gamma_1$ is determined by the sign of the correlation, and the magnitude of $\gamma_1$ is driven by the size of the correlation and the relative ratios of the variances of $X_3$ to $X_1$.

$\endgroup$
2
  • $\begingroup$ I believe I understood your explanation, thank you very much! In this case, how then could I investigate in R this possible empiric variance in X3? If it's really happening, would it be interesting to develop the Ridge Regression without this variable? $\endgroup$
    – Filipe
    Aug 13, 2021 at 4:39
  • $\begingroup$ I made some changes to my answer based on your comment. What I mean is look at the correlation between $X_1$ and $X_3$ and the variances of $X_1$ and $X_3$, i.e. cor(X1,X3), var(X1), and var(X3). Re: your question about ridge regression, it's not a bad thing to do here. Are you using AIC or something to choose your tuning parameter? $\endgroup$
    – psboonstra
    Aug 24, 2021 at 1:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.