Variable with negative coefficient in Ridge Regression and positive correlation

Question

In my research, I have four independent variables (X1, X2, X3 and X4) and one response variable (Y). Upon checking the VIF values of the explanatory variables, I noticed that they have high multicollinearity, so I decided to use a Ridge Regression to make predictions about the variables, without, of course, using their P-values. When I plotted the Ridge Trace Plot and the chart with the regression coefficients from the Ridge Regression outcome, I noticed that X2 and X4 had negative effects on the model.

I wasn't surprised that this would happen with X4, but I didn't expect the same to happen with X2. I was even more puzzled when I performed a simple linear regression comparing Y and X2, and noticed a positive correlation.

So I decided to remove X3 from my study, since this variable had the highest VIF value (about 13), and I realized that everything made more sense, with X2 presenting a positive influence on the model.

Should I then delete X3 permanently? If I can include this variable in the model, how would I explain the fact that X2 has a negative effect on the model, and yet it has a positive correlation with Y?

Answer 1

We can ignore the ridge regression bit, and we can also suppose you have just two variables, $X_1$ and $X_3$ (these are the two most relevant to your question). Suppose that $E[Y|X_1,X_3] = \beta_0+\beta_1X_1+\beta_3X_3$. Then, iterating the expectation, $E[Y|X_1] = E_{X_3|X_1} E[Y|X_1,X_3] = \beta_0 + \beta_1X_1 + \beta_3 E[X_3|X_1]$. Now, further suppose that $E[X_3|X_1] = \gamma_0+\gamma_1X_1$. Then, $$E[Y|X_1] = \beta_0 + \beta_1X_1 + \beta_3 (\gamma_0+\gamma_1X_1)\\ = \beta_0 + \beta_3 \gamma_0+(\beta_1+\gamma_1)X_1$$

So, if $\beta_1+\gamma_1 > 0$ and $\beta_1<0$, then you would expect the sort of findings that you are encountering in your data analysis. Moreover, the regression coefficients are functions of correlations: if the assumed model is correct, then $\gamma_1= \mathrm{cor}(X_1,X_3)\sqrt{\dfrac{\mathrm{var}(X_3)}{\mathrm{var}(X_1)}}$. So, the sign of $\gamma_1$ is determined by the sign of the correlation, and the magnitude of $\gamma_1$ is driven by the size of the correlation and the relative ratios of the variances of $X_3$ to $X_1$.