# Variable with negative coefficient in Ridge Regression and positive correlation

In my research, I have four independent variables (X1, X2, X3 and X4) and one response variable (Y). Upon checking the VIF values of the explanatory variables, I noticed that they have high multicollinearity, so I decided to use a Ridge Regression to make predictions about the variables, without, of course, using their P-values. When I plotted the Ridge Trace Plot and the chart with the regression coefficients from the Ridge Regression outcome, I noticed that X2 and X4 had negative effects on the model.

I wasn't surprised that this would happen with X4, but I didn't expect the same to happen with X2. I was even more puzzled when I performed a simple linear regression comparing Y and X2, and noticed a positive correlation.

So I decided to remove X3 from my study, since this variable had the highest VIF value (about 13), and I realized that everything made more sense, with X2 presenting a positive influence on the model.

Should I then delete X3 permanently? If I can include this variable in the model, how would I explain the fact that X2 has a negative effect on the model, and yet it has a positive correlation with Y?

We can ignore the ridge regression bit, and we can also suppose you have just two variables, $$X_1$$ and $$X_3$$ (these are the two most relevant to your question). Suppose that $$E[Y|X_1,X_3] = \beta_0+\beta_1X_1+\beta_3X_3$$. Then, iterating the expectation, $$E[Y|X_1] = E_{X_3|X_1} E[Y|X_1,X_3] = \beta_0 + \beta_1X_1 + \beta_3 E[X_3|X_1]$$. Now, further suppose that $$E[X_3|X_1] = \gamma_0+\gamma_1X_1$$. Then, $$E[Y|X_1] = \beta_0 + \beta_1X_1 + \beta_3 (\gamma_0+\gamma_1X_1)\\ = \beta_0 + \beta_3 \gamma_0+(\beta_1+\gamma_1)X_1$$
So, if $$\beta_1+\gamma_1 > 0$$ and $$\beta_1<0$$, then you would expect the sort of findings that you are encountering in your data analysis. Moreover, the regression coefficients are functions of correlations: if the assumed model is correct, then $$\gamma_1= \mathrm{cor}(X_1,X_3)\sqrt{\dfrac{\mathrm{var}(X_3)}{\mathrm{var}(X_1)}}$$. So, the sign of $$\gamma_1$$ is determined by the sign of the correlation, and the magnitude of $$\gamma_1$$ is driven by the size of the correlation and the relative ratios of the variances of $$X_3$$ to $$X_1$$.
• I believe I understood your explanation, thank you very much! In this case, how then could I investigate in R this possible empiric variance in X3? If it's really happening, would it be interesting to develop the Ridge Regression without this variable? Aug 13, 2021 at 4:39
• I made some changes to my answer based on your comment. What I mean is look at the correlation between $X_1$ and $X_3$ and the variances of $X_1$ and $X_3$, i.e. cor(X1,X3), var(X1), and var(X3). Re: your question about ridge regression, it's not a bad thing to do here. Are you using AIC or something to choose your tuning parameter? Aug 24, 2021 at 1:55