For selection of independent variables in multiple regression analysis, first have calculated correlation coefficient between dependent (y) and independent variables (x1, x2, x3, x4 & x5) and then I performed multiple regression analysis using significantly correlated variables. Result shows:

y & x1 = 0.91; y & x2 = 0.90; y & x3 = 0.23; y & x4 = 0.18; y & x5 = 0.23; all the correlation values are significant (pval <0.05).

In multiple regression to derive linear model, I started with two predictors, x1 & x2 which are highly correlated and sequentially added the rest of the predictors.

For model1 (x1 & x2) the R-square value is 0.88 and coefficient values of x1 and x2 are significant.

For model2 (x1, x2 & x3) the R-square value is 0.88 and coefficient values are significant for x1 & x2 while insignificant for x3.

Similar R-square value is obtained for model3 (x1, x2, x3 & x4) and the coefficent values are insignificant for x3 & x4.

For model4 (x1, x2, x3, x4 & x5) the higher R-square value (0.90) is obtained and coefficient values of x1 and x5 are significant while other variables (x2, x3 & x4) shows insignificant coefficient value.

correlation matrix







My question is, in model4 why x2 is showing insignificant coefficient value while it is highly correlated with y? And finally which model I should consider for prediction of y?

%%%%%%%%%%% Edits 25.05.2018 %%%%%%%%%%%%%%%%%%%%

I made some changes in analysis. I removed x5 variable and added another variable instead of net radiation. So, this time I used x1 (rainfall), x2 (soil moisture), x3 (temperature) and x4 (vegetation data) and all models, passed the test of significance in the regression at a p value of 0.05.Now result shows: For model1 (x1 & x2) the R-square value is 0.88 and standard error is 18.29, For model2 (x1, x2 & x3) the R-square value is 0.89 and standard error is 17.53, For model3 (x1, x2, x3 & x4) R-square value is 0.89 and standard error is 17.42. You can see R-square and standard error values are almost similar. In this case, can I calculate accuracy (in percentage) in each model?


My question is, in model4 why x2 is showing insignificant coefficient value while it is highly correlated with y?

This happens when the independent variables (x1 thru x5) are themselves correlated with each other. This is known as 'multicollineairy', and also as 'confounding' in epidemiology. When the inclusion of an independent variable (x5) artificially reduces the observed correlation of another independent variable that's already in the model (x2) with the dependent variable. You can check the correlation matrix of all independent variables, and see which ones are heavily correlated with other(s). Depending on the goal of your analysis, you may need to drop one or more variables that are heavily correlated with other variable(s).

And finally which model I should consider for prediction of y?

It's hard to answer this question without a proper context. But in general, I would recommend looking at the Adjusted R-Squared values of your models, rather than the R-Squared values that you're using to determine which model you should keep. As you add more independent variables in the model, the R-Squared value is always going to increase, so it's not a good guide to determine when you should stop adding more variables into the model.

The Adjusted R-Squared value is a modified version of the R-Squared value that takes into account the number of variables in the model. The Adjusted R-Squared value increases only if the new variable that was added to the model improves the model more than what would happen just by chance. It can even decrease if the addition of a new variables does not improve the model more than expected by chance!

  • $\begingroup$ Hello Vishal, Thanks a lot for your kind response. I have added correlation matrix in question, please see. Basically these x1 (rainfall), x2 (soil moisture), x3 (temperature), x4 (net radiation) and x5 (wind speed) are hydrologic and climatic variables which shows impact on evaporation (y). And as you suggested to check R-square and adjusted R-square values, I observed that both values are almost similar. I am not able to figure out, please throw some light. $\endgroup$ – SONY May 15 '18 at 15:10
  • $\begingroup$ Since some of your predictors are heavily correlated with each other, you're going to see some strange results (significant predictor becomes insignificant, etc.) when you include all of those predictors in the model. To determine which model to use, I would recommend using LASSO regression, which will help you select the best set of variables automatically (rather than you trying to manually pick predictors). $\endgroup$ – Vishal May 15 '18 at 15:24
  • $\begingroup$ Thank you Vishal for the guidance. I am looking at the method that you recommend. $\endgroup$ – SONY May 17 '18 at 15:16
  • $\begingroup$ Hello Vishal, Please see my edited question. $\endgroup$ – SONY May 25 '18 at 18:01

1) I see two possible reasons why such situation may happen:

A) Size: The correlation is linked to the regression coefficient in simple regression. In a multiple regression context, what determines the size of the coefficient (that is obviously related to its significance) is partial correlation, i.e. residual correlation after the other regressors have been accounted for. A brief description of the concept may be found here: http://www.statisticshowto.com/partial-correlation/

B) Standard error: Inclusion of other regressors modifies the standard error of estimated coefficients (that's the other factor determining statistical significance). Such issue is called multicollineairty, and can be measured by Variance Inflation Factor. You can take a look, for example, here:

https://statisticalhorizons.com/multicollinearity http://support.minitab.com/en-us/minitab/17/topic-library/modeling-statistics/regression-and-correlation/model-assumptions/what-is-a-variance-inflation-factor-vif/

2) In your case, it's possible that $x_2$ is particularly associated with $x_5$. However, to understand what's actually going on, you can see how inclusion and exclusion of each regressor (or group of regressors) affects other regressors' estimated coefficients and SE's.

3) In general, I think that, in your presentation of your data, you are not including a very relevant piece of information: the correlation between your $x$'s. In case some pairs are highly correlated (or, looking at groups of variables, its linear dependency is very high) then you should either drop one (or more) of them, or think about some variable-reduction technique. Model choice should however be based on theoretical grounds: the stepwise approach (i.e. adding significant and removing non-significant parameters) is today considered a very flawed one. See, for example, here: What are the advantages of stepwise regression?

***** EDIT 11/05/18******

Things are not straightforward. Your correlation matrix highlights that $X_1, X_2$ and $Y$ are highly correlated, as well as $X_3$ and $X_4$. Inclusion of the last three variables lead to the parameter associated with $X_2$ turning non-significant. Nevertheless, there is not a strong correlation between $X_2$ and $X_3, X_4$ and $X_5$: I guess that the coefficient and standard erros don't change a lot, and the result is due to the fact $X_2$ was already at the border of statistical significance. In my opinion, this is due to the strong collinearity between $X_1$ and $X_2$ (in simple regression, I guess $X_2$ would be highly significant). May you tell me what your variables are? The correlations between $X_1$, $X_2$ and $Y$, and between $X_3$ and $X_4$ are so high that I wonder whether you should consider a different approach.

***** EDIT 17/05/18*******

I think that, in your case:

1) Rainfall has a huge explanatory power on evaporation.

2) Once you control for rainfall, soil moisture retains a statistically significant predictive power.

3) However, after controlling for rainfall, soil moisture has a relevant association with wind speed: due to such association, including the latter variable among regressors make the former non-significant.

Thus, I guess there should be a strong partial correlation between soil moisture and wind speed controlling for rainfall. It seems to me this makes sense:

A) soil moisture could be affected by rainfall and wind speed so that, after we control for the former, its residual correlation with the latter becomes noticeable.

B) Most of evaporation may be predicted by rainfall and soil moisture, but also by rainfall and wind speed (due to the strong partial correlation between $X_2$ and $X_5$ given $X_1$, if I'm guessing correctly).

  • $\begingroup$ Hello Frederico Tedeschi, Thank you for the response. I have added correlation matrix in question. $\endgroup$ – SONY May 10 '18 at 19:40
  • $\begingroup$ Hello Frederico, Basically these x1 (rainfall), x2 (soil moisture), x3 (temperature), x4 (net radiation) and x5 (wind speed) are hydrologic and climatic variables which shows impact on evaporation (y). $\endgroup$ – SONY May 14 '18 at 15:48
  • $\begingroup$ Hello Sony. Please see my edits $\endgroup$ – Federico Tedeschi May 17 '18 at 21:01
  • $\begingroup$ Hi Frederico, Thanks for your comments. Your all the points are correct. Please see my edits in question. $\endgroup$ – SONY May 25 '18 at 17:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.