I am applying binary logistic regression as my dependent variable is a dichotomous variable with 740 sample size. I have used enter method to input my variables and have designed two blocks. In block 1 I have tried to see the impact of only demographic variables and in block 2 I have added 10 new predictors. Chi-Square value has increased significantly from block 1 to block 2, Hosmer and lemeshow test has value greater than 0.05 (It was 0.000 for block 1)and -2 log likelihood has also decreased drastically from block 1 to block 2, indicating good model fit. But In block 1 I can see significant impact of some of the demographic variables, however in block 2 there is no impact of any of the predictors. Please help me with my output..what does it signify? Am i mistaking somewhere?
-
1$\begingroup$ No impact, or no significant impact with p < .05? $\endgroup$– Nick StaunerJul 14, 2014 at 11:48
-
$\begingroup$ I mean no significant impact of any of the predictors. sig level is greater than 0.05 $\endgroup$– Priyanka SinhaJul 14, 2014 at 11:51
-
1$\begingroup$ does model 2 outperform model 1 if you do cross-validation as well? Also, are the 10 new predictors linearly related amongst themselves? It could just be that it's hard for the linear regression to tell which one is actually having an effect hence the high p-values. $\endgroup$– CarrKnightJul 14, 2014 at 11:54
-
1$\begingroup$ Adding predictors that explain some variance can improve model fit even if those predictors don't have clear, independent effects. Adding more predictors that aren't completely unrelated to the outcome can also decrease the amount of outcome variance that covaries uniquely with previous predictors. Are you hoping to understand relationships between your predictors and outcome, or is predicting the outcome the only major priority? $\endgroup$– Nick StaunerJul 14, 2014 at 11:57
-
1$\begingroup$ if removing the 4 correlated parameters increases the cv error that's proof enough to me they should stay in. You could do likelihood ratio test, but it seems inferior IMHO. As to what to do, the answer is not much. Given how tightly related the 4 predictors are there is no way of telling the effects of each of them separately. Apparently $\endgroup$– CarrKnightJul 14, 2014 at 12:08
2 Answers
Calibration (e.g., Hosmer-Lemeshow) is unrelated to the issue at hand, and a large $\chi^2$ does not mean that the model fit is good. But to your main point, it is common for competing (co-linear) variables to be "insignificant" whereas the whole lot is significant in a composite (pooled, chunk) test. Try to understand the relationships amongs the predictors, and get pooled tests of difficult-to-separate variables. Variable clustering is one of many ways to better understand the predictors. Redundancy analysis is another. These are implemented in the R Hmisc package's varclus and redun functions.
To add to Frank's answer, this is precisely what nested F-tests (and their GLM analogue, nested Deviance and Likelihood Ratio tests) are for. The idea is that you're testing several variables for joint significance, as if they were somehow collected into one variable.
Also remember that 0.05 is completely arbitrary. Does a 1 in 20 chance of mistakenly finding a non-zero effect make sense for your data? By boss was telling me that, for his work, he did some cost-benefit analyses with power and significance level, and found that the business risk of finding a spurious association was lower than that of having everything come out non-significant and not being able to make a decision based on the data.
And finally, the right way to proceed depends entirely on your goals. Are you interested in prediction? Then per-coefficient significance doesn't matter, although many badly non-significant coefficients could indicate overfitting. Are you interested in a particular coefficient? Then if you have highly collinear predictors you will not be able to identify true values along a causal path without a richer model.
