0
$\begingroup$

Scenario A: In Cox regression, the chi square analysis of the -2 log likelihood (or the omnibus in binomial) is not significant, but some of the values of the Wald statistic corresponding to some of the independent variables (IVs) are significant, in addition to Exp (B) values greater than 1, with a confidence interval (CI) that does not include 1.

Questions: is the model by default not worthwhile because of the insignificant chi squared? Can/should the Exp(B) values pertaining to significant Wald be reported as risk ratios?

Scenario B: Binomial univariate models with regard to certain IVs show up as insignificant or as significant but with very low Nagelkerke R Square values. Then when I lump many of the numerical and categorical IVs together as cofactors for a single test, the omnibus chi squared returns as highly significant, the Negelkerke r squared is much higher (0.3-0.5), overall predicated classification is high (60-80%), but only one or two of the Walds corresponding to the IVs are significant, with some of those that were significant in the univariate models losing significance completely.

Questions: How do I interpret these findings? Do low Nagelkerke R Square values and insignificant Omnibus chi square nullify binomial models regardless of the the Walds? What about classification prediction ratios? Perhaps none of these are relevant, but only the odds ratios? If IVs go from insignificant in univariate to significant in a multivariate model what does this tell us?

$\endgroup$
1
$\begingroup$

Scenario1: Neither of your proposed assessments of significance would be considered optimal for assessing the contributions of specific variables. You should look at the change in deviance (which is -2LL) that is associated with removal of predictors rather than looking at the "omnibus chi-square". If this were a planned experiment or pre-specfied analysis, then the difference in deviance should be compared to chi-squared distribution with the same number of degrees of freedom as the difference in in degrees of freedom between the full and reduced models.

If you are doing data dredging then to be honest, you should penalize this process by using higher numbers for your df in specifying the chi-squared critical value. You should keep track of the number of tests being run and be honest regarding whether this was "exploratory analysis" or was a "planned analysis".

The reporting of the exponentiated coefficients will depend on the model and the data. Cox models deliver hazard ratios and logistic regression deliver odds ratios. In most instances hazard ratios and risk ratios are very close and in some situations odds ratios may be very similar.

Scanario2: I generally find the Nagelkerke-R^2 virtually useless. All of the global scoring systems are very sensitive to the proportions of positive outcomes. In survival analysis where your event rates might be only 1/100 or 1/10,000, there is no way that you can use any of the various R^2's to assess the relative risks from individual or prespecified groups of predictors.

It's possible for combinations of factors to yield an jump in prediction or improvement in fit, but the situation you describe might be due to having a relatively small number of cases. It is really very difficult if not impossible to say what is happening is nsuch an instance without more specifics about the structure of the models. You would certainly want to do some validation with bootstrap estimates of optimism and calibration to see whether the predictions are accurate along the range of the linear predictors and whether the estimates are stable under controlled random perturbations of the data.

(I don't know what you mean by "classification prediction ratios".)

(These comments apply pretty much equally to logistic regression or Cox regression.)

$\endgroup$
  • $\begingroup$ Thanks so much for this answer, DWin. I forgot to return to this site to check for responses, so I apologize for the tardiness of this reply. $\endgroup$ – Edward Melamed Apr 28 '15 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.