# Importance of C statistic in logistic regression

I am working on a dataset with 1500 cancer patients. The research question is impact of various patient and surgery related variables on disease specific mortality. So the outcome/dependent variable is a binary. I ran logistic regression to assess effects of variables on outcome. The variables were chosen after univariate analysis based on the p value <0.25. Hosmer-lemeshow test confirmed goodness of fit of the model. But on calculating the C statistic the AUC was 0.65. Can both these tests contradict each other? If so, then what conclusion can i derive about the predictive ability of the model?

• Where do you see a contradiction? Commented Feb 18, 2019 at 14:43
• I thought AUC of 0.65 meant that the predictve ability of the model is poor. Though i understand that there is no specific cut off for it. Commented Feb 18, 2019 at 14:49
• Well, there is one meaningful threshold: an AUC of 0.5 means that your model is no better than chance (i.e. no better than flipping a coin to make you predictions). Anything that's (reliably) above that means your model has some predictive ability. Whether or not you're satisfied with that ability is another question that depends on the use case. So an AUC of 0.65 may not be fantastic but it wouldn't contradict another test that said the model did better than chance-level. Commented Feb 18, 2019 at 14:56
• Many Thanks for you explanation. That was helpful. Commented Feb 18, 2019 at 14:59

There are a few issues to consider here.

First, if you are analyzing survival data, then modeling with logistic regression is throwing away all the information you have about the relative timing of events and can be biased if individuals who might have died from disease simply hadn't done so by the end of data collection. Consider carefully whether that is what you want to do. Otherwise, use a form of survival analysis.

Second, your use of single-predictor relations to outcome for selection of predictors can lead you astray. This will omit predictors that might show relations to outcome when other predictors are taken into account and may include multiple highly correlated predictors that have similar relations to outcome. This is particularly an issue in logistic regression, which has an inherent omitted variable bias even if the omitted variables are uncorrelated to the retained variables. This issue also holds if you switch to survival analysis from logistic regression. It's also possible that you need to incorporate non-linear relations of predictors to log-odds of outcome or to include interactions among predictors to get a good model.

Third, the Hosmer-Lemeshow test has some limitations. See this page for further discussion.

Fourth, the AUC reports the model's ability to distinguish the classes based on the linear predictor from the logistic regression. It's the fraction of pairs of comparable cases for which the class membership is in the correct order with respect to the corresponding linear predictor values. Whether 0.65 (65% of pairs in the correct order) is a poor or a good value will depend on the field of interest.

Fifth, you might want to consider a different measure of model quality. The Brier score, for example, is a proper scoring rule that is optimized by a model having the correct probability distribution. It's related to the mean-square error between predicted probabilities and actual class membership, and unlike AUC it generalizes easily to more than 2 classes. This page and its many links is one good introduction to the underlying issues. Although that page is in the context of survival analysis, the issues are the same for logistic regression.

Modeling this type of data can be tricky. Make sure to develop a strong background in all the aspects of this modeling, from data collection to handling of missing data to choice of predictors to calibration and validation, such as provided by Harrell's course notes and book.