Importance of C statistic in logistic regression

Question

I am working on a dataset with 1500 cancer patients. The research question is impact of various patient and surgery related variables on disease specific mortality. So the outcome/dependent variable is a binary. I ran logistic regression to assess effects of variables on outcome. The variables were chosen after univariate analysis based on the p value <0.25. Hosmer-lemeshow test confirmed goodness of fit of the model. But on calculating the C statistic the AUC was 0.65. Can both these tests contradict each other? If so, then what conclusion can i derive about the predictive ability of the model?

Answer 1

There are a few issues to consider here.

First, if you are analyzing survival data, then modeling with logistic regression is throwing away all the information you have about the relative timing of events and can be biased if individuals who might have died from disease simply hadn't done so by the end of data collection. Consider carefully whether that is what you want to do. Otherwise, use a form of survival analysis.

Second, your use of single-predictor relations to outcome for selection of predictors can lead you astray. This will omit predictors that might show relations to outcome when other predictors are taken into account and may include multiple highly correlated predictors that have similar relations to outcome. This is particularly an issue in logistic regression, which has an inherent omitted variable bias even if the omitted variables are uncorrelated to the retained variables. This issue also holds if you switch to survival analysis from logistic regression. It's also possible that you need to incorporate non-linear relations of predictors to log-odds of outcome or to include interactions among predictors to get a good model.

Third, the Hosmer-Lemeshow test has some limitations. See this page for further discussion.

Fourth, the AUC reports the model's ability to distinguish the classes based on the linear predictor from the logistic regression. It's the fraction of pairs of comparable cases for which the class membership is in the correct order with respect to the corresponding linear predictor values. Whether 0.65 (65% of pairs in the correct order) is a poor or a good value will depend on the field of interest.

Fifth, you might want to consider a different measure of model quality. The Brier score, for example, is a proper scoring rule that is optimized by a model having the correct probability distribution. It's related to the mean-square error between predicted probabilities and actual class membership, and unlike AUC it generalizes easily to more than 2 classes. This page and its many links is one good introduction to the underlying issues. Although that page is in the context of survival analysis, the issues are the same for logistic regression.

Modeling this type of data can be tricky. Make sure to develop a strong background in all the aspects of this modeling, from data collection to handling of missing data to choice of predictors to calibration and validation, such as provided by Harrell's course notes and book.

Added in response to comment:

Part of what you would learn from the Harrell references or other sources is that, for survival analysis, you can reasonably consider 1 predictor for every 15 or so events. With 1500 patients and, say, 30% (450) deaths you could evaluate up to 30 predictors in a Cox multiple regression. That should be enough to include most standard clinical predictors if you use your knowledge of the subject matter to make reasonable choices. (Note, however, that in this respect multi-level categorical variables count as 1 predictor for every level beyond the first, each additional parameter fit in non-linear modeling of a continuous independent variable counts as a predictor, and an interaction term between two variables adds the product of the number of predictors contributed by each variable.)

If you have a very large number of potential predictors (e.g., thousands of gene-expression values) then to avoid overfitting and to provide usefulness of your model outside of the original sample you need to use a method that avoids data dredging. Simple stepwise selection of predictors can introduce many problems. LASSO provides a principled way to select a small number of predictors whose regression coefficients are penalized to prevent overfitting. Ridge regression keeps all the predictors in the model but penalizes all of their coefficients. This page provides an introduction to discussion of the relative strengths of these two approaches.