up vote 1 down vote favorite

I did a logistic regression with selected covariates on a dataset with about 10000 records and event rate of 10%. The cross validated c-index was 63% which admittedly is not very high. Looking at a calibration plot the model seemed well calibrated. However, the reduction in Brier score from a model without covariates was only about 3%. What can be the cause of the low reduction in Brier score?

A random forest on the same data gave a c-index of 59% and still a low reduction in Brier score compared to the logistic regression without covariates.

share|improve this question
    
I think the issue at hand here is how you are defining "low". Why do you say that a 3% reduction in brier score is "low", and a 63% c-index is "not very high"? These measurements are meant to be comparative, not absolute. Its possible that your data just does not support any better predictive power, and your models are doing the best the data can bear. – Matthew Drury Jul 12 at 20:29
    
Admittedly I don't have much experience with c-indexes and Brier scores reported in the literature, but naively I feel a 3% reduction in Brier score is low. With this thinking I would conclude that the model with covariates does not have good predictive power compared with a model without any covariates. Are you saying that the size of the reduction in Brier score is not a good measure predictive power? – L. Hansen Jul 12 at 21:17
    
No, I'm saying something different, that there is no absolute measure of predictive power. Only comparative statements like "model A has more predictive power than model B for predicting phenomena P" are meaningful. The c-index and brier score are red-herrings, this is true for any measure of model predictive power. If you only have one model, there is no scientific meaning to low and high predictive power. – Matthew Drury Jul 12 at 21:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.