Logistic regression: big difference in predicted values and highly significant, but poor goodness of fit

Question

I have a logistic regression model with a dichotomous response variable and predictors coded from $1$ to $10$ and from $0$ to $18$.

When I fit the model, I get these results:

Intercept     2.467 (p=2e-16)
Predictor#1  -0.181 (p=1.76e-07)
Predictor#2  -0.098 (p=3.34e-14)

Null deviance 2252.3 on 1741 DF
Residual dev. 2113.1 on 1739 DF
(1276 obs. deleted due to missingness) — AIC: 2119.1

And a Nagelkerke $R^2$ of $0.10$. The $R^2$ and the Deviance show the model as a very bad one, but when I calculate the predicted values I get a possibility of $24.8\%$ when both predictors are $1$, and of $89.9\%$ when the predictors are at their maximum values ($10$ and $18$, respectively).

How can I get such bad measures of the goodness of fit, and, at the same time, get such a big difference (statistical significance, $p<0.01$) in the predicted values?

Answer 1

Given that you have a large sample size (df = 1741) it is not surprising that you get highly significant results with a small effect size.

The two are answering different questions: The p value answers: "If, in the population from which this sample was drawn, there was really no effect at all, how likely is it that, in sample of this size, a test statistic as large or larger than the one we got would occur?"

The effect size is what it says - how big is the effect?

In a logistic regression, the most intuitive (at least to me) effect size measures are the odds ratios, rather than the parameter estimates.

Answer 2

In general, absolute goodness of fit measures are not available for models different than linear regression. If you obtain a significant p - value for some predictor this means that the model you are fitting is better that using the null model (no predictor, just the relative frequency of the outcome). This does not imply that you would be able to obtain a good prediction of the outcome.

In my personal opinion logistic regression models are good to rank subjects about their relative risk. The Gini index could be used to see how well the risk ranking works.

$normalizedGini <- function(aa, pp) {
    	.Gini <- function(a, p) {
    		if (length(a) !=  length(p)) stop("Actual and Predicted need to be equal lengths!")
    		temp.df <- data.frame(actual = a, pred = p, range=c(1:length(a)))
    		temp.df <- temp.df[order(-temp.df$pred, temp.df$range),]
    		population.delta <- 1 / length(a)
    		total.losses <- sum(a)
    		null.losses <- rep(population.delta, length(a)) # Hopefully is similar to accumulatedPopulationPercentageSum
    		accum.losses <- temp.df$actual / total.losses # Hopefully is similar to accumulatedLossPercentageSum
    		gini.sum <- cumsum(accum.losses - null.losses) # Not sure if this is having the same effect or not
    		sum(gini.sum) / length(a)
    	}
    	.Gini(aa,pp) / .Gini(aa,aa)
    } $