I'm running a logistic regression with R using the glm() function with family = "binomial" and a very large number of observations (37208). Only a very small number of observations have a true result (approximately 1% of the 37208 observations).
In order to check model fit, I used pchisq() to generate a p-value based on Residual Deviance (see output below). This reports '1' indicating that is highly likely that the model is a good fit for the data.
However, looking at the predicted results, the fit doesn't seem all that 'good'. For example, only 53% of the observations predicted to have a true response (i.e. predicted probability > 0.5) actually had a true response (precision), and only 21% of the observations observed to have a true response were predicted (again, predicted probability > 0.5) to have a true response (recall). I experimented with lowering probably threshold from 0.5 to 0.3 - this improved recall but degraded precision. Either way, this seems to indicate the fit isn't that great.
How should I interpret the p-value from the residual deviance in this situation? Why is the p-value reported to be 1? Do the large number of observations combined with the low numbers of true responses somehow make the chi-square fit test misleading? If so, why?
(Dispersion parameter for binomial family taken to be 1) Null deviance: 5112.9 on 37207 degrees of freedom Residual deviance: 3329.5 on 37170 degrees of freedom AIC: 3405.5 Number of Fisher Scoring iterations: 17 > 1 - pchisq(3329.5, 37170)  1