Understanding how good a prediction is, in logistic regression

I have fit a logistic regression model with four features and now I am able to use it for prediction.

When I plot the data points from my learning sample (i.e. I select two features, plot them in two axis), I see that there is more density of points in an area than in other areas. What I would like my model to return is not only the probabilities, but also a number that tells me how good these probabilities are (e.g. a p-value). I imagine that depending on the region the prediction is made, and the density of the sample points in that concrete region, the better the prediction, and conversely.

Is there a way of obtaining a value that tells me how good my prediction is?

In concrete, I am using python scikit logistic regression package, although the answer can be more broad.

• The Brier score is appropriate for evaluating predictive accuracy... en.wikipedia.org/wiki/Brier_score – Mike Hunter Nov 1 '15 at 19:28
• It's not clear to me what's actually asked for. If you want a uncertainty measure for individual predictions, then standard errors or confidence intervals for the predicted probabilities (mean prediction) would provide that information. That's different from an overall goodness of fit measure, and Logit is fully parametric so it doesn't give local fit information. – Josef Nov 2 '15 at 14:34
• @user333700 thanks for your comment. I want uncertainty measure for individual prediction on this question. I have asked for goodness of fit in another question. – Escachator Nov 4 '15 at 11:46
• @DJohnson that won't help, because what I need is the potential error when I do the prediction... – Escachator Nov 9 '15 at 18:16

The key thing to check first is the model's calibration, either using the bootstrap to correct for overfitting or using a huge independent sample not used for model development or fitting. The best way to assess calibration is using a loess smooth nonparametric regression. Once you establish calibration you can go on to predictive discrimination using the pseudo $R^2$ and Somers' $D_{xy}$ rank correlation coefficient, or a simple translation of it to the $c$-index AKA concordance probability or AUROC. The Brier score is an excellent addition to all this.

• Hi @Frank. Thanks for your answer! Re calibration, I have already worked with the parameters, with an out of sample and test sets, if that is what you mean. The results are quite bad for any set of parameters, that is why I want to check how relevant the featurea are. What do you mean by a less smooth nonparametric regression? – Escachator Nov 2 '15 at 0:03
• That is loess, which is a flexible way to estimate a smooth relationship (here between continuous predicted probabilities $\hat{P}$ and observed binary $Y$). – Frank Harrell Nov 2 '15 at 12:10
• I think your method is more for goodness of fit than to know if a probability returned by my fitted model is good. I guess what I need is the confidence interval of that obtained probability, isn't it? Thanks! – Escachator Nov 9 '15 at 18:50
• The calibration curve is the way to check calibration-in-the-small, i.e., is a high-resolution estimate of the absolute accuracy of a set of probability forecasts. In that sense the calibration curve is both a measure of goodness-of-fit and a great way to check the accuracy of probabilities estimated by the model. – Frank Harrell Nov 9 '15 at 19:08

I may be wrong with exactly what it is that you are looking for, but if you are worried about the overall distribution of your prediction quality, then I would borrow from machine learning validation tools since this is exactly what they are interested in as well.

Here, you could do a 10-fold cross-validation (or a single hold out sample but you couldn't get information about the distribution of your metric over multiple samples of your data) of an area under the ROC curve metric (AUROC), for instance, if you would like to see how your predictions behave with different class cut-off probabilities.

Another metric could be the mean squared error if you prefer that loss function.

You can return r squared, p value and chi squared for a logistic regression. Try using stats models,instead of Sci kit. Stats models returns all of this data automatically