1
$\begingroup$

I have fit a logistic regression model with four features and now I am able to use it for prediction.

When I plot the data points from my learning sample (i.e. I select two features, plot them in two axis), I see that there is more density of points in an area than in other areas. What I would like my model to return is not only the probabilities, but also a number that tells me how good these probabilities are (e.g. a p-value). I imagine that depending on the region the prediction is made, and the density of the sample points in that concrete region, the better the prediction, and conversely.

Is there a way of obtaining a value that tells me how good my prediction is?

In concrete, I am using python scikit logistic regression package, although the answer can be more broad.

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ The Brier score is appropriate for evaluating predictive accuracy... en.wikipedia.org/wiki/Brier_score $\endgroup$
    – user78229
    Commented Nov 1, 2015 at 19:28
  • 1
    $\begingroup$ It's not clear to me what's actually asked for. If you want a uncertainty measure for individual predictions, then standard errors or confidence intervals for the predicted probabilities (mean prediction) would provide that information. That's different from an overall goodness of fit measure, and Logit is fully parametric so it doesn't give local fit information. $\endgroup$
    – Josef
    Commented Nov 2, 2015 at 14:34
  • $\begingroup$ @user333700 thanks for your comment. I want uncertainty measure for individual prediction on this question. I have asked for goodness of fit in another question. $\endgroup$
    – Escachator
    Commented Nov 4, 2015 at 11:46
  • $\begingroup$ @DJohnson that won't help, because what I need is the potential error when I do the prediction... $\endgroup$
    – Escachator
    Commented Nov 9, 2015 at 18:16

3 Answers 3

Reset to default
4
$\begingroup$

The key thing to check first is the model's calibration, either using the bootstrap to correct for overfitting or using a huge independent sample not used for model development or fitting. The best way to assess calibration is using a loess smooth nonparametric regression. Once you establish calibration you can go on to predictive discrimination using the pseudo $R^2$ and Somers' $D_{xy}$ rank correlation coefficient, or a simple translation of it to the $c$-index AKA concordance probability or AUROC. The Brier score is an excellent addition to all this.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hi @Frank. Thanks for your answer! Re calibration, I have already worked with the parameters, with an out of sample and test sets, if that is what you mean. The results are quite bad for any set of parameters, that is why I want to check how relevant the featurea are. What do you mean by a less smooth nonparametric regression? $\endgroup$
    – Escachator
    Commented Nov 2, 2015 at 0:03
  • 1
    $\begingroup$ That is loess, which is a flexible way to estimate a smooth relationship (here between continuous predicted probabilities $\hat{P}$ and observed binary $Y$). $\endgroup$
    – Frank Harrell
    Commented Nov 2, 2015 at 12:10
  • $\begingroup$ I think your method is more for goodness of fit than to know if a probability returned by my fitted model is good. I guess what I need is the confidence interval of that obtained probability, isn't it? Thanks! $\endgroup$
    – Escachator
    Commented Nov 9, 2015 at 18:50
  • $\begingroup$ The calibration curve is the way to check calibration-in-the-small, i.e., is a high-resolution estimate of the absolute accuracy of a set of probability forecasts. In that sense the calibration curve is both a measure of goodness-of-fit and a great way to check the accuracy of probabilities estimated by the model. $\endgroup$
    – Frank Harrell
    Commented Nov 9, 2015 at 19:08
1
$\begingroup$

I may be wrong with exactly what it is that you are looking for, but if you are worried about the overall distribution of your prediction quality, then I would borrow from machine learning validation tools since this is exactly what they are interested in as well.

Here, you could do a 10-fold cross-validation (or a single hold out sample but you couldn't get information about the distribution of your metric over multiple samples of your data) of an area under the ROC curve metric (AUROC), for instance, if you would like to see how your predictions behave with different class cut-off probabilities.

Another metric could be the mean squared error if you prefer that loss function.

Improve this answer
$\endgroup$
-1
$\begingroup$

You can return r squared, p value and chi squared for a logistic regression. Try using stats models,instead of Sci kit. Stats models returns all of this data automatically

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.