4
$\begingroup$

I have a logit model and am trying to understand and compare the predicted and observed values generated by the model. Let's say data set had 100 values and I generate all the predicted probabilities, and then I find the actual probabilities from the data set.

If I'm comparing the predicted vs observed values, I'm thinking there are two ways to do it. One is to do it value by value, while the second would be to group by the 'predicted probabilities.'

Method 1:

x_value  pred_val   obs_val
100       0.30       0.34
102       0.33       0.36
104       0.35       0.37
106       0.40       0.40
...

I'm also thinking there has to some way to aggregate these values. So I'm thinking of aggregating all x values where the predicted probabilities is between 10 to 20% percent, then find the avg predicted value from that range, followed by the predicted value for that range.

Method 2:

Pred_probs       pred_val    obs_val
10 to 20% vals     0.10        0.11
21 to 30% vals     0.12        0.16
31 to 50% vals     0.15        0.30

What I'm wondering is:

  1. When there are a large number of data points, what use is having a list of the predicted and observed values for any given value of x?

  2. Does it ever make sense to do something as identified in 'Method 2'?

$\endgroup$
  • 4
    $\begingroup$ I am having a philosophical difficulty understanding what "actual" probabilities might mean, given that in any correct application of logistic regression all the response values are either $0$ or $1$. Are these frequencies? Of what, precisely? (I am reminded of goodness of fit techniques in which a moving average of the response can be compared to the predicted probabilities.) $\endgroup$ – whuber Jan 29 '13 at 6:48
  • 2
    $\begingroup$ @whuber makes a good point. For comparing predicted probabilities to the actual observed outcomes (0 or 1), there are existing methods for considering whether a model (i.e. the predictions) does a good job of discriminating between those with/without the outcome (i.e. the observed data). Somer's D, and the (equivalent in meaningful content) c-statistic both spring to mind as summary statistics that might be of use? $\endgroup$ – James Stanley Jan 29 '13 at 7:20
  • $\begingroup$ See Agresti's book Binary and Categorical Data or Steyerberg book on Clinical Prediction Models $\endgroup$ – user21617 Mar 6 '13 at 11:38
4
$\begingroup$

It sounds as if you are wanting to check the calibration of a model on the same dataset used to build the model. This will require the use of the bootstrap to re-fit the model 300 times. You can use a bootstrap overfitting-corrected nonparametric calibration curve with a nonparametric smoother. It is not a good idea to bin predicted probabilities. Assuming you did no variable selection here's an approach in R with the rms package.

require(rms)
f <- lrm(y ~ x1 + x2 + x3, x=TRUE, y=TRUE)  # Full pre-specified model
validate(f, B=300)  # bootstrap stats such as Somers' Dxy
cal <- calibrate(f, B=300)
plot(cal)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.