I have a trained logistic regression model that I am applying to a testing data set. The dependent variable is binary (boolean). For each sample in the testing data set, I apply the logistic regression model to generates a % probability that the dependent variable will be true. Then I record whether the acutal value was true or false. I'm trying to calculate an $R^2$ or Adjusted $R^2$ figure as in a linear regression model.
This gives me a record for each sample in the testing set like:
prob_value_is_true acutal_value .34 0 .45 1 .11 0 .84 0 .... ....
I am wondering how to test the accuracy of the model. My first attempt was to use a contingency table and say "if
prob_value_is_true > 0.80, guess that the actual value is true" and then measure the ratio of correct to incorrect classifications. But I don't like that, because it feels more like I'm just evaluating the 0.80 as a boundary, not the accuracy of the model as a whole and at all
Then I tried to just look at each prob_value_is_true discrete value, as an example, looking at all samples where
prob_value_is_true=0.34 and measuring the % of those samples where the acutal value is true (in this case, perfect accuracy would be if the % of samples that was true = 34%). I might create a model accuracy score by summing the difference at each discrete value of
prob_value_is_true. But sample sizes are a huge concern here, especially for the extremes (nearing 0% or 100%), such that the averages of the acutal values are not accurate, so using them to measure the model accuracy doesn't seem right.
I even tried creating huge ranges to ensure sufficient sample sizes (0-.25, .25-.50, .50-.75, .75-1.0), but how to measure "goodness" of that % of actual value stumps me. Say all samples where
prob_value_is_true is between 0.25 and 0.50 have an average
acutal_value of 0.45. Is that good since its in the range? Bad since its not near 37.5% (the center of the range)?
So I'm stuck at what seems like should be an easy question, and hoping someone can point me to a resource or method to calculate an accuracy stastic for a logistic regression model.