I'm using a regression model to classify some data (into two classes), which outputs a more-or-less opaque, continuous, monotonic score proportional to the likelihood of the instance falling into the positive class. I've done basic calibration on the score by binning instances by score and counting the proportion of positive instances in each score bin, so I know e.g. a score between 0.42 and 0.43 corresponds to a positive probability of 13%.
My question is how do I quantify the error in these probabilities? I'm interested in the confidence of a single probability prediction, not so much the overall quality of the model. I would like to be able to say something like, "given a score of 0.42, the probability of a positive is between 6% and 20%, 95% of the time." The problem is of course that any one instance is either positive or negative, not "13%." While I can treat negatives as 0 and positives as 1 and compute the "average" and "standard error" in each bin, I'm not sure these are very meaningful, as they only depend on the proportion of positives.
