Estimating the error in calibrated probabilities

Question

I'm using a regression model to classify some data (into two classes), which outputs a more-or-less opaque, continuous, monotonic score proportional to the likelihood of the instance falling into the positive class. I've done basic calibration on the score by binning instances by score and counting the proportion of positive instances in each score bin, so I know e.g. a score between 0.42 and 0.43 corresponds to a positive probability of 13%.

My question is how do I quantify the error in these probabilities? I'm interested in the confidence of a single probability prediction, not so much the overall quality of the model. I would like to be able to say something like, "given a score of 0.42, the probability of a positive is between 6% and 20%, 95% of the time." The problem is of course that any one instance is either positive or negative, not "13%." While I can treat negatives as 0 and positives as 1 and compute the "average" and "standard error" in each bin, I'm not sure these are very meaningful, as they only depend on the proportion of positives.

Some related questions are 1, 2, 3.

Answer 1

A very simple solution would be to calculate a binomial confidence interval for the observations within each score bin. Naturally, if you'll increase the number of bins, you'll also increase your uncertainty regarding the probability of success within each bin, resulting with wider confidence intervals.

If you prefer a continuous measure, I guess that you'll have to choose and fit a model such as logistic regression. Then the confidence interval would have to be derived from the standard error estimates of the model's parameters.