Confidence measure for classification result from a MAP estimator

Question

I'm using a maximum a posteriori probability (MAP) estimator in a classification problem. After estimating all the a posteriori probability, the standard way is to simply take the class associated with the maximum probability. I would like to measure the confidence of this decision. I.e. if the maximum a posteriori probability is not significantly larger than the rest, I would like to skip the case (which would reduce recall but increase precision).

Is there a standard way to measure such concept?

Answer 1

Your posterior probabilities should add up to 1, should they not? In that case the maximum posterior probability would itself be your confidence measure.

Answer 2

Start by defining a single effectiveness measure for classification. You mention recall and precision, but those are two independent measures that typically need to be traded off. Once you've done that, one strategy is to optimize the expected value of the measure using the posterior probabilities of class membership.

Of course one needs to remember that the posterior probabilities you get out of a MAP estimator are not necessarily correct!