14
$\begingroup$

Can we interpret posterior probability obtained from a classifier that outputs a predicted class value and a probability (for example, logistic regression or Naive Bayes) as some kind of a confidence score that is assigned to that predicted class value?

$\endgroup$
8
$\begingroup$

As other answers correctly state, the reported probabilities from models such as logistic regression and naive Bayes are estimates of the class probability. If the model were true, the probability would indeed be the probability of a correct classification.

However, it is quite important to understand that this could be misleading because the model is estimated and thus not a correct model. There are at least three issues.

  • Uncertainty of estimates.
  • Model misspecification.
  • Bias.

The uncertainty is just the everywhere present fact that the probability is just an estimate. A confidence interval of the estimated class probability could provide some idea about the uncertainty (of the class probability, not the classification).

If the model is wrong $-$ and face it, it is $-$ the class probabilities can be quite misleading even if the class predictions are good. Logistic regression can get the class probabilities wrong for two fairly well separated classes if some data points are a little extreme. It might still do a fine job in terms of classification.

If the estimation procedure (intentionally) provides a biased estimate, the class probabilities are wrong. This is something I see with regularization methods like lasso and ridge for logistic regression. While a cross-validated choice of the regularization leads to a model with good performance in terms of classification, the resulting class probabilities are clearly underestimated (too close to 0.5) on test cases. This is not necessarily bad, but important to be aware of.

$\endgroup$
2
$\begingroup$

For a test case (particular input), its class (say the label 1 for the binary output) predictive probability is the chance the test example belongs to that class. Over many such test cases, the proportion that belong class 1 will tend to the predictive probability. Confidence has connotations of confidence intervals, which are something quite different.

$\endgroup$
2
$\begingroup$

Given a classifier with 2-classes (e.g. a 2 class linear discriminant or logistic regression classifier) the discriminant value for both classes can be applied to a softmax function to yield an estimate of the posterior probability for that class:

P1 = exp(d1)/(exp(d1) + exp(d2))

Where P1 is the posterior probability estimate for class 1, d1 and d2 are discriminant values for classes 1 and 2 respectively. In this case the estimated posterior probability for a given class can be taken as a degree of confidence in the class, for a given case as P1 will equal 1 - P2.

$\endgroup$
7
  • 1
    $\begingroup$ This answer seems to equate "probability" with "confidence" whereas @Yoda's answer (correctly) distinguishes the two. $\endgroup$
    – whuber
    Aug 22 '12 at 16:46
  • $\begingroup$ @whuber I think in general terms confidence can be viewed as strength of belief. In that way it is like a probability. Confidence and confidence interval are two different things. However even for the term confidence interval the confidence level is a coverage probability for the random interval. $\endgroup$ Aug 22 '12 at 20:52
  • $\begingroup$ I do not disagree with your remarks, @Michael, in the sense that a term like "confidence score" could mean almost anything (but maybe its use should be deprecated for exactly that reason). Yet in what sense is the value fit by logistic regression a "coverage probability"? Does your proposed use of "confidence" as strength of belief make it synonymous with subjective "probability," or is some distinction still maintained? (If so, what?) $\endgroup$
    – whuber
    Aug 22 '12 at 21:01
  • 1
    $\begingroup$ @whuber I think you are going far deeper into this than I intended with my remark. I just want to say that just because we commonly connect the word "confidence" with confidence interval, it does not mean that the OPs term confidence score could not be used to mean a probability (maybe like the Bayesian view of probability as a subjective level of belief but not necessarily). $\endgroup$ Aug 22 '12 at 21:34
  • 1
    $\begingroup$ @whuber, I was indeed referring to confidence in the class label in the sense of 'strength of belief', i.e. the larger the posterior probability value for a given class, the more confidence you have in the predicted class label. Am happy to remove this answer however. $\endgroup$
    – BGreene
    Aug 23 '12 at 8:17
1
$\begingroup$

If a classifier predicts a certain class with a probability, that number can be used as a proxy for the degree of confidence in that classification. Not to be confused with confidence intervals. For example if classifier P predicts two cases as +1 & -1 with probability 80% & 60% then it is correct to say that it is more sure of the +1 classification than the -1 classification. The variance as measured by p(1-p) is also a good measure of uncertainty. Note, the baseline confidence is 50% not 0.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.