Can we interpret posterior probability obtained from a classifier that outputs a predicted class value and a probability (for example, logistic regression or Naive Bayes) as some kind of a confidence score that is assigned to that predicted class value?
Tell me more
×
Cross Validated is a question and answer site for
statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
|
|
||||
|
|
|
As other answers correctly state, the reported probabilities from models such as logistic regression and naive Bayes are estimates of the class probability. If the model were true, the probability would indeed be the probability of a correct classification. However, it is quite important to understand that this could be misleading because the model is estimated and thus not a correct model. There are at least three issues.
The uncertainty is just the everywhere present fact that the probability is just an estimate. A confidence interval of the estimated class probability could provide some idea about the uncertainty (of the class probability, not the classification). If the model is wrong $-$ and face it, it is $-$ the class probabilities can be quite misleading even if the class predictions are good. Logistic regression can get the class probabilities wrong for two fairly well separated classes if some data points are a little extreme. It might still do a fine job in terms of classification. If the estimation procedure (intentionally) provides a biased estimate, the class probabilities are wrong. This is something I see with regularization methods like lasso and ridge for logistic regression. While a cross-validated choice of the regularization leads to a model with good performance in terms of classification, the resulting class probabilities are clearly underestimated (too close to 0.5) on test cases. This is not necessarily bad, but important to be aware of. |
|||
|
|
|
For a test case (particular input), its class (say the label 1 for the binary output) predictive probability is the chance the test example belongs to that class. Over many such test cases, the proportion that belong class 1 will tend to the predictive probability. Confidence has connotations of confidence intervals, which are something quite different. |
||||
|
|
|
If a classifier predicts a certain class with a probability, that number can be used as a proxy for the degree of confidence in that classification. Not to be confused with confidence intervals. For example if classifier P predicts two cases as +1 & -1 with probability 80% & 60% then it is correct to say that it is more sure of the +1 classification than the -1 classification. The variance as measured by p(1-p) is also a good measure of uncertainty. Note, the baseline confidence is 50% not 0. |
|||
|
|
|
Given a classifier with 2-classes (e.g. a 2 class linear discriminant or logistic regression classifier) the discriminant value for both classes can be applied to a softmax function to yield an estimate of the posterior probability for that class: P1 = exp(d1)/(exp(d1) + exp(d2)) Where P1 is the posterior probability estimate for class 1, d1 and d2 are discriminant values for classes 1 and 2 respectively. In this case the estimated posterior probability for a given class can be taken as a degree of confidence in the class, for a given case as P1 will equal 1 - P2. |
|||||||||||||||||
|
