I am doing a system for classifiying documents. The project demands the use of probability based output. So a sample will have a probability for belonging to each class.
For now I use logistic regression, but this could be subject to change. So I don't want to do an R^2 approximation. I also don't want to use standard metrics for classification like F-measure because it doesn't work with probabiliies.
I have no idea what is the custom metric in this situation. Any ideas?
There are actually two things to evaluate on probability:probability-based ranking performance and probability estimation performance.
Common evaluation methods for probability-based ranking is the Area Under ROC curve (AUROC). This measure has been developed to 2-class problems but can also be extended to multi-class problems, for examle have a look to [1] in order to understand ROC analysis; but there are easier methods to calculate ROC curve, see [2] related to Probability Estimation Trees (PETs).
Similarly, Brier Score, also known as Mean Square Error, is suited to evaluate the probability estimation accuracy performance. It has been shown that the Brier Score can be decomposed in Calibration and Refinement [3]. These measures are suited for PETs, but you can reproduce them discretizing your probability into buckets. The Calibration component captures how well the PET represents the true distribution of the data; while Refinement component captures how much the model discriminate between classes. In particular, the Calibration measure has an intuitive graphical interpretation as the Reliability Plot, which shows record subset probabilities on the training data and the corresponding probabilities on the test data. Refinement measure has its graphical transposition too, called Sharpness Histogram. See [4] for 2-class problems, in the firsts paragraphs Brier Score, Calibration and Refinement are introduced. They also use Negative Cross Entropy, that is similar to Brier Score. I think it should be easy to find estension for multi-class problems.
[1] T. Fawcett, \An introduction to roc analysis," Pattern Recogn. Lett., vol. 27, no. 8, pp. 861{874, 2006.
[2] N. Chu, L. Ma, P. Liu, Y. Hu, and M. Zhou, \A comparative analysis of methods for probability estimation tree," W. Trans. on Comp., vol. 10, pp. 71{80, March 2011.
[3] G. Blattenberger and F. Lad, \Separating the Brier score into calibration and refinement components: A graphical exposition," vol. 39, pp. 26{32, 1985.
[4] K. Zhang, W. Fan, B. Buckles, X. Yuan, and Z. Xu, \Discovering unrevealed properties of probability estimation trees: On algorithm selection and performance explanation," Data Mining, IEEE International Conference on, vol. 0, pp. 741{752, 2006.
The classic error metric for probabilistic classifiers is the cross-entropy, for a two class classifier it is
$L = -\sum_{i=1}^n t_i\log(y_i) + (1-t_i)\log(1-y_i)$
where there are n test patterns, $t_i \in [0,1]$ is the target for the $i^{th}$ test pattern and $y_i$ is the outoput of the model for the $i^{th}$ test pattern.
The cross-entropy is the negative log-likelihood used in fitting a logistic regression model (or kernel logistic regression or neural networks etc.) so it is fairly natural to use it as the test metric as well. Unlike the AUROC it takes the calibration of the probabilities into account rather than just the ranking (which may or may not be important depending on the application), but it goes off to infinity if the classifier gets the answer wrong with very high confidence.
A closely related metric is the mean predictive information, which for a two class problem is
$I = \frac{1}{n}\sum_{i=1}^n\left[t_i.*log_2(y_i) + (1-t_i).*log_2(1-y_i)\right]+1$
which is very similar, but conveniently gives result that normally lies between 0 and 1 bits, which is more easily interpretable than the cross-entropy
