6
$\begingroup$

$\newcommand{\P}{\mathbb{P}}$$\newcommand{\rank}{\operatorname{rank}}$I have a testing dataset with a binary response (0, 1) and two algorithms each estimating the probability of 1 for each trial. I am assuming responses are independent. I am wondering which metric should I use to compare the algorithms.

Let $\hat{p_i}=\P(y_i=1)$ and $\hat{y_i}$ represent predicted response (i.e. $\hat{y_i}=I(\hat{p_i} \ge 0.5)$). These are the metric choices I have found:

$$ \begin{array}{rcl} \text{Misclassification} & = & \frac{1}{n}\sum_{i=1}^n I(\hat{y_i} \ne y_i) \\ \text{AUC ROC} & = & \frac{S_0-n_0(n_0+1)/2}{n_0 n_1}, \end{array} $$ where: $$S_0 = \sum\limits_{i=1}^n \rank(\hat{p_i}) I(y_i=1), \quad n_0=\sum\limits_{i=1}^n I(y_i=0), \quad \text{and} \quad n_1=\sum\limits_{i=1}^n I(y_i=1),$$ $$ \begin{array}{rcl} \text{Entropy (Log-Loss)} & = & \displaystyle-\frac{1}{n}\sum\limits_{i=1}^n (y_i \log(\hat{p_i})+(1-y_i) \log(1-\hat{p_i}), \\ \text{Brier's Score (MSE)} & = & \displaystyle\frac{1}{n}\sum\limits_{i=1}^n (\hat{p_i} - y_i)^2. \end{array}$$

It seems the most popular metrics are misclassification and AUC ROC. However, misclassification ignores the probabilities and AUC ROC only considers the ranks of the probabilities. For example, consider the three estimates $\P(Y=1)=0.51$, $\P(Y=1)=0.52$, and $\P(Y=1)=0.99$. Misclassification is the same for these three estimates, while AUC ROC ignores that the first two predictions are nearly the same.

Entropy works well, but will give an infinite penalty if an algorithm assigns a 0% or 100% chance and gets it wrong (this prediction is common for tree algorithms). Brier's score doesn't have this issue, but is rarely used in comparison papers.

I am guessing all metrics are important in some way, but I have to pick only one. Which metric do you think I should typically use to compare the overall accuracy of the two algorithms? Why are misclassification and AUC ROC so often used in the literature? Are there any other metrics not mentioned I should consider?

$\endgroup$
4
  • $\begingroup$ What about Kendall rank correlation? $\endgroup$
    – Carl
    Nov 16, 2016 at 19:39
  • $\begingroup$ I think you need to give more information. For example, suppose the errors are strictly proportional to the magnitude of the measures. Then regressing $\text{ln}Y_i$ for $a_0+a_1\text{ln}X1_i+a_2\text{ln}X2_i$ using ANOVA and looking at the partial probabilities of contribution from the $a$'s would give an answer. So, I need a little more information about the variables. $\endgroup$
    – Carl
    Nov 16, 2016 at 19:55
  • $\begingroup$ isn't kendal rank the same as ROC? $\endgroup$
    – rep_ho
    Nov 17, 2016 at 16:00
  • $\begingroup$ I am not sure why Kendall rank correlation would be informative. I'm not trying to test whether these classifiers are similar (probably would have very high correlation) or not trying combine the classifiers. Instead, I am trying to determine which classifier is better. For example, suppose classifier A assigns a 40% chance Y=1, while classifier B assigns a 60% chance. If Y=1 is observed, then clearly classifier B is better. My goal is simply to determine the best metric to report. $\endgroup$ Nov 18, 2016 at 8:16

2 Answers 2

3
+50
$\begingroup$

Why use accuracy? Accuracy metric (what you call misclassification) is horrible, don't use it. However, there are reason why it is used.

  1. Sometimes, that's what you care about. It doesn't matter if the spam filter, missclassify 5 spam emails by one percent or by 40 percent, you will still get 5 spam emails. In this case it make sense to optimize your model with respect to some weighted accuracy measure.
  2. It's easy to interpret, especially for a lay person. I have model that predict if a percent has a cancer 90 percent times correctly, is easier to understand than the model has 93 AUROC. Sadly, this is exactly the case, where the error considering the probabilities would be more appropriate.

Why rank metrics (ROC) and not continuous metrics?

  1. Good ranks => good probabilities. Reasonable assumption is that if your model predicts the ranks good, then you can also make it to predict the probabilities good. If the ROC is good, but brier score is horrible. it is possible just to refit the models output with some monotone function and get the correct probabilities without much troubles. So basically, by not using those probability based measures, you just don't have bother with it.

  2. Outliers. Brier will square errors, entropy might go to infinity. Few extremely good hits or missclassifications, can move your whole model performance.

All metrics are arbitrary. There is nothing in universe telling you that missclassification by 15 percent is 2.25 times worse than missclassification by 10 percent (brier), or even that missclassification of a positive example has exactly the same weight as missclassification of a negative example. You should know what error/cost function you care about and optimize for it. E.g. I want to ear most money on a stock market, so I will use error where loss of 1 eur will give me error 1 and loss of 10 eur will give error 10 and not 10^2

Conclusion

Use error function that you wan't to minimize, if you don't have one, use ROC

$\endgroup$
0
$\begingroup$

I'd encourage you to think about loss. Each algorithm is outputting a distribution which represents its "bet" for a value of 0 or 1. What should its loss be for such a bet, given the actual value? What would your ideal algorithm minimize? Generally it would be expected loss, but it could be something else (e.g., loss at 95% confidence).

Entropy would treat this as an information model; i.e., how many bits would it cost the algorithm to output the actual value. Indeed, if it is completely wrong, it would cost an infinite number of bits. To avoid that you could use Jensen-Shannon Divergence. Note that in its generalized form, you could also include priors as part of it.

Interestingly, you ignored the option of a simple linear loss, $|\hat{p}_i - y_i|$, which would generally be the first thing to consider after misclassification which is 0/1 loss.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.