I have an algorithm that makes 'informed guesses' about a set of inputs; the guess is either correct, incorrect, or missing (no guess made). From the correct/incorrect/missing guesses I can calculate an accuracy over some test set (for instance counting missing as incorrect). The guess has a confidence associated with it and I can thus plot for instance accuracy vs. confidence. As for a 'final result', is there a standard performance measure for this situation, like the AUC of the ROC curve? I would rather hew to standards and not (re)invent some ad-hoc measure here. It seems to me this is not a classification problem so I need something else. If I shoehorn this into classification (eg. by considering the threshold to be answering the classification problem, 'is this guess correct or not' ) then it seems I lose what I'm after (namely a higher score for more correct guesses.)
2 Answers
There are several measures you could use, but one option would be to measure accuracy via the root mean-squared-error (RMSE). So if you let $\mathbf{x} = (x_1, ..., x_n)$ be the true outcomes you are guessing (e.g., in the case where these are binary variables), and you let $\mathbf{y} = (y_1, ..., y_n)$ be your confidence values for a positive result, the RMSE would be:
$$\text{RMSE}(\mathbf{x}, \mathbf{y}) = \sqrt{ \tfrac{1}{n}||\mathbf{x}-\mathbf{y}||^2} = \sqrt{\frac{\sum_{i=1}^n (x_i-y_i)^2}{n} }.$$
In the case where you have more than two possible outcomes, you have confidence levels $y(x)$ for each outcome $x$ and you can aggregate the incorrect categories so that you have:
$$\text{RMSE}(\mathbf{x}, \mathbf{y}) = \sqrt{\frac{1}{n} \sum_{i=1}^n (x_i-y(x_i))^2 }.$$
-
$\begingroup$ my x are nominal not ordinal so I don't think this fits. E.g. I could be guessing favorite color based on year of birth, gender, and star sign. And unless I misunderstood your answer, you're not taking into account whether a given guess was right or not . If your x are 'was this guess right or not' then we're in the situation I mentioned in my question namely I lose the higher-score-for-more-correct-guesses aspect I need $\endgroup$ Commented Jun 25, 2018 at 11:11
-
$\begingroup$ I was thinking that if $x = 0,1$ is the true outcome, and $0 \leqslant y \leqslant 1$ is a predicted "confidence" of a positive outcome then $||x-y|| =0$ if you predict $y=1$ and get $x=1$ or if you predict $y=0$ and get $x=0$. That was the way I am interpreting it. $\endgroup$– BenCommented Jun 25, 2018 at 13:48
-
$\begingroup$ yeah that's again shoehorning the problem into being a classification problem, in which case the cross-entropy would be more appropriate, with the drawback again that what I care about is not only whether the confidence predicts correct guesses , but also my correct guess rate. if our only hammer is the classifier then everything looks like a classification problem; i may be after something other than a hammer $\endgroup$ Commented Jun 25, 2018 at 13:58
Predicting a result that is either correct or incorrect is pretty much what classification is. In classification you usually use the cross-entropy. You can also have a read here
Basically instead of making hard predictions where you only score if the most likely guess is correct or not, you predict the probability of each answer/class. This means that your guess is a Bernoulli distribution. You can then use the cross estimate to evaluate how close your Bernoulli distribution match the correct label.
Mathematically your loss will be $$ \mathrm{loss}= -\frac{1}{N}\sum_i^N\sum_j^M y_{ij}\log(p_j(x_i)), $$ where $i$ runs over all your $N$ samples, $j$ runs over all your $M$ classes, $y_{ij}$ is the one-hot encoding of sample $i$ and will be 1 if $j$ is the correct class and 0 otherwise and $p_j(x_i)$ is the predicted probability of class $j$ beign the correct one, given input $x_i$.
-
$\begingroup$ A classifier predicts a class from a quantized set of possible responses (the classes); in my case I am just guessing a string (for instance) out of a very large universe of possible strings, none of which necessarily will appear twice. That guess can be correct or not. However again if I treat the problem as a classification of whether my guess was right or not, then my metrics (e.g. AUC of ROC) won't take into account whether I had lots of right guesses or not , only whether my classifier has low confidence for bad guesses and high confidence for good guesses. $\endgroup$ Commented Jun 25, 2018 at 12:01
-
$\begingroup$ I think the only way to do this is an AUC analogous to the AUC of ROC but integrating e.g. the accuracy-confidence graph mentioned; that would take into account both confidences and number of correct guesses. However again this is ad-hoc and I was interested if there is some known measure $\endgroup$ Commented Jun 25, 2018 at 12:02
-
$\begingroup$ @jeremy_rutman I think the cross entropy should work as well in that case if you could translate the confidence to a probability. Then you can view it as a classification problem of only 2 classes, correct or not correct. Could you give an example of how your predictions and confidence metric looks like? $\endgroup$ Commented Jun 25, 2018 at 12:22
-
$\begingroup$ predictions:['gandalf','frodo','red','gollum','cheese'] $\endgroup$ Commented Jun 25, 2018 at 13:06
-
$\begingroup$ true_answers:['gandalf','bilbo','red','smeagull','cheese] . confidences:[0.9,0.2,0.8,0.1,0.9] correct_guess:[1,0,1,0,1]=predictions==true_answers . What i am after is a metric that maximizes not just cross entropy (ability of confidence to predict correct_guess) but also number of correct guesses (aka accuracy)) . $\endgroup$ Commented Jun 25, 2018 at 13:14