Performance measure for correct/incorrect guesses I have an algorithm that makes 'informed guesses' about a set of inputs; the guess is either correct, incorrect, or missing (no guess made). From the correct/incorrect/missing guesses I can calculate an accuracy over some test set (for instance counting missing as incorrect).   The guess has a confidence associated with it and I can thus plot for instance accuracy vs. confidence. As for a 'final result', is there a standard performance measure for this situation, like the AUC of the ROC curve? I would rather hew to standards and not (re)invent some ad-hoc measure here. It seems to me this is not a classification problem so I need something else. If I shoehorn this into classification (eg. by considering the threshold to be answering the classification problem, 'is this guess correct or not' ) then it seems I lose what I'm after (namely a higher score for more correct guesses.) 
 A: There are several measures you could use, but one option would be to measure accuracy via the root mean-squared-error (RMSE).  So if you let $\mathbf{x} = (x_1, ..., x_n)$ be the true outcomes you are guessing (e.g., in the case where these are binary variables), and you let $\mathbf{y} = (y_1, ..., y_n)$ be your confidence values for a positive result, the RMSE would be:
$$\text{RMSE}(\mathbf{x}, \mathbf{y}) = \sqrt{ \tfrac{1}{n}||\mathbf{x}-\mathbf{y}||^2} = \sqrt{\frac{\sum_{i=1}^n (x_i-y_i)^2}{n} }.$$
In the case where you have more than two possible outcomes, you have confidence levels $y(x)$ for each outcome $x$ and you can aggregate the incorrect categories so that you have:
$$\text{RMSE}(\mathbf{x}, \mathbf{y}) = \sqrt{\frac{1}{n} \sum_{i=1}^n (x_i-y(x_i))^2 }.$$
A: Predicting a result that is either correct or incorrect is pretty much what classification is. In classification you usually use the cross-entropy. You can also have a read here
Basically instead of making hard predictions where you only score if the most likely guess is correct or not, you predict the probability of each answer/class. This means that your guess is a Bernoulli distribution. You can then use the cross estimate to evaluate how close your Bernoulli distribution match the correct label.
Mathematically your loss will be
$$
\mathrm{loss}= -\frac{1}{N}\sum_i^N\sum_j^M y_{ij}\log(p_j(x_i)),
$$
where $i$ runs over all your $N$ samples, $j$ runs over all your $M$ classes, $y_{ij}$ is the one-hot encoding of sample $i$ and will be 1 if $j$ is the correct class and 0 otherwise and $p_j(x_i)$ is the predicted probability of class $j$ beign the correct one, given input $x_i$.
