Classification accuracy based on probability Let's say we have a simple binary classification problem. So for a predictor X we want to predict response Y. Y is binary, so either 0 or 1. Now let's say we use two different classifiers, model1 and model2. While predicting a new data point x_i, model1 predicts with a probability of 0.9 that y_i = 1, while model2 says with a probability of 0.6 that y_i = 1. So if in reality y_i = 0, both models result in the same wrong label. This means that normal stats such as overall accuracy, kappa etc. will be the same for both models. Yet intuitively I feel like model1 is less accurate since it was more sure about its wrong prediction.
Are there some other classifier performance metrics that actually take this into account? It makes little sense to me that whether a prediction is 0.51 or 1 does not change classifier performance as long as the labels stay the same.
 A: Classifier metrics that compare the predicted probabilities to the true classes go by the name of proper scoring rules.  The two most popular are the log-loss
$$ L = \sum_i y_i \log(p_i) + (1 - y_i) \log(1 - p_i) $$
and the brier score
$$ L = \sum_i (y_i - p_i)^2 $$
The log-loss is used more in practice, as it is the log likelihood of the Bernoulli distribution.
It is good practice to fit and compare models using proper scoring rules, as this ensures your predicted probabilities are fit well and calibrated to the data.  Once you have a well fit probability model, it can be used to answer a multitude of questions that cannot be answered with only class assignments.
Additionally, the AUC is a popular metric. It is not a proper scoring rule, but it can be used to evaluate any probabilistic classifier in terms of an average performance across a range of hard classification thresholds.  The AUC is the probability that a randomly chosen true positive class receives a greater predicted probability than a randomly chosen negative class.
A: It might be the case that one model (say M1 on your case) leads to more extreme predictions compared to the other (M2), meaning that "certainty" (I think this concept is misleading) will be also higher for correctly predicted/classified events. Instead of reporting % of correctly classified events, you could simply compute average predicted proba over the sample and compare this stat between the 2 models. However I would not try to over-interpret the diff in predicted proba. What really matters (in terms of accuracy) is whether events are correctly predicted/classified or not. The predicted proba are not necessarily 100% meaningful. For example if you use Normal errors (instead of Logistic ones) and then estimate a Probit model, you would obtain diff predicted proba.
