Let's say we have a simple binary classification problem. So for a predictor X we want to predict response Y. Y is binary, so either 0 or 1. Now let's say we use two different classifiers, model1 and model2. While predicting a new data point x_i, model1 predicts with a probability of 0.9 that y_i = 1, while model2 says with a probability of 0.6 that y_i = 1. So if in reality y_i = 0, both models result in the same wrong label. This means that normal stats such as overall accuracy, kappa etc. will be the same for both models. Yet intuitively I feel like model1 is less accurate since it was more sure about its wrong prediction.
Are there some other classifier performance metrics that actually take this into account? It makes little sense to me that whether a prediction is 0.51 or 1 does not change classifier performance as long as the labels stay the same.