Classifier metrics that compare the predicted probabilities to the true classes go by the name of proper scoring rules. The two most popular are the log-loss

$$ L = \sum_i y_i \log(p_i) + (1 - y_i) \log(1 - pi) $$

and the brier score

$$ L = \sum_i (y_i - p_i)^2 $$

The log-loss is used more in practice, as it is the log likelihood of the Bernoulli distribution.

It is good practice to fit and compare models using proper scoring rules, as this ensures your predicted probabilities are fit well and calibrated to the data. Once you have a well fit probability model, it can be used to answer a multitude of questions that cannot be answered with only class assignments.

Classifier metrics that compare the predicted probabilities to the true classes go by the name of proper scoring rules. The two most popular are the log-loss

$$ L = \sum_i y_i \log(p_i) + (1 - y_i) \log(1 - p_i) $$

and the brier score

$$ L = \sum_i (y_i - p_i)^2 $$

The log-loss is used more in practice, as it is the log likelihood of the Bernoulli distribution.

It is good practice to fit and compare models using proper scoring rules, as this ensures your predicted probabilities are fit well and calibrated to the data. Once you have a well fit probability model, it can be used to answer a multitude of questions that cannot be answered with only class assignments.

Additionally, the AUC is a popular metric. It is not a proper scoring rule, but it can be used to evaluate any probabilistic classifier in terms of an average performance across a range of hard classification thresholds. The AUC is the probability that a randomly chosen true positive class receives a greater predicted probability than a randomly chosen negative class.