5
$\begingroup$

Let's say we have a simple binary classification problem. So for a predictor X we want to predict response Y. Y is binary, so either 0 or 1. Now let's say we use two different classifiers, model1 and model2. While predicting a new data point x_i, model1 predicts with a probability of 0.9 that y_i = 1, while model2 says with a probability of 0.6 that y_i = 1. So if in reality y_i = 0, both models result in the same wrong label. This means that normal stats such as overall accuracy, kappa etc. will be the same for both models. Yet intuitively I feel like model1 is less accurate since it was more sure about its wrong prediction.

Are there some other classifier performance metrics that actually take this into account? It makes little sense to me that whether a prediction is 0.51 or 1 does not change classifier performance as long as the labels stay the same.

Improve this question
$\endgroup$
5
  • $\begingroup$ If y_i = 0 it does not mean that either model is wrong. Event with probability 0.9 sometimes do not happen, ten percent of the time in fact! $\endgroup$
    – Matthew Drury
    Commented Jun 9, 2017 at 14:26
  • $\begingroup$ thx for remark, it was poorly written, i meant wrong label $\endgroup$
    – Leander Moesinger
    Commented Jun 9, 2017 at 14:32
  • $\begingroup$ The point I believe was that a probabilistic cannot really be "wrong" on a single example (unless $p$ is 0 or 1; e.g. this is implicit in the log-loss formula below). $\endgroup$
    – GeoMatt22
    Commented Jun 9, 2017 at 15:51
  • 1
    $\begingroup$ @GeoMatt22 yeah, I understand that - but while a model can not be wrong (unless p = 1 or 1, as you said), a label most certainly can ;) The original text contained some blunders. $\endgroup$
    – Leander Moesinger
    Commented Jun 9, 2017 at 15:58
  • 1
    $\begingroup$ There is a simple rule of thumb that leads to log likelihood as an auxiliary classifier performance measure: you want your classifier to be confident about its decisions. $\endgroup$
    – g3o2
    Commented Jun 9, 2017 at 18:06

2 Answers 2

Reset to default
10
$\begingroup$

Classifier metrics that compare the predicted probabilities to the true classes go by the name of proper scoring rules. The two most popular are the log-loss

$$ L = \sum_i y_i \log(p_i) + (1 - y_i) \log(1 - p_i) $$

and the brier score

$$ L = \sum_i (y_i - p_i)^2 $$

The log-loss is used more in practice, as it is the log likelihood of the Bernoulli distribution.

It is good practice to fit and compare models using proper scoring rules, as this ensures your predicted probabilities are fit well and calibrated to the data. Once you have a well fit probability model, it can be used to answer a multitude of questions that cannot be answered with only class assignments.

Additionally, the AUC is a popular metric. It is not a proper scoring rule, but it can be used to evaluate any probabilistic classifier in terms of an average performance across a range of hard classification thresholds. The AUC is the probability that a randomly chosen true positive class receives a greater predicted probability than a randomly chosen negative class.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Exactly what I was looking for. Thx a lot! $\endgroup$
    – Leander Moesinger
    Commented Jun 9, 2017 at 15:37
  • $\begingroup$ @Matthew Drury RE: "It is good practice to fit and compare models using proper scoring rules, as this ensures your predicted probabilities are fit well and calibrated to the data.", do you happen to have a reference for this? (I am not trying to contradict you and only want to read more on the topic.) $\endgroup$
    – darXider
    Commented Jun 9, 2017 at 15:46
  • 2
    $\begingroup$ Yup. Check out Frank Harrell's "Regression Modeling Strategies". $\endgroup$
    – Matthew Drury
    Commented Jun 9, 2017 at 15:48
0
$\begingroup$

It might be the case that one model (say M1 on your case) leads to more extreme predictions compared to the other (M2), meaning that "certainty" (I think this concept is misleading) will be also higher for correctly predicted/classified events. Instead of reporting % of correctly classified events, you could simply compute average predicted proba over the sample and compare this stat between the 2 models. However I would not try to over-interpret the diff in predicted proba. What really matters (in terms of accuracy) is whether events are correctly predicted/classified or not. The predicted proba are not necessarily 100% meaningful. For example if you use Normal errors (instead of Logistic ones) and then estimate a Probit model, you would obtain diff predicted proba.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ I disagree. Its best to model binary outcomes with probability models based on proper scoring rules, this is the consesus on this site and in the literature. Hard classification should be used in response to business or problem objectives, and the had classification should be done in response to the predictions from a probabilistic model. The quality of the classification can be evaluated based on business objectives, not an arbitrary metric like accuracy. $\endgroup$
    – Matthew Drury
    Commented Jun 9, 2017 at 14:57
  • $\begingroup$ The discussion here may also be of interest. $\endgroup$
    – GeoMatt22
    Commented Jun 9, 2017 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.