4
$\begingroup$

I understand the fundamentals in the decision theory about accuracy being an improper scoring rule compared to other proper scoring rules like Brier score & log loss. And, that threshold setting in binary outcomes is highly subjective.

To give some background, the following questions actually emerged initially from my previous question about setting thresholds for binary predictions of a fire occurring. Here, I did not know the cost of false-positive and false-negative in case of fire and was, therefore, advised to use proper scoring rules. I understand one should use these proper scoring rules when one does not know the cost of misclassifying. But in my head, utilizing a proper scoring rule does not change the fact that there still is a probability of misclassifying a fire as no fire and vice versa, for instance.

(1) So, how can one be sure/argue that 1's are more likely to be predicted as 1's and vice versa just because a proper scoring rule is applied rather than an improper?

(2) How come the semi-proper scoring rule AUC is sometimes suggested to be the evaluation parameter, as here, and completely seen as bogus other times?

(3) Is the confusion matrix and everything coming with it actually mostly used as it is comprehensible and easy to report to others?

Improve this question
$\endgroup$
12
  • $\begingroup$ I don't understand what you mean in #3. You have evaluated model performance using log-loss. $\endgroup$
    – Dave
    Jul 23 '20 at 17:48
  • $\begingroup$ Sorry, I was just told that: "The deviance criterion in cv.glmnet() (the default for logistic regression) is equivalent to a strictly proper log-loss scoring rule". So, I thought that by using deviance, one already had a scoring rule and did not need to evaluate the probabilistic prediction differently then. Does it makes sense? $\endgroup$
    – Thomas
    Jul 23 '20 at 17:56
  • $\begingroup$ The scoring rule is an evaluation of the probabilistic prediction. $\endgroup$
    – Dave
    Jul 23 '20 at 18:12
  • $\begingroup$ True, and then we are back to my questions. E.g. my Brier score is low (0.00086), so the probabilistic prediction is doing well. In my mind, the improper scoring rule accuracy shows similar high probabilistic predictive performance with certain thresholds. How can I then be sure that 1's are predicted as 1's when using e.g. Brier score compared to other scoring rules? $\endgroup$
    – Thomas
    Jul 23 '20 at 18:28
  • 1
    $\begingroup$ I interpret Harrell to mean that AUC is an adequate diagnostic of a particular model to check if it is any good at making predictions; it is easier to interpret $AUC=0.93$ as an $A$ grade than to interpret a Brier score of $0.18$. (It is this same reason why accuracy initially seems appealing, I believe.) $\endgroup$
    – Dave
    Jul 23 '20 at 20:30
4
$\begingroup$

I won't be able to answer all your questions, but here goes.

  1. So, how can one be sure/argue that 1's are more likely to be predicted as 1's and vice versa just because a proper scoring rule is applied rather than an improper?

You can't be sure, but you can argue.

A scoring rule is a function $S$ that takes a probabilistic prediction or classification $\hat{f}$ and a corresponding actual observation $y$ and maps these to a loss value, $S(\hat{f},y)\in\mathbb{R}$.

Now, both $\hat{f}$ and $y$ are random. For $y$, this is obvious, and for $\hat{f}$, this is due to the fact that we typically sample predictors and corresponding noisy actuals and build our model based on this.

So it makes sense to consider the expectation of our scoring rule, $E\big(S(\hat{f},y)\big)$. Let's only denote the unknown distribution $f$ of $y$ in this expectation for convenience: $E_{y\sim f}\big(S(\hat{f},y)\big)$.

A scoring rule is called proper if this expectation is minimized over all $\hat{f}$ by the true distribution $f$:

$$ E_{y\sim \hat{f}}\big(S(\hat{f},y)\big) \leq E_{y\sim f}\big(S(\hat{f},y)\big). $$

(There is also the opposite convention, where scoring rules are positively oriented and maximized in this situation. We will stick with this one.)

Thus, if we have two competing probabilistic predictions $\hat{f}$ and $\hat{g}$, and one of them is the true distribution $f$, we expect the scoring rule to give us a lower (or at least not-higher) value for this one compared to the other.

And the arguing you asked about happens when we flip this around: one prediction $\hat{f}$ gives us a lower scoring rule than another one $\hat{g}$, so it stands to reason that $\hat{f}$ is "closer" to the true $f$ than $\hat{g}$. But of course, since we are only talking about expectations, it may well be that for our particular sample, a wrong prediction gave us a lower score than the true distribution.

(Also, I'll admit that we are committing a similar error in flipping implications as when people misinterpret $p$ values as probabilities for hypotheses.)

And if we do the exercise with an improper scoring rule, then the problem just is that this improper rule has no reason to be minimized by the true distribution - if it did, it would not be improper any more, but proper.

  1. ow come the semi-proper scoring rule AUC is sometimes suggested to be the evaluation parameter, as here, and completely seen as bogus other times?

I'll be honest: I don't have a handle on this. It might be a good separate question.

  1. Is the confusion matrix and everything coming with it actually mostly used as it is comprehensible and easy to report to others?

Well... People think they understand it. Just as they think they understand accuracy. Easily "understood" falsehoods often have an advantage over harder to understand truths.

(From your comment):

We will still get some misclassified fires and non-fires in my case with using proper scoring rules.

Yes, certainly. Proper scoring rules are not magic silver bullets that will give you perfect predictions. After all, they evaluate probabilistic predictions. If your prediction is 80% for class A, and this is the correct probability, then there is still a 20% chance for non-A.

Proper scoring rules have the advantage that they work in expectation. As above, they may not give you the best result in each and every instance. But they will work better than alternatives in the long run.

Finally, if you get bad predictions even with a proper scoring rule, then of course you need to revisit your model. Was there some predictor you didn't include, because you simply didn't know it? Very bad (probabilistic) predictions can be a source of a lot of learning.

Improve this answer
$\endgroup$
4
  • $\begingroup$ "Also, I'll admit that we are committing a similar error in flipping implications as when people misinterpret p values as probabilities for hypotheses." That seems like a huge deal. $\endgroup$
    – Dave
    Jul 23 '20 at 19:11
  • 1
    $\begingroup$ This answer from @Sycorax gets to Point 2. AUC isn't "bogus" but has been noted to be less sensitive to detecting differences among models than strictly proper scoring rules; see my comment on the question above. My sense is that AUC doesn't well represent the calibration of the probability model, as it just evaluates the rank ordering of probabilities. $\endgroup$
    – EdM
    Jul 23 '20 at 19:56
  • $\begingroup$ Once again, thank you @Stephan. I believe, in my own personal case, the highly imbalanced dataset (ratio of 99.5 to 0.5 %) also affects the higher number of false positive and false negative detected with my model. Nothing really to do about that. $\endgroup$
    – Thomas
    Jul 23 '20 at 20:23
  • 1
    $\begingroup$ This answer from @usεr11852 provides a helpful heuristic explanation of different classes of scoring rules, including the uses and limits of AUC as a semi-proper scoring rule. $\endgroup$
    – EdM
    Jul 24 '20 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.