2
$\begingroup$

Mathematically speaking, when using a logistic regression model for binary classification, the output of the model $\hat{y}_i$ for any instance $x_i$ not only can be interpreted as, but is defined as the probability of that instance belonging to the positive class (see this answer).

$$\hat{y}_i = P(y_i = 1 | x_i)$$

If $\hat{y}_i < t$, a classification threshold, we assign $x_i$'s class to 0. Otherwise, 1.

There are many applications in which we set $t$ to a value $t^*$ that optimizes a certain metric. For example, when having false positives is much more costly than having false negatives (as in automated trading systems), we may want to optimize recall, therefore choosing an arbitrarily high $t^*$.

Question:

  • knowing the training set's prior probability $p_{\text{prior}} = P(Y = 1)$;
  • accounting for a difference $d$ in training / production sets variance;

In theory, for probabilistic classification purposes, shouldn't we always assign $t \gets (p_{\text{prior}} \pm d)$, leaving $t^*$ as a purely decision making threshold?

Seems like assigning, for example, $t^* \gets 0.95$ because it bumps recall to a comfortable level has more to do with playing safe and has nothing to do with classifying an event based on probability.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Cross Validated advocates for so-called “proper scoring rules” that do not involve any kind of threshold, but if you’re going to use a threshold for decision-making purposes, consider if, despite having balanced classes, it’s MUCH worse to have a false positive than a false negative. $\endgroup$
    – Dave
    Jul 26, 2020 at 4:45
  • $\begingroup$ There's no difference by doing so, you just have clipped the loss at some points, the plus term is incorrect, (might violate probability threshold) , $t=p_{prior}±d$. $\endgroup$
    – 4.Pi.n
    Jul 26, 2020 at 4:45
  • $\begingroup$ @m-zayan I don't get what you mean by clipping the loss, since in this case, the loss function is based on maximum likelihood and it's only a factor on the training phase. And making $t = p_{\text{prior}} \pm d$ is valid because posterior probability can be greater than prior prob (see Thomas Lumley's answer). $\endgroup$
    – Victor Valente
    Jul 26, 2020 at 17:15
  • $\begingroup$ @Dave I think your comment about proper scoring rules pretty much sums up and solves my concerns. I wasn't aware of such methods. Thank you. $\endgroup$
    – Victor Valente
    Jul 26, 2020 at 17:16
  • $\begingroup$ Trained model has some local information about dataset prior probability, (model learns prior probability of $Y$), in case it, having a threshold exactly equal the prior for example, It's quite same as clipping optimal result at some point for each class exactly at training set prior, the problem that, it's not fair in the case of having rare events (unbalanced data), and data set has inaccurate distribution which fixes by the model, means model learning new probability distribution based on bernoulli distribution assumption $\endgroup$
    – 4.Pi.n
    Jul 26, 2020 at 21:03

2 Answers 2

Reset to default
4
$\begingroup$

There are two important reasons.

One important reason is when the training set is known to have a different prior from what will be seen in production use.

For example, at the moment there is interest in predicting from symptoms whether someone is likely to have COViD-19, in order to triage people for testing. $P(Y=1)$ in the training set is likely to differ from $P(Y=1)$ in production because

  • the prevalence of COViD-19 is changing over time as outbreaks progress or are controlled
  • the prevalence of other infections is changing seasonally (and for other reasons).

In that situation it would be helpful to adjust the threshold based on the expected prevalence of COViD-19 and (eg) influenza in the population where it's being used.

The second important reason is when the cost of false positive and false negative errors are not the same. If a false positive error is much worse than a false negative error, you want a lower threshold; if a false negative error is worse you want a higher threshold.

Improve this answer
$\endgroup$
3
  • $\begingroup$ First example is definitely something I missed. I guess setting $d$ to be approximately the expected future prevalence would solve this issue. How to calculate $d$ in this case is a different matter. $\endgroup$
    – Victor Valente
    Jul 26, 2020 at 5:53
  • $\begingroup$ For the second example, one question remains: in theory, wouldn't a good model with $t = p_{\text{prior}} \pm d$ be able to correctly classify most of the instances seen in production? False positives or false negatives should occur only on outlier instances, and that should affect $t$ only if the cost of a misclassification was VERY asymmetric. Even then, seems like $t = p_{\text{prior}} \pm d \pm e$ ($e$ being an error asymmetry factor) would suffice, no? $\endgroup$
    – Victor Valente
    Jul 26, 2020 at 5:53
  • $\begingroup$ I edited the question in order to make a clear distinction between probabilistic classification and decision making. $\endgroup$
    – Victor Valente
    Jul 26, 2020 at 17:19
2
$\begingroup$

The answer from Thomas Lumley (+1) states the issues with respect to logistic regression quite clearly.

This question might, however, be extended to whether logistic regression is the best choice for developing a probability model when specific downstream uses and associated cost-based probability cut-offs are in mind, as posited in the question.

Logistic regression implicitly involves log-loss as the scoring rule, which puts emphasis on extremes of probability estimates. Other scoring rules emphasize other regions of the probability scale, which might be more attuned to the anticipated downstream cost/benefit tradeoffs. See this paper as an example of how to tune the choice of a proper scoring rule to handle such situations and to develop a probability model that isn't strictly a logistic regression.

Buja, Andreas, Werner Stuetzle, and Yi Shen. "Loss functions for binary class probability estimation and classification: Structure and applications." Working draft, November 3 (2005): 13.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.