In Logistic Regression models, should classification cutoff always be approximately equal to the prior $p = P(Y = 1)$? Mathematically speaking, when using a logistic regression model for binary classification, the output of the model $\hat{y}_i$ for any instance $x_i$ not only can be interpreted as, but is defined as the probability of that instance belonging to the positive class (see this answer).
$$\hat{y}_i = P(y_i = 1 | x_i)$$
If $\hat{y}_i < t$, a classification threshold, we assign $x_i$'s class to 0. Otherwise, 1.
There are many applications in which we set $t$ to a value $t^*$ that optimizes a certain metric. For example, when having false positives is much more costly than having false negatives (as in automated trading systems), we may want to optimize recall, therefore choosing an arbitrarily high $t^*$.
Question:

*

*knowing the training set's prior probability $p_{\text{prior}} = P(Y = 1)$;

*accounting for a difference $d$ in training / production sets variance;

In theory, for probabilistic classification purposes, shouldn't we always assign $t \gets (p_{\text{prior}} \pm d)$, leaving $t^*$ as a purely decision making threshold?

Seems like assigning, for example, $t^* \gets 0.95$ because it bumps recall to a comfortable level has more to do with playing safe and has nothing to do with classifying an event based on probability.
 A: There are two important reasons.
One important reason is when the training set is known to have a different prior from what will be seen in production use.
For example, at the moment there is interest in predicting from symptoms whether someone is likely to have COViD-19, in order to triage people for testing. $P(Y=1)$ in the training set is likely to differ from $P(Y=1)$ in production because

*

*the prevalence of COViD-19 is changing over time as outbreaks progress or are controlled

*the prevalence of other infections is changing seasonally (and for other reasons).

In that situation it would be helpful to adjust the threshold based on the expected prevalence of COViD-19 and (eg) influenza in the population where it's being used.
The second important reason is when the cost of false positive and false negative errors are not the same. If a false positive error is much worse than a false negative error, you want a lower threshold; if a false negative error is worse you want a higher threshold.
A: The answer from Thomas Lumley (+1) states the issues with respect to logistic regression quite clearly.
This question might, however, be extended to whether logistic regression is the best choice for developing a probability model when specific downstream uses and associated cost-based probability cut-offs are in mind, as posited in the question.
Logistic regression implicitly involves log-loss as the scoring rule, which puts emphasis on extremes of probability estimates. Other scoring rules emphasize other regions of the probability scale, which might be more attuned to the anticipated downstream cost/benefit tradeoffs. See this paper as an example of how to tune the choice of a proper scoring rule to handle such situations and to develop a probability model that isn't strictly a logistic regression.
Buja, Andreas, Werner Stuetzle, and Yi Shen. "Loss functions for binary class probability estimation and classification: Structure and applications." Working draft, November 3 (2005): 13.
