# Binary probability models: considering event probability above the prior probability

Consider a model of the probability of a binary, yes/no-type of event. The event is infrequent, say it happens only once every thousand times. In that regard, the prior probability of the event is $$0.001$$.

Consequently, if a predictive model like a logistic regression predicts a probability of $$0.05$$ given a certain situation (the model features), while there is still only a $$5\%$$ chance of the event happening, the event is $$50$$-times more likely to occur than usual.

$$\dfrac{0.05}{0.001}$$

If that event is something catastrophic, I would want to know if the chance of it happening if $$50$$ times higher than usual, even if the event remains unlikely ($$5\%$$).

What drawbacks might there be to looking at predicted probability in this way? My reservation is that I don't want to get hung up on something like, "The chance of it happening is up from ultra-super-duper-unlikely to ultra-unlikely," something like a change in probability from a prior of $$0.000001$$ to $$0.0001$$. At the same time, a $$100$$-fold increase in event probability seems like a big deal, even if the event remains unlikely.

• Nice question (+1). Some quick points: 1. Such low probability scores are usually helpful to contextualise as part of their associated expected cost. 2. Visualisation of these results might be helped with the use of lift curves. 3. Associated with point 1. Can you look into the NetBenefit of the model? Some form of (clinical) usefulness? 4. Think a big draw-back is that quantifying "uncertainty" is pretty hard. Do we show a calibration plot focusing at the[0,10%] interval and show that we have well calibrated probabilities there for example? Tall order... Commented Mar 5, 2023 at 0:18