1
$\begingroup$

I would like to know what would be the disadvantages of sampling from a Bernoulli with p probability (p being the output of a logistic regression) to generate the binary classification?

Choosing a threshold to classify the logistic regression output probability p seems to be the most common approach. I can understand that using ROC to choose a threshold, for example, might help with reasoning about false positive and false negative rates.

Thank you in advance for our help.

$\endgroup$
2
  • $\begingroup$ What happens when you get unlucky and sample an unlikely outcome? Imagine predicting a probability of death of $0.99$ (say getting lost in the Sahara Desert), only to sample the less likely (but still possible) "survival" event (Mauro Prosperi), and acting as if the action is one that should lead to survival. $//$ There are plenty of issues with setting thresholds and classifying based on those thresholds (Harrell's blog discusses these here and here), but an upside is that it is deterministic. $\endgroup$
    – Dave
    Oct 4, 2023 at 20:24
  • 1
    $\begingroup$ I think I would like to flip this around and ask what you see as the advantage of sampling in this way. $\endgroup$
    – Dave
    Oct 4, 2023 at 20:26

1 Answer 1

2
$\begingroup$

It depends entirely on the purpose of your classification!

Say you have access to biometric data (blood pressure, height, weight, age, etc) for a lot of people and you want to predict the probability that they will have a heart attack this year. The purpose is to advise them on whether or not to get further screenings.

In this case it would be inappropriate to advise the individual based on a weighted coin flip! You should probably set a relatively low threshold and advise them to get further screenings if your predicted probability that they would have a heart attack exceeds this threshold. Perhaps something like twice the background rate would be appropriate?

Imagine, instead, that you have access to demographic information (age, sex, number of fitness influencers they are subscribed to, etc) about the users of a website and you would like to decide which version of an advertisement to show them. You will either show them a "funny" version or a "serious" version. In this case it might not be a bad idea to show them one version or the other with probability $p$ generated by the model. A nice feature of doing this is that you will show people the version of the ad you think they will click on, but you are still generating A/B testing data to address model drift over time.

$\endgroup$
2
  • $\begingroup$ Thank you for your response, this makes sense. I am using a logistic regression to look at the probability of individuals reporting good health depending on certain characteristics. I then need to use this model to assign a self-reported health status to individuals from a artificial population, so I can calculate specific metrics that depend on self-reported health. I thought that classification using a Bernoulli is best in this case as I don't have a clear reasoning for choosing thresholds. $\endgroup$
    – user11849
    Oct 4, 2023 at 21:22
  • $\begingroup$ @user11849 if you have an artificial population and you are wanting to calculate statistics about this population assuming they follow this probability distribution, then you can probably calculate these statistics "exactly". If you need to create a "fake population" following the distribution, then classification using a Bernoulli seems fine. $\endgroup$ Oct 15, 2023 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.