3
$\begingroup$

This question already has an answer here:

I'm taking Andrew Ng's Machine Learning Course. In the lesson on classification algorithms, he presents the logit function ($\frac{1}{1+e^{-x}}$) and the way it converts parameterized functions to probabilities. He marks the "decision boundary" at .5, noting that the logit function cross .5 at $x=0$. If the value of the logistic regression function is .5 or greater, the result is given as 1. If it's less than .5, it's given as 0.

This sounds arbitrary. In real-world applications, is it common to adjust the decision boundary, or disregard the boundary and list only the probabilities?

  • Scenario 1 Adjusting the decision boundary: In the logit function, when x is -.3, y is .4256. Because y is less than .5, the predicted value is 0. If the lump in my patient's breast has a 42.56% probability of being malignant, I don't want that rounded down to 0%.

    I would adjust the decision boundary from .5 to maybe .01. 1% or less probability is the threshold where I'd tell my patient their lump is nothing to worry about.

  • Scenario 2 Removing the decision boundary: My team of social workers helps teens in my city. My classification parameters predict that 15% of the teen population is at high risk of committing a crime. That is, it predicts a positive result for 15% of P. My team is small, however, and can only reach 3% of the teen population.

    Knowing that the 15% prediction includes teens at 51% probability along with teens at 99% probability, I disregard the binary predictions entirely and order teens based on their probability, thus reaching the most at-risk teens first.

Are these scenarios—where you adjust decision boundaries and/or list probabilities alone—common in real-world usage?

|cite|improve this question
$\endgroup$

marked as duplicate by Stephan Kolassa, kjetil b halvorsen, Peter Flom♦ Mar 16 '18 at 14:42

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ Its absolutely critical to adjust the boundary in real world applications of these ideas. And, as you mention, for some applications a decision boundary is irrelevent. $\endgroup$ – Matthew Drury Mar 15 '18 at 14:28
2
$\begingroup$
  • Logistic regression can be viewed as a model output probabilities, not just a classifier.
  • As @Matthew Drury mentioned, adjusting the boundary is used very frequently in real world to make trade-off between true positive and false positive rates. Details see [ROC curve] for details (https://en.wikipedia.org/wiki/Receiver_operating_characteristic)

In addition to optimizing the logistic loss, which is the view from the machine learning perspective. There are very rich literature in statistics about logistic regression can answer your question perfectly and give more interpretation of the probability produced from the model.

These literatures also establish the relationship between max likelihood and binomial assumptions of logistic regression. (Note, not any number between 0 and 1 is a probability number. When we talk about probability we also want to think about the distribution.) The link function (sigmoid, $\text{logit}^{-1}$) is carefully designed with rigorous math behind.

I would strongly recommend you to re-learn logistic regression from statistics literature in addition to machine learning.

Here are some starting resources:

FAQ: HOW DO I INTERPRET ODDS RATIOS IN LOGISTIC REGRESSION?

LOGIT REGRESSION | R DATA ANALYSIS EXAMPLE

And some related topics in CV

What is the difference between linear regression and logistic regression?

Is there i.i.d. assumption on logistic regression?

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thank you very much. I imagine that, after completing Dr. Ng's course, I've only scratched the surface of regression studies. Thanks for giving me some places to start. $\endgroup$ – Bagheera Mar 15 '18 at 15:03
  • $\begingroup$ @Bagheera thanks for accepting my answer. I also learned logistic regression from machine learning first. Optimization plays an important role in machine leanring. However, people are doing linear regression and logistic regression for a very long time in statistics community. There are many books talk about "Generalized Linear Model", that tells every detail about the probabilistic interpretation of it. $\endgroup$ – Haitao Du Mar 15 '18 at 15:07
  • $\begingroup$ As an example, in credit risk (banking) you use it for scorecards for acceptance/rejection of clients for credit cards for example application (that's when you need the boundary threshold) or for classifying clients in terms of probabilities of default (between 0 and 1) which is then used further down for pricing (mortgages) for capital buffer calculation. $\endgroup$ – R. Prost Mar 15 '18 at 16:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.