2
$\begingroup$

I am testing different a couple of different binary classification models using xgboost to predict the likelihood to convert. The difference between the 2 probability distributions shown below is based on different fields being used to train the models.

The first image shows a distribution that makes sense to me given my understanding of the industry. There's a large number of predictions that are unlikely to convert, a handful that is very likely to convert, and the rest fall somewhere in between. Note that the first image has had the probabilities rounded and converted to a scale falling between 0 to 100 instead of 0 to 1.

enter image description here

The second image shows the distribution from a different model where the list of available features was restricted, and not only is the range between the lowest and the highest distribution very narrow (lowest ~0.48, highest ~0.51), but the distribution is very heavy on just a handful of specific probabilities.

enter image description here

My questions are:

  • Does a probability distribution like the latter give reason to disregard that model as a candidate?
  • Is it more typical to see results like the first or second distribution for these types of models?
  • What kind of features would typically cause the results to be more like one or the other?

Apologies if these are dumb questions - I don't have a stats background just a very rudimentary understanding of the basics. Any pointers to additional reading are appreciated.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ It seems your second model didn't learn anything out of the data, it's like random guess as a binary task. $\endgroup$
    – doubllle
    Commented Dec 14, 2021 at 12:34
  • $\begingroup$ Are your features primarily categorical in nature? Do only a small number of different combinations of categories appear in your data? // Are you using regularization? // Have you tried using a different learning rate? $\endgroup$
    – Sycorax ♦
    Commented Dec 14, 2021 at 15:23

1 Answer 1

Reset to default
-1
$\begingroup$

Let me try to answer your question. First of all, I wanted to highlight that what you are observing here in terms of distribution of the probability of the model is a common effect especially for boosted models, SVM or even naive bayes. You tend to see a sigmoid distortion of the probabilities more prominent towards the extremes rather than an even distribution. If you look at the logic behind these models, you will see it makes sense in the way that they do the classification.

  1. Whether to use the model or not is a decision that ideally should be taken based on the metrics that you are looking for like ROC etc. Unless what you really intend to use are the probability. In that case you should check out something called platt scaling. Check out this link to get more details on it. https://www.analyticsvidhya.com/blog/2016/07/platt-scaling-isotonic-regression-minimize-logloss-error/
  2. I hope I have answered your concern already.
  3. This trend is not a function of features but the result of the model you are using.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.