I am testing different a couple of different binary classification models using xgboost to predict the likelihood to convert. The difference between the 2 probability distributions shown below is based on different fields being used to train the models.
The first image shows a distribution that makes sense to me given my understanding of the industry. There's a large number of predictions that are unlikely to convert, a handful that is very likely to convert, and the rest fall somewhere in between. Note that the first image has had the probabilities rounded and converted to a scale falling between 0 to 100 instead of 0 to 1.
The second image shows the distribution from a different model where the list of available features was restricted, and not only is the range between the lowest and the highest distribution very narrow (lowest ~0.48, highest ~0.51), but the distribution is very heavy on just a handful of specific probabilities.
My questions are:
- Does a probability distribution like the latter give reason to disregard that model as a candidate?
- Is it more typical to see results like the first or second distribution for these types of models?
- What kind of features would typically cause the results to be more like one or the other?
Apologies if these are dumb questions - I don't have a stats background just a very rudimentary understanding of the basics. Any pointers to additional reading are appreciated.