I have read somewhere that a bonus of using logistic regression (as a classic statistical tool) is that is gives out probabilities. But then by reviewing all the other machine learning methods, most of them give out some continuous scores that can be converted into probability instead of thresholding.
Is there a difference between these two? Are the probabilities from LR somehow superior to those of other ML methods?
Due to the no-free-lunch theorem, we can't say for sure which approach is superior (there's always a possible dataset where our approach of choice is not optimal).
However, logistic regression is a linear learning algorithm that maximizes directly the Bernoulli likelihood. Bar considerations regarding bias and variance, maximizing the likelihood of choice if it matches the problem is a good idea, compared with, e.g. decision trees, where this notion of probability in terminal leafs is more ad-hoc than likelihood-based (the posterior probability is not optimized directly).
Other methods, however, also do that, e.g. neural networks with sigmoid activation in the output layer and cross-entropy loss.
