Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am aware that classifiers are not necessarily calibrated in probability (see: scikit-learn manual here).
I wanted to know more about what we can say about the link between scores and probabilities. For exemple, are we ensured that probabilities are strictly increasing with the score ?
What kind of probability? If you ask about subjectivist probability, the degree of belief, then the higher score would mean that you find some output as more likely then other, so it would directly correspond to it.
If you are talking about frequentist probability, so you ask if the scores would correspond to empirical proportions, then it gets more complicated. By minimizing some loss, you are pushing the scores to be close to the target categories (where the definition of closeness depends on choice of the loss function). You'd expect to see higher scores more often in the target classes as compared to lower scores, so they would somehow correspond to the probabilities.
If the probabilities are not calibrated, there's no exact correspondence. For example, decision trees pack similar values into bins, so the probabilities would not be a continuous function of the scores. In other cases, scores may be pushed towards extreme values, or towards moderate values, and the function may be complicated.
