1
$\begingroup$

This is more of a semantic question, but is the term probability, in the strictest sense, the correct term to use when describing the output of a predictive model that outputs values between 0 and 1? That is, if my logistic regression outputs a value 0.35 for some unseen sample, can and should that be interpreted as "there is a 35% probability of this sample being the 1 class"?

In my understanding, the definition of a probability refers to the frequency of occurrence of some event. Unless it is the case that ~35 out of every 100 samples that my classifier gives a score of 0.35 truly are of the 1 class, is it correct to call this a probability score? Would it be more accurate to just call it a prediction score if it doesn't truly reflect frequency?

$\endgroup$
1
  • $\begingroup$ You may be interested in learning more about calibration. $\endgroup$
    – Sycorax
    Commented Feb 14, 2020 at 16:12

1 Answer 1

2
$\begingroup$

The logistic regression in fact does produce probability output. If your process truly fits the logistic model, then you will get the probability distribution as an output too: $$Pr(y_i=1|X)=\frac{e^{X_i\beta}}{e^{X_i\beta}+1}$$

This does not have to be true with every classifier, of course. In case of logistic regression, we typically calibrate (fit) it using maximum likelihood estimation (MLE) or a similar procedure. There, we use the above probability in one way or another. So, the fitting procedure itself is based on the probabilistic interpretation of the logistic regression.

Other classifier don't need to be based on probabilities, and can use any kind of loss function minimization. This probably makes them not necessarily produce the probabilities as outputs. The question is whether we can interpret any bounded output [0,1] as a probability? I'd say NO, not necessarily.

$\endgroup$
2
  • $\begingroup$ Is the same true for all classifications models which can output scores between 0 and 1 as a prediction? $\endgroup$ Commented Feb 13, 2020 at 20:40
  • 1
    $\begingroup$ It's probably worth discussing model calibration. $\endgroup$
    – Sycorax
    Commented Feb 13, 2020 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.