3
$\begingroup$

What does it mean that 'The SVM hinge loss estimates the mode of the posterior class probabilities'(Elements of statistical Learning p.427).

The decision function f(x) assigns to the positive class(+1) if it is greater than 0 or if it is less than 0 to the negative class(-1) but how does that relate to the posterior class probabilities?

$\endgroup$

2 Answers 2

4
+50
$\begingroup$

Preamble: The mode of a distribution is the value that is most commonly observed. A bit more formally, it is the value with the highest probability of occurrence.

Now, in the context of the posterior class probabilities as here, "mode" refers to the class that has the highest probability for a given data-point $x$, i.e. it is the class that the SVM classifies assumes it has the highest probability of being the correct class. (and hopefully that class indeed is the correct class)

Importantly, the SVM hinge loss is minimized when the margin between the two classes is as large as possible. And that margin is in terms of "scores", not probabilities. In that sense, the minimizer does not faithfully reflect the posterior probabilities. We use the logistic function to get them - the logistic function turns the decision function's scores into probabilities.

(And that's why SVMs, as they are not inherently probabilistic models, do not directly output probabilities and while SVMs might "accurate" in their class assignments (i.e. finding the mode) their probabilities are often not well-calibrated.)

$\endgroup$
2
  • 1
    $\begingroup$ Can you explain how the decision function $\text{sign}(f(x))$, ($f(x) = w^T x + b$) directly relates to the posterior probabilities $\mathbb{P}(Y = +1|x)$ and $\mathbb{P}(Y = -1|x)$? Specifically, how does the minimizer of the SVM hinge loss reflect these probabilities? @usεr11852 $\endgroup$
    – J.doe
    Dec 27, 2023 at 21:22
  • $\begingroup$ Good follow-up question (+1), the answer was bit long for a comment so I edited the main answer. In short, the minimizers reflects them not very well, as it focuses on their mode. $\endgroup$
    – usεr11852
    Dec 28, 2023 at 1:11
5
$\begingroup$

As stated in Elements of Statistical Learning (same page you mention) the SVM hinge loss $L(y, f(x)) = [1 - y f(x)]_+$ is minimized in expectation by the minimizing function $$ f^*(x) = \mathrm{sign} \left[ \mathbb{P} ( Y = 1 \vert x) - \frac{1}{2} \right] . $$ This is equivalent to selecting the mode of the posterior class probabilities, i.e. to selecting the class with the higher probability: We have $\mathbb{P}(Y=1 \vert x) > \mathbb{P}(Y=-1 \vert x)$ if and only if $\mathbb{P}(Y=1 \vert x) > 1/2$ in the binary classification setting.

The optimal $f^*$ is the theoretical solution of the loss minimization problem, when we impose no restrictions on the shape. For the SVM we however choose $f(x) = w^T x + b$ as a separating hyperplane, so the SVM decision function will only come somewhat close to the theoretical $f^*$. Additionally, the SVM does not estimate probabilities, but looks for a good separation of the classes. The hinge loss gives the optimization no incentive to represent probabilities with $f(x)$, as only the signs of these values are relevant for an optimal solution.

The minimizer of the SVM hinge loss thus reflects the posterior probabilities only in the sense that it indicates which probability is higher. Stated differently, in most cases where $f(x) > 0$ it should hold that $\mathbb{P}(Y=1 \vert x) > \mathbb{P}(Y=-1 \vert x)$. But the value of $f(x)$ will not give you any info on the exact probability $\mathbb{P} (Y = 1 \vert x)$.

$\endgroup$
2
  • 1
    $\begingroup$ Can you explain how the decision function $\text{sign}(f(x))$, ($f(x) = w^T x + b$) directly relates to the posterior probabilities $\mathbb{P}(Y = +1|x)$ and $\mathbb{P}(Y = -1|x)$? Specifically, how does the minimizer of the SVM hinge loss reflect these probabilities? @picky_porpoise $\endgroup$
    – J.doe
    Dec 27, 2023 at 21:29
  • $\begingroup$ @J.doe See my edits. $\endgroup$ Dec 28, 2023 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.