4
$\begingroup$

I'm reading a book on data science and get confused about how the book describes the hinge loss of SVM. Here is a figure from the book on Page 94:

enter image description here

This figure shows the loss function of a NEGATIVE instance. It seems there is no penalty if the instance lies within the margin even on the positive side. Also on Page 95, the author explains:

enter image description here

However, from what I read about SVM, there should be penalty as long as the negative instance lies within the negative margin or on the wrong side. I updated the loss function as below with orange color:

enter image description here

Can someone tell me if I'm correct or not? Thanks!

Updates: In Wikipedia, it says:

Correctly classified points lying outside the margin boundaries of the support vectors are not penalized, whereas points within the margin boundaries or on the wrong side of the hyperplane are penalized in a linear fashion compared to their distance from the correct boundary

$\endgroup$
3
$\begingroup$

Searching for the quoted text, it seems the book is Data Science for Business (Provost and Fawcett), and they're describing the soft-margin SVM. Their description of the hinge loss is wrong. The problem is that it doesn't penalize misclassified points that lie within the margin, as you mentioned.

In SVMs, smaller weights correspond to larger margins. So, using this "version" of the hinge loss would have pathological consequences: We could achieve the minimum possible loss (zero) simply by choosing weights small enough such that all points lie within the margin. Even if every single point is misclassified. Because the SVM optimization problem contains a regularization term that encourages small weights (i.e. large margins), the solution will always be the zero vector. This means the solution is completely independent of the data, and nothing is learned. Needless to say, this wouldn't make for a very good classifier.

The correct expression for the hinge loss for a soft-margin SVM is:

$$\max \Big( 0, 1 - y f(x) \Big)$$

where $f(x)$ is the output of the SVM given input $x$, and $y$ is the true class (-1 or 1). When the true class is -1 (as in your example), the hinge loss looks like this:

enter image description here

Note that the loss is nonzero for misclassified points, as well as correctly classified points that fall within the margin.

For a proper description of soft-margin SVMs using the hinge loss formulation, see The Elements of Statistical Learning (section 12.3.2) or the Wikipedia article.

$\endgroup$
  • $\begingroup$ I've got to add here: great graph and explanation. I directly got the point by glancing at the graph alone, something other explanations (such as Wikipedia's) failed to do so. $\endgroup$ – Alberto A Aug 30 '19 at 20:17
  • $\begingroup$ @AlbertoA Thanks glad it was useful to you $\endgroup$ – user20160 Aug 31 '19 at 11:11
1
$\begingroup$

The (A) hinge function can be expressed as

$$y_{i} = \gamma \max{\left(x_{i}-\theta, 0\right)} + \varepsilon_{i},$$

where:

  • $\gamma$ is the change in slope after the hinge. In your example, this amounts to the slope following the hinge, since your hinge-only model (see below) assumes zero effect of $x$ on $y$ until the hinge.

  • $\theta$ is the point (in $\boldsymbol{x}$) at which the hinge is located, and is a parameter estimated for the model. I believe your question is answered by considering that the location of the hinge is informed by the loss function.

  • $\varepsilon_{i}$ is some error term with some distribution.

Hinge functions can also be useful in changing any line:

$$y_{i} = \alpha_{0} + \beta x_{i} + \gamma \max{\left(x_{i}-\theta, 0\right)} + \varepsilon_{i},$$

where:

  • $\alpha$ is the model constant, and the intercept of the curve before the hinge (i.e. for $x < \theta$). Of course, if $\theta < 0$, then the curve intersects the $y$-axis after the hinge so $\alpha$ will not necessarily be the $y$-intercept of the bent line.
  • $\beta$ is the slope of the line relating $y$ to $x$
  • $\gamma$ is the change in slope after the hinge.

In addition, the hinge can be used to model how a functional relationship between $y$ and $x$ changes form, as in this model where the relationship becomes quadra

$$y_{i} = \alpha_{0} + \beta x_{i} + \gamma \max{\left(x_{i}-\theta, 0\right)^{2}} + \varepsilon_{i},$$

$\endgroup$
  • $\begingroup$ Hi Alexis, thanks for your great input. Actually I'm confused about whether there are different hinge functions defined for SVM or just this is a typo in the book. $\endgroup$ – James Oct 21 '18 at 18:46
  • $\begingroup$ Well, I gave you an example of another hinge function in my last equation. Hinge functions can also be defined sequentially, as in slope changes at more than a single point. They can also be defined using the $\min()$ function, instead of $\max()$. $\endgroup$ – Alexis Oct 21 '18 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.