cost function of a svm

I am unable to understand the above cost function. The two possible outputs ($y$) are $-1$ and $+1$. As far as i know, $x_i$ values are individual training example values, $y_i$ are actual true values for that example and w is the weights that we are adjusting so as to minimise the cost. if the actual output is $1$ and and my SVM predicts it as $-1$ my classifier is penalised, how will my SVM penalise? how much will be the penalty? I know this function gives cost 0 for that training example when prediction is correct and some penalty if predicted wrong. I am also confused how $\max(0,1-y(wx+b))$ outputs the cost for one training example. please enlighten me how this cost function works.


The following diagram I made illustrates the geometric meaning of $w \cdot x_i + b$ in a 2D plane: enter image description here where $w$ is a norm vector that controls the direction of the linear boundary $L$; for simplicity, let's assume $\|w\| = 1$, then $b$ represents an offset applied to $L$ along the opposite direction of $w$. The distance of a point $x_i$ to $L$ is therefore $w \cdot x_i - \left(-b\right) = w \cdot x_i + b$.

Let $\hat{y_i} = w \cdot x_i + b$ be the model's prediction for sample $i$; then we have:

  • If $y_i = +1$ and $\hat{y_i} \lt +1$, then $\mathrm{max}\left(0, 1 - \hat{y_i}\right) = 1 - \hat{y_i} \gt 0$;
  • If $y_i = -1$ and $\hat{y_i} \gt -1$, then $\mathrm{max}\left(0, 1 - \hat{y_i}\right) = 1 + \hat{y_i} \gt 0$;
  • If $y_i = +1$ and $\hat{y_i} \ge +1$, then $\mathrm{max}\left(0, 1 - \hat{y_i}\right) = 0$;
  • If $y_i = -1$ and $\hat{y_i} \le -1$, then $\mathrm{max}\left(0, 1 - \hat{y_i}\right) = 0$;

The intuition here is that the penalty is $0$ only when the model's prediction is sufficiently correct (i.e. $y_i$ and $\hat{y_i}$ are of the same symbol and $\left|\hat{y_i}\right| \gt 1$).

| cite | improve this answer | |
  • $\begingroup$ ok thank u ..i get it now...but still i have a confusion about the 1−y(wx+b) part. if i am not wrong yi in above equation is the actual output of training example xi. so if suppose yi is -1 , the equation becomes 1+(wx+b) (replacing yi =-1). how this equation will give a value less than or equal to zero so that no penalty is their for right prediction...plz give some intuition on "1−y(wx+b)" this part of equation $\endgroup$ – rushikesh chaskar Sep 12 at 4:17
  • $\begingroup$ @rushikeshchaskar In the case where $y_i$ is $-1$, as you said, the second argument of the $\mathrm{max}$ function becomes $1 + \left(w x_i + b\right)$. For the loss to stay at $0$, $\left(w x_i + b\right)$ has to be less or equal to $-1$. To simplify things, let's assume we are working with 2-dimensional data, so the two parameters, $w$ and $b$ specifies a line on the 2D plane; the key here is to understand the expression $w x_i + b$ describe the "distance" from $x_i$ to the line defined by $w$ and $b$ (the unit of that distance is determined by $w$). $\endgroup$ – Xiubo Zhang Sep 12 at 11:43
  • $\begingroup$ @rushikeshchaskar This "distance" has a direction. if $y_i$ is $-1$, then a "good" prediction that wouldn't incur penalty must have the same direction as $y_i$, i.e. $w x_i + b$ has to be negative; but this is not enough as SVM is a "maximum margin classifier"; this means the "distance" from $x_i$ to the linear boundary defined by the function $y = w x + b$ has to exceed the threshold of $1$ for the SVM algorithm to wavier the penalty --- when $y_i = -1$, this means $w x_i + b$ must be less than or equal to $-1$ (same direction as $y_i$ and magnitude is greater than or equal to $1$). $\endgroup$ – Xiubo Zhang Sep 12 at 11:55
  • $\begingroup$ i know we shud avoid comments thanking others..but thank u so much for ur time.. i understood it well Xiubo Zhang $\endgroup$ – rushikesh chaskar Sep 12 at 16:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.