For a SVM with soft margin, we want to minimize the following: $$ \lambda||\hat w||^2 +(1/n)\sum max(0,1-y_i(\hat w \hat x_i -b)) $$

we know that $2/||\hat w||$ is the width of the margin.

The second term penalizes a misclassified point for how far away it is from the margin relative to the width of the margin. E.g., suppose there is a misclassified point $x_0$: $$ 1-y_0(\hat w \hat x_0 -b)=3 $$ That means $x_0$ is $3/||\hat w||$ away from $1-y_i(\hat w \hat x_i -b)=0$ and is penalized for $3$.

The first term penalizes for the inverse of the width of the margin squared. I find it hard to reconcile with the second term, they seem to be of different scales. Is there any reason (intuitively) why $||\hat w||^2$ is used instead of just $||\hat w||$?

PS: Perhaps one reason is that $||\hat w||^2 $ is easier computation-wise (quadratic programming)? Or perhaps norm squared assumes sample noise to be Gaussian? I am not sure. Has anyone seen the use of $||\hat w||$ instead of $||\hat w||^2$?


As far as I know, the square is introduced in the formulation for convenience. The norm will reach the optimum at the same point, and we get rid of an ugly square root.

With respect to the hinge loss term, the square just makes no difference either, because of the presence of $\lambda$. Both $f(x)=\|x\|$ and $g(x)=\|x\|^2$ are surjective functions of the form $\mathbb R^d \rightarrow \mathbb R_+$. This implies that for any value of $w$ there exists $\lambda \in \mathbb R$ such that $\lambda \|w\|=\|w\|^2$.

That is, for any solution that you find for the squared objective, you can find exactly the same one for the non-squared objective by tweaking $\lambda$.

Since the square is introduced for convenience and it makes no effective difference, I doubt you'll be able to find an intuitive reason for its being there.

  • $\begingroup$ Thanks for the reply. When the errors are normally distributed, the least squares estimates are maximum likelihood (dsplog.com/2012/01/15/…). My additional question is, if the noise in the samples are far from Gaussian, wouldn't $||w||^2$ be a bad choice? Or does that hardly matter in practice? $\endgroup$ – Math J Oct 16 '17 at 14:58
  • $\begingroup$ Here, the norm being minimized does not represent the same thing as in least squares estimation, so that perspective does not translate to this context. $\endgroup$ – cangrejo Oct 16 '17 at 15:36
  • $\begingroup$ Right my mistake. I think you've already answered my original question very well, so I am accepting it. $\endgroup$ – Math J Oct 16 '17 at 16:02
  • $\begingroup$ Suppose the minimizer of the loss $\lambda_1 \|w\|^2+f(w)$ is $w_0$. Does that mean if we choose $\lambda_2 = \lambda_1\|w_0\|$, then the minimizer of the loss $\lambda_2\|w\|+f(w)$ is also $w_0$? $\endgroup$ – DirkGently Mar 23 '20 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.