For a SVM with soft margin, we want to minimize the following: $$ \lambda||\hat w||^2 +(1/n)\sum max(0,1-y_i(\hat w \hat x_i -b)) $$

we know that $2/||\hat w||$ is the width of the margin.

The second term penalizes a misclassified point for how far away it is from the margin relative to the width of the margin. E.g., suppose there is a misclassified point $x_0$: $$ 1-y_0(\hat w \hat x_0 -b)=3 $$ That means $x_0$ is $3/||\hat w||$ away from $1-y_i(\hat w \hat x_i -b)=0$ and is penalized for $3$.

The first term penalizes for the inverse of the width of the margin squared. I find it hard to reconcile with the second term, they seem to be of different scales. Is there any reason (intuitively) why $||\hat w||^2$ is used instead of just $||\hat w||$?

PS: Perhaps one reason is that $||\hat w||^2 $ is easier computation-wise (quadratic programming)? Or perhaps norm squared assumes sample noise to be Gaussian? I am not sure. Has anyone seen the use of $||\hat w||$ instead of $||\hat w||^2$?

  • $\begingroup$ In case anyone is curious, here is another answer on mathematics.SE. $\endgroup$
    – Couchy
    Dec 5, 2023 at 20:55

1 Answer 1


As far as I know, the square is introduced in the formulation for convenience. The norm will reach the optimum at the same point, and we get rid of an ugly square root.

With respect to the hinge loss term, the square just makes no difference either, because of the presence of $\lambda$. Both $f(x)=\|x\|$ and $g(x)=\|x\|^2$ are surjective functions of the form $\mathbb R^d \rightarrow \mathbb R_+$. This implies that for any value of $w$ there exists $\lambda \in \mathbb R$ such that $\lambda \|w\|=\|w\|^2$.

That is, for any solution that you find for the squared objective, you can find exactly the same one for the non-squared objective by tweaking $\lambda$.

Since the square is introduced for convenience and it makes no effective difference, I doubt you'll be able to find an intuitive reason for its being there.

  • $\begingroup$ Thanks for the reply. When the errors are normally distributed, the least squares estimates are maximum likelihood (dsplog.com/2012/01/15/…). My additional question is, if the noise in the samples are far from Gaussian, wouldn't $||w||^2$ be a bad choice? Or does that hardly matter in practice? $\endgroup$
    – Math J
    Oct 16, 2017 at 14:58
  • $\begingroup$ Here, the norm being minimized does not represent the same thing as in least squares estimation, so that perspective does not translate to this context. $\endgroup$
    – cangrejo
    Oct 16, 2017 at 15:36
  • $\begingroup$ Right my mistake. I think you've already answered my original question very well, so I am accepting it. $\endgroup$
    – Math J
    Oct 16, 2017 at 16:02
  • $\begingroup$ Suppose the minimizer of the loss $\lambda_1 \|w\|^2+f(w)$ is $w_0$. Does that mean if we choose $\lambda_2 = \lambda_1\|w_0\|$, then the minimizer of the loss $\lambda_2\|w\|+f(w)$ is also $w_0$? $\endgroup$
    – DirkGently
    Mar 23, 2020 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.