6
$\begingroup$

I was watching the MIT open course on machine learning. In the session on SVM, the professor derived that the margin is $\frac{1}{\|w\|}$. However, the professor then said for the convenience of mathematics, to maximize $\frac{1}{\|w\|}$, which implies to minimize $\|w\|$, is minimizing $\frac{\|w\|^2}{2}$.

What is the rationale behind this substitution?

$\endgroup$

1 Answer 1

8
$\begingroup$

Notice that $\frac{1}{x}$ is a decreasing function over the positive domain and $\frac{x^2}{2}$ is an increasing function over the non-negative domain.

If $g$ is a decreasing function (a function where as the input increases, the output decreases). Maximizing $f_1(x)$ is equivalent to minimizing $g(f_1(x))$. Here $f_1(w)=\frac{1}{\|w\|}$, and $g(x)=\frac{1}{x}$, hence $g(f_1(w))=\|w\|$.

If $h$ is an increasing function (a function where if the input increases, the output increases, similarly, if the input decreases, the output decreases). Minimizing $f_2(x)$ is equivalent to minimizing $h(f_2(x))$. Here $f_2(w)=\|w\|$ and $h(x)=\frac{x^2}{2}$, hence $h(f_2(w))=\frac{\|w\|^2}{2}$.

$\endgroup$
2
  • $\begingroup$ Thank you Siong! But we're forcing $||w||$, a linear function, to change to $\frac{||w||^2}{2}$ the convex function. It makes me feel uncomfortable. $\endgroup$
    – Yujian
    Commented Jan 23, 2018 at 15:42
  • 2
    $\begingroup$ Perhaps working with some example might help, let $w$ be one dimension, such that $3 \le w \le 5$, you can check that both $|w|$ and $|w|^2$ share the same maximum and minimum point. thanks to the monotonicity. maximizing $|w|^3$ gives you the same result too since it is increasing over the domain $3 \le w \le 5$. $\endgroup$ Commented Jan 23, 2018 at 15:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.