1
$\begingroup$

Actually the solve the SVM is to solve the following Lagrangian Equation:enter image description here

If we don't use kernel function, $\langle x^{(i)},x^{(j)}\rangle$ is just the vector vector inner product. The $a_ia_j\langle x^{(j)},x^{(j)}\rangle $ is the same to $\langle a_ix^{(i)}, a_jx^{(j)}\rangle $ in the formula.
But if we use the kernel function. That's totally different, right?

Q: So Can I get the same train result between $\langle a_ix^{(i)}, a_jx^{(j)}\rangle$ and $a_ia_j\langle x^{(j)},x^{(j)}\rangle $?

$\endgroup$
3
  • 2
    $\begingroup$ What is the question exactly? If you are looking to understand how kernel functions help, you can check this answer - stats.stackexchange.com/questions/43779/… $\endgroup$ Commented May 9, 2013 at 4:54
  • $\begingroup$ Regards to your modified question : First of all, kernel functions are implicitly computing inner product of the points "in some space", so they are just inner products all the time. Secondly, a_i and a_j's are the variables that we are trying to solve for in this dual. So, I am not entirely clear what you are trying to accomplish by pushing these variables inside the inner product. I guess my question, let us say that we push these variables inside the inner product, how do you intend to solve the optimization problem and what do you hope to achieve? $\endgroup$ Commented May 9, 2013 at 9:08
  • $\begingroup$ Heh, we have the same cursor themes (; $\endgroup$
    – user88
    Commented May 9, 2013 at 12:52

1 Answer 1

4
$\begingroup$

When using a nonlinear kernel $\alpha_i \alpha_j \langle x^{(i)},x^{(j)}\rangle\neq \langle \alpha_i x^{(i)},\alpha_j x^{(j)}\rangle $. Keep in mind that the support values ($\alpha$'s) are part of the solution of the dual problem.

For nonlinear kernels, the training outcome of $\alpha_i \alpha_j \langle x^{(i)},x^{(j)}\rangle$ is completely different from $\langle \alpha_i x^{(i)},\alpha_j x^{(j)}\rangle$. In fact, for nonlinear kernels, the optimization problem you would get when using $\langle \alpha_i x^{(i)},\alpha_j x^{(j)}\rangle$ is no longer a QP and is probably intractable. There is really no reason for doing this.

Take an RBF kernel for example, with $\kappa(x^{(i)},x^{(j)})=e^{-\gamma \|x^{(i)}-x^{(j)}\|^2}$. In your idea, this becomes: $$\kappa(\alpha_i x^{(i)},\alpha_jx^{(j)})=e^{-\gamma \|\alpha_i x^{(i)}-\alpha_j x^{(j)}\|^2}.$$ You can easily see that solving this to $\alpha$ (and possible $b$) is much more complex!

$\endgroup$
4
  • $\begingroup$ why using $⟨α_i^{x(i)},α_jx^{(j)}⟩$ is no longer a QP. I think it can also get the result if I use that form. can that result work? $\endgroup$
    – Samuel
    Commented May 10, 2013 at 1:59
  • $\begingroup$ The optimization problem is no longer a QP because your constraints are no longer linear in $\alpha$. Like I said previously, that result has no interpretation like standard SVM so it probably won't behave like you expect it to. $\endgroup$ Commented May 10, 2013 at 7:38
  • $\begingroup$ Note that since inner products are linear, of course you can pull the $\alpha$s inside without changing anything – you just need to apply them to the feature map, not the kernel input. That is, if $k(x, y) = \langle \varphi(x), \varphi(y) \rangle$, it's $\alpha_i \alpha_j \langle \varphi(x), \varphi(y) \rangle = \langle \alpha_i \varphi(x), \alpha_j \varphi(y) \rangle \ne k(\alpha_i x, \alpha_j y)$. $\endgroup$
    – Danica
    Commented May 21, 2015 at 19:27
  • $\begingroup$ If this is the answer OP was looking for, it is informed by a prescient reading of the question. (+1) $\endgroup$
    – Sycorax
    Commented May 21, 2015 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.