# The Lagrange multipliers of SVM

Actually the solve the SVM is to solve the following Lagrangian Equation: If we don't use kernel function, $\langle x^{(i)},x^{(j)}\rangle$ is just the vector vector inner product. The $a_ia_j\langle x^{(j)},x^{(j)}\rangle$ is the same to $\langle a_ix^{(i)}, a_jx^{(j)}\rangle$ in the formula.
But if we use the kernel function. That's totally different, right?

Q: So Can I get the same train result between $\langle a_ix^{(i)}, a_jx^{(j)}\rangle$ and $a_ia_j\langle x^{(j)},x^{(j)}\rangle$?

## migrated from stackoverflow.comMay 9 '13 at 3:48

This question came from our site for professional and enthusiast programmers.

• What is the question exactly? If you are looking to understand how kernel functions help, you can check this answer - stats.stackexchange.com/questions/43779/… – TenaliRaman May 9 '13 at 4:54
• Regards to your modified question : First of all, kernel functions are implicitly computing inner product of the points "in some space", so they are just inner products all the time. Secondly, a_i and a_j's are the variables that we are trying to solve for in this dual. So, I am not entirely clear what you are trying to accomplish by pushing these variables inside the inner product. I guess my question, let us say that we push these variables inside the inner product, how do you intend to solve the optimization problem and what do you hope to achieve? – TenaliRaman May 9 '13 at 9:08
• Heh, we have the same cursor themes (; – user88 May 9 '13 at 12:52

## 1 Answer

When using a nonlinear kernel $\alpha_i \alpha_j \langle x^{(i)},x^{(j)}\rangle\neq \langle \alpha_i x^{(i)},\alpha_j x^{(j)}\rangle$. Keep in mind that the support values ($\alpha$'s) are part of the solution of the dual problem.

For nonlinear kernels, the training outcome of $\alpha_i \alpha_j \langle x^{(i)},x^{(j)}\rangle$ is completely different from $\langle \alpha_i x^{(i)},\alpha_j x^{(j)}\rangle$. In fact, for nonlinear kernels, the optimization problem you would get when using $\langle \alpha_i x^{(i)},\alpha_j x^{(j)}\rangle$ is no longer a QP and is probably intractable. There is really no reason for doing this.

Take an RBF kernel for example, with $\kappa(x^{(i)},x^{(j)})=e^{-\gamma \|x^{(i)}-x^{(j)}\|^2}$. In your idea, this becomes: $$\kappa(\alpha_i x^{(i)},\alpha_jx^{(j)})=e^{-\gamma \|\alpha_i x^{(i)}-\alpha_j x^{(j)}\|^2}.$$ You can easily see that solving this to $\alpha$ (and possible $b$) is much more complex!

• why using $⟨α_i^{x(i)},α_jx^{(j)}⟩$ is no longer a QP. I think it can also get the result if I use that form. can that result work? – Samuel May 10 '13 at 1:59
• The optimization problem is no longer a QP because your constraints are no longer linear in $\alpha$. Like I said previously, that result has no interpretation like standard SVM so it probably won't behave like you expect it to. – Marc Claesen May 10 '13 at 7:38
• Note that since inner products are linear, of course you can pull the $\alpha$s inside without changing anything – you just need to apply them to the feature map, not the kernel input. That is, if $k(x, y) = \langle \varphi(x), \varphi(y) \rangle$, it's $\alpha_i \alpha_j \langle \varphi(x), \varphi(y) \rangle = \langle \alpha_i \varphi(x), \alpha_j \varphi(y) \rangle \ne k(\alpha_i x, \alpha_j y)$. – Dougal May 21 '15 at 19:27
• If this is the answer OP was looking for, it is informed by a prescient reading of the question. (+1) – Sycorax May 21 '15 at 19:44