0
$\begingroup$

Most texts I've seen hide a lot of what's under the hood in SVMs (support vector machines). The book I'm reading says SVMs can be solved using quadratic programming by using slack variables, giving solution: $$w^* = \sum _{i=1}^N \alpha _i \boldsymbol x_i$$ where $\boldsymbol x_i$ are the training samples and $\alpha _i$ are the dual variables. There's an analogy here to the kernel trick applied to ridge regression where in the ridge regression case, $$\boldsymbol \alpha = (K + \lambda I)^{-1} \boldsymbol y$$

In other words, $\alpha_i$ depends on the Gram Matrix $K$ which depends on the kernel function.

Question 1: in the SVM case do the dual variables $\alpha_i$ also depend on the kernel function? If not, why? If so, how is this implemented/dealt with in finding the solutions? My concern here is the kernel trick for ridge regression only worked because we could write down $\boldsymbol \alpha$ in terms of $X X^T \rightarrow K$ via kernel trick. But if we're solving $\boldsymbol \alpha$ using quadratic programming, it's not clear how we can get the $X$ terms in the form $XX^T$ to apply the kernel trick (and hence avoid transforming the data features explicitly)?

Question 2: do most SVM packages find the solution in the dual space ($\alpha$'s) rather than primal space ($w$'s)?

$\endgroup$
4
  • $\begingroup$ Perhaps this answer might be helpful: stats.stackexchange.com/questions/19181/…. Notice the presence of the dot product in the Lagrangian dual optimisation problem. Whilst the thread doesn't explicitly answer your point about the use of the dual formulation in well known packages, it contains compelling computational reasons why we would opt for it, and that is precisely because of the kernel trick. $\endgroup$
    – microhaus
    Commented Aug 18, 2021 at 23:21
  • $\begingroup$ I'm not exactly sure what you mean by whether $\alpha_i$ "depends" on the kernel function -- in what way? Choice of kernel function can be thought of as a way to reformulate your optimization problem by transforming your input space, with each kernel entailing a different transformation, such that the final values obtained for the $\alpha_i$ should be different based on the kernel. $\endgroup$ Commented Aug 19, 2021 at 1:33
  • $\begingroup$ The general form of the dual objective, however, remains unchanged -- as @microhaus pointed out above, in some cases, the kernel inner product can be very easily [computationally] computed, so a solver might just plug-in that easy expression for the kernel inner product. $\endgroup$ Commented Aug 19, 2021 at 1:34
  • $\begingroup$ You are indeed correct though that it's often non-trivial to actually get things not only in the $XX^\top$ form specified, but to find an easy-to-compute kernel $K$. You can find in Sec. 7 here some worked examples involving a polynomial kernel. $\endgroup$ Commented Aug 19, 2021 at 1:36

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.