How to use the kernel trick on data that can't be visualized

Question

I'm reviewing the kernel trick and there are a lot of toy examples of how a 2D classification which can't usually be separated by a linear SVM can be separated in 3 space.

This is fine, but how is the kernel trick applied in a real environment where you have maybe hundreds of features? The immediate problem I see is that you can't visualize that many dimensions. So how would one even know what, if any, kernel needs to be used?

Off the top of my head, it seems like if you did dimensionality reduction first to get down to 3D, then maybe one would have an easier time applying the kernel trick.

Answer 1

The kernel trick implies mapping to a higher dimensional feature space. Typically, performing dimensionality reduction first doesn't really help much, except for some minor reductions in computation time. Dimensionality reduction might help in making a problem easier to understand, but what do visualized dimensions really mean after applying PCA? Physical interpretation is already largely lost at that point anyway.

Which kernel to use depends on the problem. In general the RBF kernel tends to work well: $$\kappa(\mathbf{u},\mathbf{v}) = \exp(-\gamma \|\mathbf{u}-\mathbf{v}\|^2).$$ When using this kernel, you still have to find a suitable value for $\gamma \in \mathbb{R}^+_0$, which is typically part of the hyperparameter optimization. A specific value of $\gamma$ can be evaluated via techniques like cross-validation.