What is the motivation or objective for adopting Kernel methods? Is kernel trick a feature engineering method?

Question

I come to know that kernel methods can be used in not only SVM but also many machine learning algorithms. I understanding that in SVM, the reason for using kernel trick is that some data are linearly inseparable, and people think the data may be linearly separable in a certain higher dimensional space. While it is difficult or impossible to map the original feature vectors to a higher-dimensional space. Therefore a kernel function can be used to latently do the mapping and the dot product in higher-dimensional space together. So I think the motivation for using kernel methods in SVM is: to convert a linear SVM to a nonlinear SVM, which can classify linearly inseparable data.

Q1: So I am wondering what are the purposes of using Kernel methods in other ML algorithms in general? Is it the same: to classify linearly inseparable data with an originally linear model?

Q2: I am trying to have a knowledge structure in my mind, so Which super category do kernel methods fall into? Since the kernel functions operate on feature vectors, I couldn't help but relate kernel methods to Feature Engineering(feature selection, feature extraction like PCA, LDA, etc.). Can I state this: kernel methods is a kind of nonlinear Feature Engineering methods?

Answer 1

Q1: One (still very close) example is support vector clustering. This can be used for outlier detection (fraud prevention and so on). The strategy is very similar: start with an easy algorithm and then kernelize it. In this case (SVC) the “easy algorithm” goes like “find the smallest ball such that almost all of the data points lie inside the ball”. When kernelized one does exactly the same: search for the smallest ball such that ... but in a very weird, infinite dimensional space.

Q2: I would say that you are absolutely correct. One can even explicitly write down the map that lies behind the kernel trick. I think that for the RBF kernel and two input dimensions the map takes some input (x,y) and transforms it to something like $(x^ay^b)_{a,b \in \mathbb{N}_0}$, i.e. take x and y and create the infinite vector of all possible monomials from it. So yes, SVM with RBF kernel = ...

1. transform the data (create new features) by the map above

2. create a linear separator in the infinite dimensional space ($l^2$)

NB: I think that People also try to mimic a finite dimensional version of this by soewhat randomly create features like finite sums and products of the input features. However, without any “reason” why these should work behind them, I found them not to be very helpful in the real world problems that I have seen...

NB2: caution with the term inseparable because in SVM there are two different countermeasures: the kernel trick and “gently penalized misclasification “ (the additional term in the cost function usually named something like $C\sum \xi_i$). These are there because perfectly separating the data in the infinite dimensional space might lead to “veeery wiggly” decision boundaries in the original space which might brutally overfit the data. The solution is to let some of the points be misclassified and make the ratio “how many points are actually bad” a hyperparameter C.