# What is the motivation or objective for adopting Kernel methods? Is kernel trick a feature engineering method?

I come to know that kernel methods can be used in not only SVM but also many machine learning algorithms. I understanding that in SVM, the reason for using kernel trick is that some data are linearly inseparable, and people think the data may be linearly separable in a certain higher dimensional space. While it is difficult or impossible to map the original feature vectors to a higher-dimensional space. Therefore a kernel function can be used to latently do the mapping and the dot product in higher-dimensional space together. So I think the motivation for using kernel methods in SVM is: to convert a linear SVM to a nonlinear SVM, which can classify linearly inseparable data.

Q1: So I am wondering what are the purposes of using Kernel methods in other ML algorithms in general? Is it the same: to classify linearly inseparable data with an originally linear model?

Q2: I am trying to have a knowledge structure in my mind, so Which super category do kernel methods fall into? Since the kernel functions operate on feature vectors, I couldn't help but relate kernel methods to Feature Engineering(feature selection, feature extraction like PCA, LDA, etc.). Can I state this: kernel methods is a kind of nonlinear Feature Engineering methods?

• Terminology: data cannot be linear or nonlinear. That modifier applies to models, not to data. Jun 22 '18 at 9:06
• This search stats.stackexchange.com/search?q=feature+kernel+ gives some posts which looks very similar to yours. Can you find answer to your Q in any of them? Jun 22 '18 at 9:07
• I have changed "nonlinear data" to "linearly inseparable data". Other questions are more about specific algorithms. My questions are abstract. I did not find answers in other Questions. Jun 22 '18 at 9:26

Q2: I would say that you are absolutely correct. One can even explicitly write down the map that lies behind the kernel trick. I think that for the RBF kernel and two input dimensions the map takes some input (x,y) and transforms it to something like $(x^ay^b)_{a,b \in \mathbb{N}_0}$, i.e. take x and y and create the infinite vector of all possible monomials from it. So yes, SVM with RBF kernel = ...
2. create a linear separator in the infinite dimensional space ($l^2$)
NB2: caution with the term inseparable because in SVM there are two different countermeasures: the kernel trick and “gently penalized misclasification “ (the additional term in the cost function usually named something like $C\sum \xi_i$). These are there because perfectly separating the data in the infinite dimensional space might lead to “veeery wiggly” decision boundaries in the original space which might brutally overfit the data. The solution is to let some of the points be misclassified and make the ratio “how many points are actually bad” a hyperparameter C.