Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
For example if we have two options to use non-linear classifier like SVM with kernel or use linear classifier like linear SVM with data preprocessing like some non-linear dimensionality reduction which one is better?
In other words why we should use a complicated classifier if we can do some data preprocessing (or "projection" if I am not mistaken it's called manifold learning) to make data linearly separable?
You assume that you can separate the data linearly in a lower dimensional space after some nonlinear transformation.
Kernel methods are popular due to the exact opposite: the data might only be linearly separable in a higher dimensional feature space. For any data you provide a higher dimensional feature space exists in which the data is linearly separable. Ofcourse, you don't need to go to a higher dimensional space for all problems but it is often necessary.
What you are interested in is VC dimension. The VC dimension of a model $\mathcal{F}$ is the maximum number of points that can be shattered by $\mathcal{F}$. The VC dimension of oriented hyperplanes in $\mathbb{R}^n$ is $n+1$. This shows that, for any amount of data points, a linear separation is always possible when we go to a feature space of sufficient dimensionality (but not lower!).
Kernel methods provide an efficient means to embed data in such higher dimensional feature spaces at low computational cost thanks to the kernel trick.
Explicitly transforming the data to a space in which it is linearly separable is not always possible. This is easy to see when you recall that some kernel functions, such as the popular RBF kernel, compute inner products in an infinite dimensional feature space. Obviously, you cannot create an infinite dimensional vector yourself.
If you know what kind of non-linear transformation in the preprocessing could/should give you a linearly separable problem, then do it.
This is often the case if physical/chemical/biological/... relations about the type of data are known. E.g. you may have transmittance spectra but know that $- log T = A \propto c$ (concentration). Or you have diffuse reflection data and know that Kubelka-Munk-units are worth a try.
The reason for this is that such customized pre-processing often allows to use "external" knowledge about your data structure in a more specialized and direct way than a general-purpose non-linear model.
A second advantage of this is that a consistent interpretation of the data analysis is easier with pre-processing + linear model. This can be very useful as sanity-check e.g. against knowledge from other studies about the application: Does the model use features that are already known to be related to the classes? Does the model use features that are possibly sensible. In some special cases such as the spectroscopic data I work with, I can even get indications of overfitting this way.
Theoretically speaking, the two approaches are equivalent.
That said, you can tweak your data preprocessing much more than the kernel parameters in most SVM tools, so the second approach will afford you much more flexibility.
