5
$\begingroup$

For example if we have two options to use non-linear classifier like SVM with kernel or use linear classifier like linear SVM with data preprocessing like some non-linear dimensionality reduction which one is better?

In other words why we should use a complicated classifier if we can do some data preprocessing (or "projection" if I am not mistaken it's called manifold learning) to make data linearly separable?

$\endgroup$

3 Answers 3

5
$\begingroup$

You assume that you can separate the data linearly in a lower dimensional space after some nonlinear transformation.

Kernel methods are popular due to the exact opposite: the data might only be linearly separable in a higher dimensional feature space. For any data you provide a higher dimensional feature space exists in which the data is linearly separable. Ofcourse, you don't need to go to a higher dimensional space for all problems but it is often necessary.

What you are interested in is VC dimension. The VC dimension of a model $\mathcal{F}$ is the maximum number of points that can be shattered by $\mathcal{F}$. The VC dimension of oriented hyperplanes in $\mathbb{R}^n$ is $n+1$. This shows that, for any amount of data points, a linear separation is always possible when we go to a feature space of sufficient dimensionality (but not lower!).

Kernel methods provide an efficient means to embed data in such higher dimensional feature spaces at low computational cost thanks to the kernel trick.

Explicitly transforming the data to a space in which it is linearly separable is not always possible. This is easy to see when you recall that some kernel functions, such as the popular RBF kernel, compute inner products in an infinite dimensional feature space. Obviously, you cannot create an infinite dimensional vector yourself.

$\endgroup$
1
  • 2
    $\begingroup$ ... in other words: kernels circumvent the explicit calculation of the transformed feature vector. (very nice summary of the pro-nonlinear-model points :-) ) $\endgroup$ Dec 23, 2013 at 22:31
2
$\begingroup$

If you know what kind of non-linear transformation in the preprocessing could/should give you a linearly separable problem, then do it.
This is often the case if physical/chemical/biological/... relations about the type of data are known. E.g. you may have transmittance spectra but know that $- log T = A \propto c$ (concentration). Or you have diffuse reflection data and know that Kubelka-Munk-units are worth a try.

The reason for this is that such customized pre-processing often allows to use "external" knowledge about your data structure in a more specialized and direct way than a general-purpose non-linear model.

A second advantage of this is that a consistent interpretation of the data analysis is easier with pre-processing + linear model. This can be very useful as sanity-check e.g. against knowledge from other studies about the application: Does the model use features that are already known to be related to the classes? Does the model use features that are possibly sensible. In some special cases such as the spectroscopic data I work with, I can even get indications of overfitting this way.

$\endgroup$
1
$\begingroup$

Theoretically speaking, the two approaches are equivalent.

That said, you can tweak your data preprocessing much more than the kernel parameters in most SVM tools, so the second approach will afford you much more flexibility.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.