# Mixed SVM kernel of RBF and linear

I've read some introduction about different kernels for SVM. It seems RBF is a measure of point distance while the basic kernel (i.e. no kernel) splits the space by hyper-planes.

I could imagine that for a mix of features, some features should be treated with RBF and some with the basic kernnel.

Is it possible use RBF for some features and the basic vector product for the other features?

-
I'd look into something called multiple kernel learning. –  gamerx Mar 23 '13 at 7:35

What you're describing would send your gram matrix to a much larger feature space than a single kernel. Consider the following. Given two valid mercer kernels, $\alpha_1K_1(x_i,x_j) + \alpha_2K_2(x_i, x_j)$ is a valid kernel for all $\alpha_1,\alpha_2\in\mathbb R^+$. Similarly $K_1(x_i,x_j)K_2(x_i,x_j)$ is also a valid kernel. These results imply that arbitrary polynomial expansion can be applied to kernels, allowing for interaction between kernels. You can get the desired result, at the normal speed of evaluating a gram matrix.

-
"What you're describing would send your gram matrix to a much larger feature space than a single kernel." – If you're adding kernels, it's merely additive on the dimension (concatenating the feature embeddings). If you're multiplying, though, it's much less obvious how the maps are computed and they can definitely blow up. –  Dougal May 16 at 8:22
You get the same implicit feature space of concatenating kernels as you would by directly adding them! The resulting feature space might even have a lower basis. –  Jacob Mick May 17 at 16:17
Yes, that's exactly what I meant. :) –  Dougal May 17 at 21:12
Kernels were blowing my mind when I wrote this back in 2013. I haven't actually tried multiply the output of two kernel functions, ever. I did find a good use case for controlling the differentiability of time-series modelling in gaussian processes. I found this today after reading you comment: cs.columbia.edu/~dplewis/research/thesis/slides.pdf –  Jacob Mick May 18 at 0:44

That's a really interesting idea! I've done some work with ensemble classifiers that treats different samples with a particular classifier, depending on the confidence level of each, but never at the feature level. Off the top of my head, I think it makes the most since to just do what you're describing at the classification level--classify each sample with both classifiers, and accept the answer of the most-confident one. I'm not sure how it would work at the feature level, as you describe, but I'd be interested to hear your thoughts!

Do you use a standard classification package (e.g., Weka), or have you coded your own pipeline? In my experience, these sorts of outside the box ideas are much easier to do with a system you've coded from the ground up. It gives you a much better grasp of the plumbing connecting the different components of the framework.

-

Given:

1. SVMs are similar conceptually to regression modelling

2. regression models can be fit with a mixture of locally-weighted kernels and linear features (they're called generalised additive models)

I'd say sure, go for it.

-