4
$\begingroup$

I've read some introduction about different kernels for SVM. It seems RBF is a measure of point distance while the basic kernel (i.e. no kernel) splits the space by hyper-planes.

I could imagine that for a mix of features, some features should be treated with RBF and some with the basic kernnel.

Is it possible use RBF for some features and the basic vector product for the other features?

Improve this question
$\endgroup$
1
  • $\begingroup$ I'd look into something called multiple kernel learning. $\endgroup$
    – gamerx
    Mar 23, 2013 at 7:35

3 Answers 3

Reset to default
3
$\begingroup$

What you're describing would send your gram matrix to a much larger feature space than a single kernel. Consider the following. Given two valid mercer kernels, $\alpha_1K_1(x_i,x_j) + \alpha_2K_2(x_i, x_j)$ is a valid kernel for all $\alpha_1,\alpha_2\in\mathbb R^+$. Similarly $K_1(x_i,x_j)K_2(x_i,x_j)$ is also a valid kernel. These results imply that arbitrary polynomial expansion can be applied to kernels, allowing for interaction between kernels. You can get the desired result, at the normal speed of evaluating a gram matrix.

See: Classes of Kernels for Machine Learning: A Statistics Perspective

Improve this answer
$\endgroup$
4
  • $\begingroup$ "What you're describing would send your gram matrix to a much larger feature space than a single kernel." – If you're adding kernels, it's merely additive on the dimension (concatenating the feature embeddings). If you're multiplying, though, it's much less obvious how the maps are computed and they can definitely blow up. $\endgroup$
    – Danica
    May 16, 2015 at 8:22
  • $\begingroup$ You get the same implicit feature space of concatenating kernels as you would by directly adding them! The resulting feature space might even have a lower basis. $\endgroup$
    – Jessica Collins
    May 17, 2015 at 16:17
  • $\begingroup$ Yes, that's exactly what I meant. :) $\endgroup$
    – Danica
    May 17, 2015 at 21:12
  • $\begingroup$ Kernels were blowing my mind when I wrote this back in 2013. I haven't actually tried multiply the output of two kernel functions, ever. I did find a good use case for controlling the differentiability of time-series modelling in gaussian processes. I found this today after reading you comment: cs.columbia.edu/~dplewis/research/thesis/slides.pdf $\endgroup$
    – Jessica Collins
    May 18, 2015 at 0:44
1
$\begingroup$

Given:

  1. SVMs are similar conceptually to regression modelling

  2. regression models can be fit with a mixture of locally-weighted kernels and linear features (they're called generalised additive models)

I'd say sure, go for it.

Improve this answer
$\endgroup$
0
$\begingroup$

That's a really interesting idea! I've done some work with ensemble classifiers that treats different samples with a particular classifier, depending on the confidence level of each, but never at the feature level. Off the top of my head, I think it makes the most since to just do what you're describing at the classification level--classify each sample with both classifiers, and accept the answer of the most-confident one. I'm not sure how it would work at the feature level, as you describe, but I'd be interested to hear your thoughts!

Do you use a standard classification package (e.g., Weka), or have you coded your own pipeline? In my experience, these sorts of outside the box ideas are much easier to do with a system you've coded from the ground up. It gives you a much better grasp of the plumbing connecting the different components of the framework.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.