1
$\begingroup$

I am working on a binary classification problem using SVM. I am currently using ksvm in R (kernlab package). The input is a combination of text and scores. I would like to be able to use substring counts (as in using a string kernel in SVM) and in addition, use the score. Is there a way to combine a string kernel for the text feature and a linear kernel for the numeric score? Should I build the kernel matrix externally and pass it in to the ksvm function in R?

$\endgroup$
1
$\begingroup$

Yes, you can combine kernels, usually through either positive linear combinations or pointwise products.

Let $k_1(x, y)$ be the kernel on only the string components of inputs $x$ and $y$, and $k_2(x, y)$ the kernel on only the score components of inputs $x$ and $y$. You can apply any kernel to these subsets; adding extra (ignored) dimensions doesn't break its kernel-ness.

Positive linear combinations: if $w_1, w_2 > 0$, then $(x, y) \mapsto w_1 k_1(x, y) + w_2 k_2(x, y)$ is a valid kernel function. If you think of kernels as a similarity function, this means that $x$ and $y$ are similar if either $k_1$ or $k_2$ thinks they are, and more similar if both say they are, with relative importances determined by the weights $w_1, w_2$.

Pointwise products: $(x, y) \mapsto k_1(x, y) \, k_2(x, y)$ is a valid kernel. This is like an "and" condition: points are similar only if both $k_1$ and $k_2$ think they're similar.

In terms of implementing this in ksvm: you can either use a matrix, or implement your own kernel function. help(ksvm) has an example of doing this (under "#### Use custom kernel").

$\endgroup$
0
$\begingroup$

You could also create a dictionary with the all distinct words of your dataset. Suppose you have |M|, the cardinality of your dictionary (Distinct words). There are many techniques now for creating a vector of numbers based on the dictionary, for example you can try to use one-hot encoding.

Let Alfred is the best Dungeon Master in the world be your dictionary. Now for each world you can create a one hot vector (The length of the vector is the cardinality of your vocabulary):

Alfred = [1,0,0,0,0,0,0,0]

Dungeon = [0,0,0,0,1,0,0,0]

Now since your dictionary will probably be huge. You can apply some compositionality techniques obtaining only a vector for each row of your dataset, for example you can sum up vectors associated to each input and take the mean value.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.