# Combining text and non-text features

I am working on a binary classification problem using SVM. I am currently using ksvm in R (kernlab package). The input is a combination of text and scores. I would like to be able to use substring counts (as in using a string kernel in SVM) and in addition, use the score. Is there a way to combine a string kernel for the text feature and a linear kernel for the numeric score? Should I build the kernel matrix externally and pass it in to the ksvm function in R?

Yes, you can combine kernels, usually through either positive linear combinations or pointwise products.

Let $k_1(x, y)$ be the kernel on only the string components of inputs $x$ and $y$, and $k_2(x, y)$ the kernel on only the score components of inputs $x$ and $y$. You can apply any kernel to these subsets; adding extra (ignored) dimensions doesn't break its kernel-ness.

Positive linear combinations: if $w_1, w_2 > 0$, then $(x, y) \mapsto w_1 k_1(x, y) + w_2 k_2(x, y)$ is a valid kernel function. If you think of kernels as a similarity function, this means that $x$ and $y$ are similar if either $k_1$ or $k_2$ thinks they are, and more similar if both say they are, with relative importances determined by the weights $w_1, w_2$.

Pointwise products: $(x, y) \mapsto k_1(x, y) \, k_2(x, y)$ is a valid kernel. This is like an "and" condition: points are similar only if both $k_1$ and $k_2$ think they're similar.

In terms of implementing this in ksvm: you can either use a matrix, or implement your own kernel function. help(ksvm) has an example of doing this (under "#### Use custom kernel").

You could also create a dictionary with the all distinct words of your dataset. Suppose you have |M|, the cardinality of your dictionary (Distinct words). There are many techniques now for creating a vector of numbers based on the dictionary, for example you can try to use one-hot encoding.

Let Alfred is the best Dungeon Master in the world be your dictionary. Now for each world you can create a one hot vector (The length of the vector is the cardinality of your vocabulary):

Alfred = [1,0,0,0,0,0,0,0]

Dungeon = [0,0,0,0,1,0,0,0]


Now since your dictionary will probably be huge. You can apply some compositionality techniques obtaining only a vector for each row of your dataset, for example you can sum up vectors associated to each input and take the mean value.