I am trying to train a Gaussian process regression model based on some data with a large amount of categorical features. Each data point can be represented by a vector of strings. Right now I am using the most common RBF (or squared exponential) kernel:

$k(x_1, x_2) = \exp(-\frac{||x_1-x_2||^2)}{2\sigma^2})$

However, I don't want to explicitly encode the strings into some numerical representations, so instead I just directly compare the categorical feature vectors $x_1$ and $x_2$ between every two data points. For example, if the first element in $x_1$ is the same as the first element in $x_2$, the first element of the difference vector $x_1-x_2$ will be 0, otherwise 1. The problem arises when all element pairs are of different categories between the two vectors. In this case, the RBF kernel still returns a postive value due to exponentiation, which I very much prefer it to be 0, indicating there is 0 similarity between the two data points.

Can anyone suggest a more suitable kernel for this kind of task? Thanks in advance.

  • $\begingroup$ Why do you want to use the Gaussian process for such data? With this kind of data, it is much more common (e.g. NLP) for people to use neural networks that can generate embeddings. It is also computationally cheaper since Gaussian processes scale poorly. $\endgroup$
    – Tim
    Jul 27 at 13:23
  • 1
    $\begingroup$ Because the $y$ of these data are very expensive to get. I want to do an active learning, i.e., be able to fit the model on a small amount of data, and only add new data with high uncertainty (which is provided by GP). $\endgroup$
    – Shaun Han
    Jul 27 at 13:31
  • $\begingroup$ Why not use neural network to encode the categorical data as embeddings and pass this as features to Gaussian process then? $\endgroup$
    – Tim
    Jul 27 at 14:32
  • 1
    $\begingroup$ @Tim As I said, I want to avioud explicit encoding of categorical data, which is the main purpose of my project. This is also the main reason I need to use a kernel-based model. $\endgroup$
    – Shaun Han
    Jul 27 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.