I have fitted a Gaussian Process (GP) to perform a binary classification task. The dataset is balanced, so I have an equal number of samples with 0/1 label for the training. The covariance function used is an RBF kernel, which needs the hyperparameter "length scale" to be tuned.

To be sure that I am not overfitting data, and that I am selecting proper kernel hyperparameters, I performed a grid search to find out the best percentage of training data and length scale, obtaining as statistical metrics the overall accuracy (OA) and the log-marginal likelihood (LML) on the test set.

You can see the results in the following image (left for OA, right for LML):

OA and LML after fitting a GP to data

EDIT: I re-uploaded the image with the normalized log-marginal likelihood. Common sense indicates that optimal model should find a trade-off between model complexity and accuracy metrics. Thus, these models lay somewhere between 30%-40% of training data and 0.7-0.9 of length scale of the RBF kernel within the GP. This is great for model selection, but unfortunately, I think I still cannot answer the questions below... Any new insights on the interpretation of the LML?

Overall accuracy and normalized log-marginal likelihood

After exploring the effect of training size and the hyperparameter on the statistical metrics, I think it would be safe to select a model using at least 30% of data for training and a length scale for RBF of 0.1. However, I do not understand the role of LML to select the model (or even whether it needs to be considered), but common sense suggests that it should be as small as possible (i.e. around -400, represented in yellow). This means my best model is located at training size = 10-20% and length_scale=0.1.

I have seen that other people (here and here) have (somewhat) similar questions regarding LML, but I can't find ideas that help me understanding the link between good error OA metrics and LML. In other words, I am having trouble at interpreting the LML.

In concrete, I would like to get more insights on:

  1. What is the impact of a high/low LML on the predictive power of the GP?
  2. How much better is a model with LML=-400 compared to one with LML=-700?
  3. What does it mean to have a LML of -400? Isn't -400 a lot for a statistical metric?
  4. Did I really found a solution for my problem with these LML metrics?

Thanks for your help!

  • 1
    $\begingroup$ Log marginal likelihood scales linearly with the number of training examples. So if you want to compare those for different numbers of training examples, you need to normalize. Furthermore, are you evaluating based on the training accuracy? If so, you really should use a held-out validation set to avoid overfitting. $\endgroup$ – Kevin Yang May 16 '18 at 16:49
  • $\begingroup$ Perhaps I used a misleading label for the x-axis of the figure (only makes reference to the % of training samples used). My dataset is splitted in "train" and "test" sets, so the testing set is not shown to the model during training phase. Thus, the OA and LML metrics are obtained with the "test" set. Regarding what you said about normalizing, please, can you elaborate a bit more on that? I am not sure on how to do this normalization to compare models. Thanks! $\endgroup$ – iamgin May 17 '18 at 6:42
  • 1
    $\begingroup$ How do you compute LML on the test set? $\endgroup$ – Kevin Yang May 17 '18 at 7:02
  • 1
    $\begingroup$ You can normalize the LML by dividing by the size of the training set. You're adding up the log probability per training point when you calculate it. $\endgroup$ – Kevin Yang May 17 '18 at 19:35
  • 1
    $\begingroup$ On the y-axis, what do you call length scale? The initial length_scale provided to the RBF kernel or the optimized one (optimization performed by the GaussianProcessRegressor)? I am surprised by the fact that any length scale would lead to the same (good) performance at the 60-70% training fraction range (except if your data can be trivially classified). $\endgroup$ – Romain Reboulleau Sep 21 '18 at 11:41

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.