I have fitted a Gaussian Process (GP) to perform a binary classification task. The dataset is balanced, so I have an equal number of samples with 0/1 label for the training. The covariance function used is an RBF kernel, which needs the hyperparameter "length scale" to be tuned.

To be sure that I am not overfitting data, and that I am selecting proper kernel hyperparameters, I performed a grid search to find out the best percentage of training data and length scale, obtaining as statistical metrics the overall accuracy (OA) and the log-marginal likelihood (LML) on the test set.

You can see the results in the following image (left for OA, right for LML):

OA and LML after fitting a GP to data

EDIT: I re-uploaded the image with the normalized log-marginal likelihood. Common sense indicates that optimal model should find a trade-off between model complexity and accuracy metrics. Thus, these models lay somewhere between 30%-40% of training data and 0.7-0.9 of length scale of the RBF kernel within the GP. This is great for model selection, but unfortunately, I think I still cannot answer the questions below... Any new insights on the interpretation of the LML?

Overall accuracy and normalized log-marginal likelihood

After exploring the effect of training size and the hyperparameter on the statistical metrics, I think it would be safe to select a model using at least 30% of data for training and a length scale for RBF of 0.1. However, I do not understand the role of LML to select the model (or even whether it needs to be considered), but common sense suggests that it should be as small as possible (i.e. around -400, represented in yellow). This means my best model is located at training size = 10-20% and length_scale=0.1.

I have seen that other people (here and here) have (somewhat) similar questions regarding LML, but I can't find ideas that help me understanding the link between good error OA metrics and LML. In other words, I am having trouble at interpreting the LML.

In concrete, I would like to get more insights on:

  1. What is the impact of a high/low LML on the predictive power of the GP?
  2. How much better is a model with LML=-400 compared to one with LML=-700?
  3. What does it mean to have a LML of -400? Isn't -400 a lot for a statistical metric?
  4. Did I really found a solution for my problem with these LML metrics?
  • 2
    $\begingroup$ Log marginal likelihood scales linearly with the number of training examples. So if you want to compare those for different numbers of training examples, you need to normalize. Furthermore, are you evaluating based on the training accuracy? If so, you really should use a held-out validation set to avoid overfitting. $\endgroup$
    – Kevin Yang
    Commented May 16, 2018 at 16:49
  • $\begingroup$ Perhaps I used a misleading label for the x-axis of the figure (only makes reference to the % of training samples used). My dataset is splitted in "train" and "test" sets, so the testing set is not shown to the model during training phase. Thus, the OA and LML metrics are obtained with the "test" set. Regarding what you said about normalizing, please, can you elaborate a bit more on that? I am not sure on how to do this normalization to compare models. Thanks! $\endgroup$
    – iamgin
    Commented May 17, 2018 at 6:42
  • 1
    $\begingroup$ How do you compute LML on the test set? $\endgroup$
    – Kevin Yang
    Commented May 17, 2018 at 7:02
  • 1
    $\begingroup$ You can normalize the LML by dividing by the size of the training set. You're adding up the log probability per training point when you calculate it. $\endgroup$
    – Kevin Yang
    Commented May 17, 2018 at 19:35
  • 1
    $\begingroup$ On the y-axis, what do you call length scale? The initial length_scale provided to the RBF kernel or the optimized one (optimization performed by the GaussianProcessRegressor)? I am surprised by the fact that any length scale would lead to the same (good) performance at the 60-70% training fraction range (except if your data can be trivially classified). $\endgroup$ Commented Sep 21, 2018 at 11:41


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.