6
$\begingroup$

I found this post very helpful in learning about GPC: https://krasserm.github.io/2020/11/04/gaussian-processes-classification/

Looking at the last figure in the post, I'm puzzled by the variances of the latent logit variables. I assumed the variances would be highest near the decision 0.5 line. However, this is where the uncertainty is the smallest. Why is this?

A related question. I believe the implementation in the post assumes noise free inputs. How do can I add that the outcome (0 or 1) is noisy? The goal is to prevent overfitting. I tried including a white noise kernel but not sure if it's doing anything.

$\endgroup$
2
  • 1
    $\begingroup$ there are two types of uncertainty to look out for here: one is the uncertainty we see on the Gaussian process in logit space, which is high far from data and low close to data. Since we have data near the boundary, this type of variance, which you are asking about, is low there (as that figure shows). However, even if we are 100% sure the logodds is 0 (as it is at the boundary), the variance of the outcome variable $y$ is the variance of a bernoulli variable with $p=0.5$, which has variance $0.25$, which is maximal. It is this second variance your intuition was leading you towards. $\endgroup$ Commented Sep 14, 2022 at 22:50
  • $\begingroup$ Indeed, I think we could argue that your confusion is evidence that the author plotted the wrong variance. $\endgroup$ Commented Sep 14, 2022 at 22:51

1 Answer 1

3
$\begingroup$

The confusion regarding the variances of the latent logit variables in Gaussian Process Classification (GPC), as illustrated in the last figure of the blog post, arises from the difference between two types of uncertainty: the uncertainty in the Gaussian Process (GP) in the logit space versus the uncertainty in the outcome variable $y$.

In GPC, the latent function values (logit space) are modelled as a GP, which provides a distribution over functions. The variance in this context reflects our uncertainty about the function's value at a given point in the input space. This uncertainty is typically higher far from observed data points because the GP has less information to constrain its predictions in these regions. Near observed data, especially near the decision boundary (where the predicted probability is close to 0.5), the model is more confident about its predictions because it is directly informed by the data. Thus, the variance in the latent function values is lower near the decision boundary where data points are present, as observed in the figure.

While the variance in the latent logit space might be low near the decision boundary due to the presence of data, the uncertainty in the predicted class outcome (0 or 1) is a different matter. The outcome variable in binary classification follows a Bernoulli distribution, which, when the predicted probability is 0.5, has a variance of 0.25. This variance is maximal because it reflects the highest uncertainty in the class outcome; the model is essentially saying that the outcome is as likely to be 0 as it is to be 1.

To address your second question about including noise in the outcome: adding a noise component to the GP model aims to account for noisy observations, thereby preventing overfitting. In GPC, a common way to model observation noise is through the likelihood function, rather than directly in the GP prior. For binary classification, the noise in the outcomes is implicitly modelled through the Bernoulli likelihood, which relates the latent function values to the observed binary outcomes. If you're looking to add explicit noise to the GP model, a white noise kernel can be added to the covariance function of the GP. However, in the context of classification, this might not directly influence the observed outcomes because the noise is typically considered in the process of mapping from latent function values to predicted probabilities (e.g., through the logistic function).

The effect of a white noise kernel in the context of GP classification might be subtle because it adds a constant variance to the diagonal of the GP covariance matrix, which can make the model slightly more robust to noise in the training data. However, the primary mechanism for handling uncertainty and preventing overfitting in GPC is through the specification of the covariance function (kernel) and the likelihood model, rather than through direct addition of noise to the outcomes.

To more effectively prevent overfitting and manage noise in the outcome data, you might consider approaches such as employing model selection techniques (e.g., cross-validation) to choose hyperparameters that balance model complexity and fit to the data; incorporating a more complex likelihood function that explicitly models the noise in the observed outcomes, if applicable beyond the standard Bernoulli likelihood for binary classification.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.