3
$\begingroup$

I'm facing a problem with my GP regression, where I have (noisy) observations with repeating training inputs x.

I.e. I see observations f(x)=[1.1 1.2 3.0 2.9 4.3 4.4 4.9 5.0] for x = [1 1 2 2 3 3 4 4 5 5].
However, in my case I have 8 different training locations, each with 13 noisy observations, making a total of 104 observations.
I am unsure what to do with these duplicate training inputs/observations.

I see some posts about merging data points, since the kernel matrix inversion might get singular. Indeed I do see that the rank of my 104*104 kernel matrix is only 8, but when a noise term is added to the diagonal of the kernel (optimized with marginal likelihood) it is possible to invert the matrix.

Furthermore, when I compare the following two methods:

  1. Use all 104 observations as input to the GP,
  2. Take the mean of each different training location, making the amount of inputs to the GP 8,

I see that method 1 actually gives better performance. Could this be coincidence or does this make sense?

Thanks

$\endgroup$
1
  • 1
    $\begingroup$ Welcome to CV.SE. Your intuition that we should use all the availabe points is correct. (+1) Please see my answer below for more details. $\endgroup$
    – usεr11852
    Commented Oct 5, 2020 at 8:24

1 Answer 1

4
$\begingroup$

It makes perfect sense to use the "repeated training examples" as they endoce information about the noise in our readings.

What you observed is no coincident; the occurance of repeated $x$ instances allows us to more easily capture the noise variability $\sigma_n$. We have a very good initial estimate about how much regularisation we should consider. It also gives us as modellers a direct insight as to how much should we trust our data readings. Regarding this last point, it is worth noting that we should check that we do not have corrupted data. Indeed, having repeated $x$ instances does not inform us as to how different points $x$ covary (to estimate something like our length scale $l$) or the magnitude of that covariance (to estimate something like $\sigma_f$); the off-diagonal shape of the covariance is not directly informed but the usefulness of these readings should not be downplayed as the noisevariance $\sigma_n$ directly affects both our fitting procedure as well as the associated intervals.

$\endgroup$
3
  • $\begingroup$ Hi! Thanks for your clear answer! One remaining question though, suppose I would take the mean of the 13 'repeated training inputs' observations and use the estimated variance of these 13 values as the sigma_n. What would be the difference with this method and using all 104 data points? $\endgroup$
    – MvHaren
    Commented Oct 5, 2020 at 8:30
  • 1
    $\begingroup$ I am glad I could help. Yes, it would a different variance estimates for $\sigma_n$ because through estimating $\sigma_f$ and $l$ at the same time as $\sigma_n$ we able to account for the covariance between the points (mostly likely that $\sigma_n$ will be a bit lower than the final one). That said, it is perfectly valid to use that "13-point" $\sigma_n$ as our initial guess for the $\sigma_n$. If this answer is helpful please consider upvoting it and if it resolves your question marking it as the accepted answer. $\endgroup$
    – usεr11852
    Commented Oct 5, 2020 at 8:40
  • $\begingroup$ Thanks a lot for your rapid response and answer! This helps my project a lot :) $\endgroup$
    – MvHaren
    Commented Oct 5, 2020 at 8:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.