1
$\begingroup$

I'm currently trying to select the optimal points for a Gaussian Process Regression, and the important thing is that i already know the whole target function.

Therefore, it's not Online Learning because I'm not trying to find the optimal points that will reduce the uncertainty of my prediction (see Bayesian Optimization with Expected Improvement and Probability of Improvement), but i try to look afterwards what are the points that mostly matters in a time series prediction to have a good accuracy.

Several theories have been established to select relevant points in order to reduce the computational cost (Sparse Gaussian Process with Inducing Input), but once again, those technics have been developed to select points before the training of the GP to reduce its training time. In my case, I can train my GP on the whole time series, I just want to know if I had to reduce the number of observed points, what will be the optimal ones (the ones that generate the best GP with respect to the MSE for example)?

If we want to select 10 points for example, my first idea was the following :

  • Find the posterior of the GP when we use all the training points we have : let's call him GP1
  • Select a random point at the beginning and add this point to a list L
  • For i in range(10)
    • For new_point not in L :
      • compute the posterior of the GP with all the points in L + the new point
      • compute the KL divergence between the founded GP and the one with all the training points(GP1)
    • select the point that has generated the GP with the lowest KL divergence and add this point to L

The problem is that it's really really expensive in terms of computational cost and that it's not the best idea we can have. Have you a better one?

(Please don't tell me to compute the loss of all the possible subset of k elements from a fixed set of n elements, of course it's impossible...)

It seems to be related to Data Point Relevance/Importance...

Thank you so much for your help because it seems that there is no documentation about this problem!

$\endgroup$
1
$\begingroup$

You could look at ideas of optimal experimental design for Gaussian processes, which should be simplified in your case because you already have an estimated model. The data points in your data could then be used as possible points for a design, and you could use some exchange algorithm to search for an optimal subset. A related question is Best DoE method to fit Gaussian Process Regressor.

Some published papers is this, and this one.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.