I'm currently trying to select the optimal points for a Gaussian Process Regression, and the important thing is that i already know the whole target function.
Therefore, it's not Online Learning because I'm not trying to find the optimal points that will reduce the uncertainty of my prediction (see Bayesian Optimization with Expected Improvement and Probability of Improvement), but i try to look afterwards what are the points that mostly matters in a time series prediction to have a good accuracy.
Several theories have been established to select relevant points in order to reduce the computational cost (Sparse Gaussian Process with Inducing Input), but once again, those technics have been developed to select points before the training of the GP to reduce its training time. In my case, I can train my GP on the whole time series, I just want to know if I had to reduce the number of observed points, what will be the optimal ones (the ones that generate the best GP with respect to the MSE for example)?
If we want to select 10 points for example, my first idea was the following :
- Find the posterior of the GP when we use all the training points we have : let's call him GP1
- Select a random point at the beginning and add this point to a list L
- For i in range(10)
- For new_point not in L :
- compute the posterior of the GP with all the points in L + the new point
- compute the KL divergence between the founded GP and the one with all the training points(GP1)
- select the point that has generated the GP with the lowest KL divergence and add this point to L
The problem is that it's really really expensive in terms of computational cost and that it's not the best idea we can have. Have you a better one?
(Please don't tell me to compute the loss of all the possible subset of k elements from a fixed set of n elements, of course it's impossible...)
It seems to be related to Data Point Relevance/Importance...
Thank you so much for your help because it seems that there is no documentation about this problem!
