I am trying to build a Bayesian optimisation model for a scientific application using Gaussian process. I am far from an expert, so my question might be naïve and full of misconceptions. I was wondering if under a Bayesian process it is possible to derive different results depending on the path of the process.
More specifically, suppose I have a training set of 150 data points. I initialize my GP with say a subset of 25 and then run the optimization. Suppose also that I use 2 acquisition functions:
- a standard one say EI
- a ‘random’ one where the next data point is picked in random.
I let the optimization to be exhaustive and in the end I measure say $R^2$ (using some test set). Is it possible to get two different $R^2$ values?
And finally: Suppose that I infer the parameters of a GP model based on the whole set of 150 (ie point estimate and no optimization at all). Is it possible to get an $R^2$ value that will be different than the values from either 1 or 2?
Just to add that I am asking if there is a difference beyond the Gaussian noise of the process