Let's say we have a one dimensional dataset of 24 points along with their responses. I am reserving three boundary points for testing (i=1,23,24) and i am fitting a Gaussian process model based on a squared exponential covariance function using GP_fit in R (optimizing the parameters by minimizing the deviance). What i am trying to do is to fit the model based on an initial number of training datapoints (let's say n = 5), evaluate the prediction accuracy, and then select the next point based on an optimal sampling criterion which selects the next training point that minimizes the uncertainty in the prediction.
The problem is that, logically, increasing your training sample size, should lead to better prediction accuracy, but this is not the case here.
For instance, Figure 1 shows the Gaussian process model based on 11 points .
Figure 2 shows the Gaussian process model based on 17 training points
The red points in the Figures are the true values of the test points, the blue line is the GP fit, while the red lines are the error bounds.
Note how the prediction accuracy in the GP model of Figure 1 (based on a lesser number of training points) is better. Quantitatively, RMSE of GPmodel1 = 0.3405, while RMSE of GPmodel2 = 0.5348.
Is this due to overfitting? because i notice that the GP model fit in both figures goes down significantly after the boundary training points, failing to predict that the original relationship is going up (convex).
If yes/no, what are possible reasons and what are practical ways to overcome this situation? Thanks.