I have some underlying function of parameters $\theta_i$ that I'm trying to minimize. I sample this function using a latin hypercube and then, using some acquisition function, I obtain successive samples which move me closer to the true minimum.
I would like to fit a Gaussian process to the samples drawn from the function but I'm running into difficulty because some of my samples will be very far apart requiring a large lengthscale for the kernel and some of the samples will be very close together requiring a much smaller lengthscale.
I tried using a sum of kernels (both exponential quad) with two different lengthscales (using GPy) and then optimizing them but I'm not sure I chose good initial values for my lengthscales. To estimate my lengthscales I computed the distance from each point to its 5 closest neighbors, fit a KDE to the distribution of distances, and then constrained each lengthscale in my sum to be between the 0 and 50th percentile and 50 and 100th percentile. This worked okay but I seemed to have more emphasis on the larger lengthscale such that I smoothed out the response near the minimum more than I wanted.
Does anyone have any suggestions for how to better estimate the lengthscales or to fit a Gaussian process to this sort of problem?
In response to a question about why I need a larger lengthscale:
In the domain where data is sparse, I want to model the function as a smoothly varying field because I don't care about fine resolution there. If I use a small lengthscale everywhere then I have regression to my mean function (zero mean) which adds complexity which is non-physical for my problem. If I use a larger lengthscale everywhere (which is what the optimizer frequently does) I treat fine variation in the domain where my data is more plentiful as noise. I know my data has, effectively, zero error so this is inappropriate.
I want to have a rough estimate of the function (large lengthscale) and then a refinement that brings in the details I'm interested in where I care about them (small lengthscale).