Fitting Gaussian process with varying sample density

I have some underlying function of parameters $$\theta_i$$ that I'm trying to minimize. I sample this function using a latin hypercube and then, using some acquisition function, I obtain successive samples which move me closer to the true minimum.

I would like to fit a Gaussian process to the samples drawn from the function but I'm running into difficulty because some of my samples will be very far apart requiring a large lengthscale for the kernel and some of the samples will be very close together requiring a much smaller lengthscale.

I tried using a sum of kernels (both exponential quad) with two different lengthscales (using GPy) and then optimizing them but I'm not sure I chose good initial values for my lengthscales. To estimate my lengthscales I computed the distance from each point to its 5 closest neighbors, fit a KDE to the distribution of distances, and then constrained each lengthscale in my sum to be between the 0 and 50th percentile and 50 and 100th percentile. This worked okay but I seemed to have more emphasis on the larger lengthscale such that I smoothed out the response near the minimum more than I wanted.

Does anyone have any suggestions for how to better estimate the lengthscales or to fit a Gaussian process to this sort of problem?

In response to a question about why I need a larger lengthscale:

In the domain where data is sparse, I want to model the function as a smoothly varying field because I don't care about fine resolution there. If I use a small lengthscale everywhere then I have regression to my mean function (zero mean) which adds complexity which is non-physical for my problem. If I use a larger lengthscale everywhere (which is what the optimizer frequently does) I treat fine variation in the domain where my data is more plentiful as noise. I know my data has, effectively, zero error so this is inappropriate.

I want to have a rough estimate of the function (large lengthscale) and then a refinement that brings in the details I'm interested in where I care about them (small lengthscale).

• Why would the large distance between samples require a large lengthscale? I don't see the issue with using just one kernel unless you believe that the function has some general large scale behavior that may be different to shorter scale quirks – InfProbSciX Apr 11 '19 at 13:35
• My point was that I want to model my underlying function as smoothly varying and have a refined, and potentially noisier, response in some localized domain of interest. If I have a small lengthscale then I get a faster regression to the mean function where my data points are sparse. Since I'm using 0 mean then this leads to undesirable behavior. If I use a larger lengthscale then I can loose the fine behavior, which I believe, in my domain of interest. I'll edit my post to make this clear. – NateM Apr 12 '19 at 14:15