Partial plots for Gaussian Process: definition and computation

Question

How do I make partial plot (or partial dependence plot) for Gaussian Process classification? I mean plot which would show a dependence of a response probability to a single covariate variable?

Partial plots for non-linear models can be very tricky as I found. The SW package I am using only provides plots which fix the other covariates on a single fixed value (e.g., an average value of the covariates, or centroid of the "niche", peak of the response, etc.) . But I found that it depends a lot on where you fix those values. For illustration, here is the example from my GP when I fix the other covariates on (a) mean, (b) median niche (i.e. where y = 1), (c) peak (i.e. max E(y)):

As you can see, these are almost 3 different plots; thanks to the non-linear multi-dimensional nature of the model. It depends a lot on where you fix those values, and not a single one is guaranteed to give you a "general idea" of how the covariate works.

Thus, I think it only makes sense to average (integrate) the predictions over the distribution of the values of all the other covariates, given the particular value of the covariate of interest. In a similar manner how Gelman & Pardoe 2007 calculate APC:

$$\mbox{partialPlot}(u_1) = \int dv \int d\theta ~ \mbox{E}(y|u_1,v,\theta) \cdot p(v|u_1) \cdot p(\theta)$$

where $u$ is the covariate for which we plot the partial dependence plot, for a single value of $u_1$; $v$ are all the other covariates, $\theta$ are hyper-parameters. $\mbox{E}(y|u_1,v,\theta)$ is a single prediction.

But not only this, I would also like to know the variance of that prediction (confidence interval).

Questions:

1. How are partial plots usually done in Gaussian Processes? Have you come across a better way than fixing the other covariates on a single value?
2. Is my definition reasonable for partial plots in GP? Or, does it exist and already have a name?
3. Is there any very simple R package, or a simple algorithm description, to implement a reasonable partial plot for Gaussian process? Gelman & Pardoe describe some algorithm using Mahalanobis distances but to implement this using the paper seems too complicated for me now. I guess someone had to do it already!

---

PS: please try to resist to write too long and too super-scientific answer; I will probably never have time (and ability) to read and understand. I got many such answers here and it's a pitty to see so much effort going another direction. Please as simple as possible, my knowledge and time for this is limited. Thanks! :-)

Answer 1

I was also looking for partial plots for GPs. The 'brute' method described by scikit is a generic method that works with any estimator. This could be eventually extended to ALE plots.

Then you could also do kriging. It's not the same as partial dependence plots, but nevertheless it can show you dependence of a response probability to a single covariate variable. The other thing is using Automatic Relevance Determination (ARD), which creates separate length-scale parameters for each input. The inverse of each length-scale should give you the predictive relevance of each corresponding input.

Sorry, that I couldn't directly answer your questions. Nevertheless, I shared my thoughts. In case you have some updates, please let us know.

Answer 2

Unless you want to visualize a scenario when they are fixed to particular values, you basically need to integrate out the other dimensions. The simplest, generally applicable method for approximating integrals is Monte Carlo. Gaussian Process is a probabilistic model, you can easily sample from it. To approximate the integral, just take many samples and average the dimensions you don't care about.