Gaussian Processes: How to use GPML for multi-dimensional output

Question

Is there a way to perform Gaussian Process Regression on multidimensional output (possibly correlated) using GPML?

In the demo script I could only find a 1D example.

A similar question on CV that tackles case of multidimensional input.

---

I went through their book to see if I could find anything. In the 9th chapter of this book (section 9.1), they have mentioned this case of multiple outputs. They have mentioned a couple of ways to deal with this, One - using a correlated noise process and Two - Cokriging (Correlated prior).

I still don't know, how I can incorporate any of these ideas into the GPML framework.

---

Also, are there any other GP libraries/frameworks that support multi-dimensional output?

Answer 1

I believe Twin Gaussian Processes is exactly what you are looking for. I can't describe the model better than the abstract of the paper itself, so I'm just gonna copy paste it:

We describe twin Gaussian processes (TGP) 1, a generic structured prediction method that uses Gaussian process (GP) priors [2] on both covariates and responses, both multivariate, and estimates outputs by minimizing the Kullback-Leibler divergence between two GP modeled as normal distributions over finite index sets of training and testing examples, emphasizing the goal that similar inputs should produce similar percepts and this should hold, on average, between their marginal distributions. TGP captures not only the interdependencies between covariates, as in a typical GP, but also those between responses, so correlations among both inputs and outputs are accounted for. TGP is exemplified, with promising results, for the reconstruction of 3d human poses from monocular and multicamera video sequences in the recently introduced HumanEva benchmark, where we achieve 5 cm error on average per 3d marker for models trained jointly, using data from multiple people and multiple activities. The method is fast and automatic: it requires no hand-crafting of the initial pose, camera calibration parameters, or the availability of a 3d body model associated with human subjects used for training or testing.

The authors have generously provided code and sample datasets for getting started.

Answer 2

Short answer Regression for multi-dimensional output is a little tricky and in my current level of knowledge not directly incorporated in the GPML toolbox.

Long answer You can break down your multi-dimensional output regression problem into 3 different parts.

1. Outputs are not related with each other - Just regress the outputs individually like the demo script for 1d case.
2. Outputs are related but don't know the relation between them - You would basically like to learn the inner relations between the outputs. As the book mentions coKriging is a good way to start. There are softwares other than GPML which can directly let you perform cokriging eg. ooDace
3. Outputs are related and you know the relation between them - Perform a regular cokriging but you can apply hard-constraints between the outputs either by applying the constraints in the optimizer (while you minimize the log marginal likelihood) as said by Hall & Huang 2001 or apply the relationships in the prior function as said by Constantinescu & Anitescu 2013.

I hope it helps :)

Answer 3

This is a module from scikit-learn which worked for me surprisingly good:

http://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gp_regression.html

# Instanciate a Gaussian Process model
gp = GaussianProcess(corr='cubic', theta0=1e-2, thetaL=1e-4, thetaU=1e-1,
                     random_start=100)

# Fit to data using Maximum Likelihood Estimation of the parameters
gp.fit(X, y)

# Make the prediction on the meshed x-axis (ask for MSE as well)
y_pred, MSE = gp.predict(x, eval_MSE=True)
sigma = np.sqrt(MSE)

Answer 4

I was searching for multi output Gaussian Processes and found many ways to act with it like, convolution method, mixed effects modeling method and latest this one Twin Gaussian Processes (TGP).

I have a doubt in the concept of Twin Gaussian Processes(TGP). Can anybody help me with that?

In TGP, the authors are finding out the predicted output ($\hat{y}$) minimizing the KL divergence between the input and output vice versa. But in general, we look for the predictive distribution of output i.e. $p(y^*|\mathbf{y}) \sim \mathcal(\mu, \sigma^2)$. One thing to be remarked here that the predictive variance i.e. $\sigma^2$, $y$ does not have any role in it. In the case of TGP, is the predicted output $\hat{y}$ is same as the mean of the predictive distribution of $y$?