# Gaussian Process regression for high dimensional data sets

Just wanted to see if anyone has any experience applying Gaussian process regression (GPR) to high dimensional data sets. I'm looking into some of the various sparse GPR methods (e.g. sparse pseudo-inputs GPR) to see what could work for high dimensional data sets where ideally feature selection is part of the parameter selection process.

Any suggestions on papers/code/or various methods to try is definitely appreciated.

Thanks.

• As stated, this question is quite vague. Questions that are self-contained, concrete and well-motivated tend to receive the most attention and best answers here. (For example, if you have a particular problem you are trying to solve, consider providing enough detail that readers can understand what you're trying to do.) Jun 11, 2012 at 23:37

Gaussian process models are generally fine with high dimensional datasets (I have used them with microarray data etc). They key is in choosing good values for the hyper-parameters (which effectively control the complexity of the model in a similar manner that regularisation does).

Sparse methods and pseudo-input methods are more for datasets with a large number of samples (> approx 4000 for my computer) rather than a large number of features. If you have a powerful enough computer to perform a Cholesky decomposition of the covariance matrix (n by n where n is the number of samples), then you probably don't need these methods.

If you are a MATLAB user, then I'd strongly recommend the GPML toolbox and the book by Rasmussen and Williams as good places to start.

HOWEVER, if you are interested in feature selection, then I would avoid GPs. The standard approach to feature selection with GPs is to use an Automatic Relevance Determination kernel (e.g. covSEard in GPML), and then achieve feature selection by tuning the kernel parameters to maximise the marginal likelihood. Unfortunately that is very likely to end up over-fitting the marginal likelihood and ending up with a model that performs (possibly much) worse than a model with a simple spherical radial basis function (covSEiso in GPML) covariance.

My current research focus lies on over-fitting in model selection at the moment and I have found that this is as much a problem for evidence maximisation in GPs as it is for cross-validation based optimisation of hyper-paraneters in kernel models, for details see this paper, and this one.

Feature selection for non-linear models is very tricky. Often you get better performance by sticking to a linear model and using L1 regularisation type approaches (Lasso/LARS/Elastic net etc.) to achieve sparsity or random forest methods.

• Thanks Dikran. I've tried looking at glmnet in R for regularized linear models. Unfortunately, my predictions end up being all the same (i think the mean of my training set). Linear models seem to have a hard time pulling out the signal in my data. That's why I've been looking for non-linear models that can deal with many features/potential feature interactions. I'm pretty sure that's asking a lot though. Any suggestions on that front? I don't have a P >> N problem. Using 150 features, 1000 examples. Jun 12, 2012 at 22:08
• Hey Dikran. That was a pretty vague question I asked in my comments sorry about that. I put a more specific question up on the boards. Thanks again for your help. stats.stackexchange.com/questions/30411/… Jun 13, 2012 at 20:37
• no problem, often working out what the questions are is more difficult than answering them! I'll look out for the other questions. Jun 14, 2012 at 8:51
• Thanks for this answer. In the case of high dimensional features but not so large dataset (n~10k d~1k), is it possible to use ARD to speed up computation ? I'm using GPML toolbox. Could we automatically "sparsify" the covariance matrix to focus on relevant features ? Oct 31, 2012 at 15:27
• the "r.csail.mit.edu/papers/v8/cawley07a.html" link is not working... Is it this one? jmlr.org/papers/v8/cawley07a.html . Perhaps adding full citations instead of just links would be beneficial :-) Mar 28, 2020 at 10:42

You can try to use covariance functions designed specially to treat high dimensional data. Look through the paper on Additive covariance function for example. They have worked better than other state-of-the-art covariance functions in my numerical experiments with some real data of rather big input dimension (about $30$).

However, if the input dimension is really huge (more than $100$ or $200$) it seems that any kernel method will fail, and there is no exclusion for Gaussian processes regression.

Something you can try is to use Gaussian process regressors (GPRs) as the base regressors of a Bagging Regressor. Each base regressor will be trained on a subset of the samples and the features such that the dimensionality of the training data is drastically reduced for each regressor. I have observed that doing so increases the speed and the accuracy because it may be faster to train multiple GPRs on subsets of the data than one GPR on the whole high-dimensional data. Moreover, aggregating the outcomes of multiple weak GPRs improves performance by mitigating variance.