# Gaussian Process regression for high dimensional data sets

Just wanted to see if anyone has any experience applying Gaussian process regression (GPR) to high dimensional data sets. I'm looking into some of the various sparse GPR methods (e.g. sparse pseudo-inputs GPR) to see what could work for high dimensional data sets where ideally feature selection is part of the parameter selection process.

Any suggestions on papers/code/or various methods to try is definitely appreciated.

Thanks.

• As stated, this question is quite vague. Questions that are self-contained, concrete and well-motivated tend to receive the most attention and best answers here. (For example, if you have a particular problem you are trying to solve, consider providing enough detail that readers can understand what you're trying to do.) – cardinal Jun 11 '12 at 23:37

Gaussian process models are generally fine with high dimensional datasets (I have used them with microarray data etc). They key is in choosing good values for the hyper-parameters (which effectively control the complexity of the model in a similar manner that regularisation does).

Sparse methods and pseudo-input methods are more for datasets with a large number of samples (> approx 4000 for my computer) rather than a large number of features. If you have a powerful enough computer to perform a Cholesky decomposition of the covariance matrix (n by n where n is the number of samples), then you probably don't need these methods.

If you are a MATLAB user, then I'd strongly recommend the GPML toolbox and the book by Rasmussen and Williams as good places to start.

HOWEVER, if you are interested in feature selection, then I would avoid GPs. The standard approach to feature selection with GPs is to use an Automatic Relevance Determination kernel (e.g. covSEard in GPML), and then achieve feature selection by tuning the kernel parameters to maximise the marginal likelihood. Unfortunately that is very likely to end up over-fitting the marginal likelihood and ending up with a model that performs (possibly much) worse than a model with a simple spherical radial basis function (covSEiso in GPML) covariance.

My current research focus lies on over-fitting in model selection at the moment and I have found that this is as much a problem for evidence maximisation in GPs as it is for cross-validation based optimisation of hyper-paraneters in kernel models, for details see this paper, and this one.

Feature selection for non-linear models is very tricky. Often you get better performance by sticking to a linear model and using L1 regularisation type approaches (Lasso/LARS/Elastic net etc.) to achieve sparsity or random forest methods.

• Thanks Dikran. I've tried looking at glmnet in R for regularized linear models. Unfortunately, my predictions end up being all the same (i think the mean of my training set). Linear models seem to have a hard time pulling out the signal in my data. That's why I've been looking for non-linear models that can deal with many features/potential feature interactions. I'm pretty sure that's asking a lot though. Any suggestions on that front? I don't have a P >> N problem. Using 150 features, 1000 examples. – tomas Jun 12 '12 at 22:08
• Hey Dikran. That was a pretty vague question I asked in my comments sorry about that. I put a more specific question up on the boards. Thanks again for your help. stats.stackexchange.com/questions/30411/… – tomas Jun 13 '12 at 20:37
• no problem, often working out what the questions are is more difficult than answering them! I'll look out for the other questions. – Dikran Marsupial Jun 14 '12 at 8:51
• Thanks for this answer. In the case of high dimensional features but not so large dataset (n~10k d~1k), is it possible to use ARD to speed up computation ? I'm using GPML toolbox. Could we automatically "sparsify" the covariance matrix to focus on relevant features ? – Emile Oct 31 '12 at 15:27
• ARD is probably going to be a bad idea if you have a large number of features as you are likely to just over-fit the marginal likelihood when optimising the parameters of the covariance function. Feature selection is generally best avoided unless identifying the relevant features is a key aim of the analysis. – Dikran Marsupial Oct 31 '12 at 17:32

You can try to use covariance functions designed specially to treat high dimensional data. Look through the paper on Additive covariance function for example. They have worked better than other state-of-the-art covariance functions in my numerical experiments with some real data of rather big input dimension (about $30$).

However, if the input dimension is really huge (more than $100$ or $200$) it seems that any kernel method will fail, and there is no exclusion for Gaussian processes regression.