Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Just wanted to see if anyone has any experience applying Gaussian process regression (GPR) to high dimensional data sets. I'm looking into some of the various sparse GPR methods (e.g. sparse pseudo-inputs GPR) to see what could work for high dimensional data sets where ideally feature selection is part of the parameter selection process.
Any suggestions on papers/code/or various methods to try is definitely appreciated.
Thanks.
Gaussian process models are generally fine with high dimensional datasets (I have used them with microarray data etc). They key is in choosing good values for the hyper-parameters (which effectively control the complexity of the model in a similar manner that regularisation does).
Sparse methods and pseudo-input methods are more for datasets with a large number of samples (> approx 4000 for my computer) rather than a large number of features. If you have a powerful enough computer to perform a Cholesky decomposition of the covariance matrix (n by n where n is the number of samples), then you probably don't need these methods.
If you are a MATLAB user, then I'd strongly recommend the GPML toolbox and the book by Rasmussen and Williams as good places to start.
HOWEVER, if you are interested in feature selection, then I would avoid GPs. The standard approach to feature selection with GPs is to use an Automatic Relevance Determination kernel (e.g. covSEard in GPML), and then achieve feature selection by tuning the kernel parameters to maximise the marginal likelihood. Unfortunately that is very likely to end up over-fitting the marginal likelihood and ending up with a model that performs (possibly much) worse than a model with a simple spherical radial basis function (covSEiso in GPML) covariance.
My current research focus lies on over-fitting in model selection at the moment and I have found that this is as much a problem for evidence maximisation in GPs as it is for cross-validation based optimisation of hyper-paraneters in kernel models, for details see this paper, and this one.
Feature selection for non-linear models is very tricky. Often you get better performance by sticking to a linear model and using L1 regularisation type approaches (Lasso/LARS/Elastic net etc.) to achieve sparsity or random forest methods.
You can try to use covariance functions designed specially to treat high dimensional data. Look through the paper on Additive covariance function for example. They have worked better than other state-of-the-art covariance functions in my numerical experiments with some real data of rather big input dimension (about $30$).
However, if the input dimension is really huge (more than $100$ or $200$) it seems that any kernel method will fail, and there is no exclusion for Gaussian processes regression.
Something you can try is to use Gaussian process regressors (GPRs) as the base regressors of a Bagging Regressor. Each base regressor will be trained on a subset of the samples and the features such that the dimensionality of the training data is drastically reduced for each regressor. I have observed that doing so increases the speed and the accuracy because it may be faster to train multiple GPRs on subsets of the data than one GPR on the whole high-dimensional data. Moreover, aggregating the outcomes of multiple weak GPRs improves performance by mitigating variance.
