# Additive Gaussian Processes with Penalized Likelihood

I have a problem with many - say $$D$$ - input variables, $$\mathbf x=(x_1,\dotsc,x_D)^\top$$. I have have dataset $$\mathcal D$$ of $$n$$ input/outputs, with $$n. Only $$\delta< should suffice to explain the response $$y$$. Among these $$D$$ dimensions, the first ones should be the most important ones... but this is not necessarily the case. The additional question being how to choose $$\delta$$ in that case.

I therefore want to perform a variable selection. However, I would like to use some non-linear techniques, instead of using the LASSO, Elastic Net etc.

I thought about using an additive Gaussian Process Model, to deal with each dimension separately: $$Y_\text{add}(\mathbf x)=\mu+\sum_{i=1}^D\sigma^2_iY_i(x_i)$$, where each $$Y_i$$ is a zero-mean and unit variance GP, with (correlation) kernel $$k_i(x_i,x_i')$$ depending on a hyperparameter $$\theta_i$$ (the lengthscale). This model has $$2D$$ hyperparameters to be estimated using maximum likelihhod, which is intractable. Furthermore, my ambition is to make the less relevant $$Y_i$$ vanish. This can be achieved by forcing a large amount of less relevant GPs to have a $$\sigma^2_i=0$$ (or also $$\theta_i=\infty$$)

I am therefore looking for a "Lasso-like" Maximum Likelihood for this model, that makes the irrelevant GPs vanish, by setting their $$\sigma^2_i$$ to 0.

$$$$\underset{\substack{\theta_1,\dotsc,\theta_D\\\sigma^2_1,\dotsc,\sigma^2_D}}{\min}\mathcal{L}(Y_\text{add}(\cdot);\mathcal D)+\lambda\Vert\pmb\sigma^2\Vert_1$$$$

where $$\mathcal L$$ is the negative log-likelihood of the additive GP,

$$$$\mathcal{L}(Y_\text{add}(\cdot);\mathcal D)=\log(\vert K\vert)+(y-\mathbf1\widehat\mu)^\top K^{-1}(y-\mathbf1\widehat\mu)$$$$

$$K=\sigma_1^2K_1+\dotsc+\sigma_D^2K_D$$ is the $$n\times n$$ Gram matrix using the mono-dimensional (correlation) kernels $$k_1,\dotsc,k_D$$, $$\pmb\sigma^2=(\sigma^2_1,\dotsc,\sigma^2_D)$$ the vector of GP variances (if needed, I can also make the $$\frac1{\theta_i}$$ appear here), and $$\widehat{\mu}=\frac{\mathbf 1^\top K^{-1}y}{\mathbf 1^\top K^{-1}\mathbf1}$$ (I can potentially let $$\mu$$ off in a first trial).

My questions are the following :

1) Do you believe such a model is a good idea? Do you think about another technique to make (non-linear) variable selection? May it be Worth to stay with the "classic" Lasso, even if it is not linear?

2) How to implement this ? Have ou an idea about how to use a "Lasso-Like" method, replacing the $$\frac12\Vert y-X\beta\Vert^2_2$$ term by $$\mathcal{L}(Y_\text{add}(\cdot);\mathcal D)$$?

3) I also read about Penalized Maximum Likelihood, but which mostly penalizes the regressors of the mean function of a GP, $$\beta_i$$. May it be a good idea? Do you know some implementations?

Thank you very very much for your question, remarks, suggestions and help,

David