I have a problem with many - say $D$ - input variables, $\mathbf x=(x_1,\dotsc,x_D)^\top$. I have have dataset $\mathcal D$ of $n$ input/outputs, with $n<D$. Only $\delta<<D$ should suffice to explain the response $y$. Among these $D$ dimensions, the first ones should be the most important ones... but this is not necessarily the case. The additional question being how to choose $\delta$ in that case.

I therefore want to perform a variable selection. However, I would like to use some non-linear techniques, instead of using the LASSO, Elastic Net etc.

I thought about using an additive Gaussian Process Model, to deal with each dimension separately: $Y_\text{add}(\mathbf x)=\mu+\sum_{i=1}^D\sigma^2_iY_i(x_i)$, where each $Y_i$ is a zero-mean and unit variance GP, with (correlation) kernel $k_i(x_i,x_i')$ depending on a hyperparameter $\theta_i$ (the lengthscale). This model has $2D$ hyperparameters to be estimated using maximum likelihhod, which is intractable. Furthermore, my ambition is to make the less relevant $Y_i$ vanish. This can be achieved by forcing a large amount of less relevant GPs to have a $\sigma^2_i=0$ (or also $\theta_i=\infty$)

I am therefore looking for a "Lasso-like" Maximum Likelihood for this model, that makes the irrelevant GPs vanish, by setting their $\sigma^2_i$ to 0.

\begin{equation} \underset{\substack{\theta_1,\dotsc,\theta_D\\\sigma^2_1,\dotsc,\sigma^2_D}}{\min}\mathcal{L}(Y_\text{add}(\cdot);\mathcal D)+\lambda\Vert\pmb\sigma^2\Vert_1 \end{equation}

where $\mathcal L$ is the negative log-likelihood of the additive GP,

\begin{equation} \mathcal{L}(Y_\text{add}(\cdot);\mathcal D)=\log(\vert K\vert)+(y-\mathbf1\widehat\mu)^\top K^{-1}(y-\mathbf1\widehat\mu)\end{equation}

$K=\sigma_1^2K_1+\dotsc+\sigma_D^2K_D$ is the $n\times n$ Gram matrix using the mono-dimensional (correlation) kernels $k_1,\dotsc,k_D$, $\pmb\sigma^2=(\sigma^2_1,\dotsc,\sigma^2_D)$ the vector of GP variances (if needed, I can also make the $\frac1{\theta_i}$ appear here), and $\widehat{\mu}=\frac{\mathbf 1^\top K^{-1}y}{\mathbf 1^\top K^{-1}\mathbf1}$ (I can potentially let $\mu$ off in a first trial).

My questions are the following :

1) Do you believe such a model is a good idea? Do you think about another technique to make (non-linear) variable selection? May it be Worth to stay with the "classic" Lasso, even if it is not linear?

2) How to implement this ? Have ou an idea about how to use a "Lasso-Like" method, replacing the $\frac12\Vert y-X\beta\Vert^2_2$ term by $\mathcal{L}(Y_\text{add}(\cdot);\mathcal D)$?

3) I also read about Penalized Maximum Likelihood, but which mostly penalizes the regressors of the mean function of a GP, $\beta_i$. May it be a good idea? Do you know some implementations?

Thank you very very much for your question, remarks, suggestions and help,



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.