0
$\begingroup$

I have a problem with many - say $D$ - input variables, $\mathbf x=(x_1,\dotsc,x_D)^\top$. I have have dataset $\mathcal D$ of $n$ input/outputs, with $n<D$. Only $\delta<<D$ should suffice to explain the response $y$. Among these $D$ dimensions, the first ones should be the most important ones... but this is not necessarily the case. The additional question being how to choose $\delta$ in that case.

I therefore want to perform a variable selection. However, I would like to use some non-linear techniques, instead of using the LASSO, Elastic Net etc.

I thought about using an additive Gaussian Process Model, to deal with each dimension separately: $Y_\text{add}(\mathbf x)=\mu+\sum_{i=1}^D\sigma^2_iY_i(x_i)$, where each $Y_i$ is a zero-mean and unit variance GP, with (correlation) kernel $k_i(x_i,x_i')$ depending on a hyperparameter $\theta_i$ (the lengthscale). This model has $2D$ hyperparameters to be estimated using maximum likelihhod, which is intractable. Furthermore, my ambition is to make the less relevant $Y_i$ vanish. This can be achieved by forcing a large amount of less relevant GPs to have a $\sigma^2_i=0$ (or also $\theta_i=\infty$)

I am therefore looking for a "Lasso-like" Maximum Likelihood for this model, that makes the irrelevant GPs vanish, by setting their $\sigma^2_i$ to 0.

\begin{equation} \underset{\substack{\theta_1,\dotsc,\theta_D\\\sigma^2_1,\dotsc,\sigma^2_D}}{\min}\mathcal{L}(Y_\text{add}(\cdot);\mathcal D)+\lambda\Vert\pmb\sigma^2\Vert_1 \end{equation}

where $\mathcal L$ is the negative log-likelihood of the additive GP,

\begin{equation} \mathcal{L}(Y_\text{add}(\cdot);\mathcal D)=\log(\vert K\vert)+(y-\mathbf1\widehat\mu)^\top K^{-1}(y-\mathbf1\widehat\mu)\end{equation}

$K=\sigma_1^2K_1+\dotsc+\sigma_D^2K_D$ is the $n\times n$ Gram matrix using the mono-dimensional (correlation) kernels $k_1,\dotsc,k_D$, $\pmb\sigma^2=(\sigma^2_1,\dotsc,\sigma^2_D)$ the vector of GP variances (if needed, I can also make the $\frac1{\theta_i}$ appear here), and $\widehat{\mu}=\frac{\mathbf 1^\top K^{-1}y}{\mathbf 1^\top K^{-1}\mathbf1}$ (I can potentially let $\mu$ off in a first trial).

My questions are the following :

1) Do you believe such a model is a good idea? Do you think about another technique to make (non-linear) variable selection? May it be Worth to stay with the "classic" Lasso, even if it is not linear?

2) How to implement this ? Have ou an idea about how to use a "Lasso-Like" method, replacing the $\frac12\Vert y-X\beta\Vert^2_2$ term by $\mathcal{L}(Y_\text{add}(\cdot);\mathcal D)$?

3) I also read about Penalized Maximum Likelihood, but which mostly penalizes the regressors of the mean function of a GP, $\beta_i$. May it be a good idea? Do you know some implementations?

Thank you very very much for your question, remarks, suggestions and help,

David

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.