1
$\begingroup$

I am going to implement Lasso regression (is described below) with metropolis hasting algorithm. Assume we have a linear model, $y=X\beta +e$ and want to estimate coefficients $\beta$ with respect to an $l_1$ (simply $\sum|\beta_i|)$ penalty term

To this end I assume a Gaussian likelihood, a double exponential prior for $\beta$ and a gamma distribution for penalty term saying, $\lambda$. the the posterior would be like $$ posterior\propto N(\mu=X\beta,\sigma=1) \times Dexp_{\beta}(\mu=0,s=\lambda)\times Gamma_{\lambda}(2,1) $$

for simplicity I assumed that the variance of error term is known and equals to 1.

above configuration results in a lasso like problem. Then we expect that some of the coefficients are exactly zero. On the other hand if I run a metropolis-hastings(MH)s algorithm, I get some estimates that are not close enough to zero. So my question is about adjusting the algorithm to give us exactly zero estimations. Notice that I know that similar algorithms like Gibbs sampling are already developed to use in the Lasso context and I am just curious to know how to adjust MH algorithm to give us exactly zero coefficients.

$\endgroup$

1 Answer 1

2
$\begingroup$

The confusion comes from the fact that Lasso is an optimisation problem, $$(\hat\beta,\hat\lambda)=\arg\min_{(\beta,\lambda)}\sum_{i=1}^n (y_i-x_i\beta)^2+\lambda|\beta|+\lambda-2\log(\lambda)$$while Metropolis-Hastings aims at simulating from the target $$\pi(\beta,\lambda)\propto\exp-\left\{\sum_{i=1}^n (y_i-x_i\beta)^2+\lambda|\beta|+\lambda-2\log(\lambda)\right\}$$producing a chain that never visits exactly the mode solution of the Lasso procedure. If you want to reach the mode, you need to modify Metropolis-Hastings into simulated annealing, i.e., aim at the target$$\pi_t(\beta,\lambda)\propto\exp-\left\{\sum_{i=1}^n (y_i-x_i\beta)^2+\lambda|\beta|+\lambda-2\log(\lambda)\right\}\Big/T_t$$at iteration $t$, with $T_t\downarrow 0$ as $t$ grows.

$\endgroup$
1
  • $\begingroup$ Thanks Xi'an but why $T_t \rightarrow 0$? $\endgroup$
    – TPArrow
    Oct 10, 2015 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.