Metropolis Hastings algorithm and model selection

Question

I am going to implement Lasso regression (is described below) with metropolis hasting algorithm. Assume we have a linear model, $y=X\beta +e$ and want to estimate coefficients $\beta$ with respect to an $l_1$ (simply $\sum|\beta_i|)$ penalty term

To this end I assume a Gaussian likelihood, a double exponential prior for $\beta$ and a gamma distribution for penalty term saying, $\lambda$. the the posterior would be like $$ posterior\propto N(\mu=X\beta,\sigma=1) \times Dexp_{\beta}(\mu=0,s=\lambda)\times Gamma_{\lambda}(2,1) $$

for simplicity I assumed that the variance of error term is known and equals to 1.

above configuration results in a lasso like problem. Then we expect that some of the coefficients are exactly zero. On the other hand if I run a metropolis-hastings(MH)s algorithm, I get some estimates that are not close enough to zero. So my question is about adjusting the algorithm to give us exactly zero estimations. Notice that I know that similar algorithms like Gibbs sampling are already developed to use in the Lasso context and I am just curious to know how to adjust MH algorithm to give us exactly zero coefficients.

Answer 1

The confusion comes from the fact that Lasso is an optimisation problem, $$(\hat\beta,\hat\lambda)=\arg\min_{(\beta,\lambda)}\sum_{i=1}^n (y_i-x_i\beta)^2+\lambda|\beta|+\lambda-2\log(\lambda)$$while Metropolis-Hastings aims at simulating from the target $$\pi(\beta,\lambda)\propto\exp-\left\{\sum_{i=1}^n (y_i-x_i\beta)^2+\lambda|\beta|+\lambda-2\log(\lambda)\right\}$$producing a chain that never visits exactly the mode solution of the Lasso procedure. If you want to reach the mode, you need to modify Metropolis-Hastings into simulated annealing, i.e., aim at the target$$\pi_t(\beta,\lambda)\propto\exp-\left\{\sum_{i=1}^n (y_i-x_i\beta)^2+\lambda|\beta|+\lambda-2\log(\lambda)\right\}\Big/T_t$$at iteration $t$, with $T_t\downarrow 0$ as $t$ grows.