Bayesian lasso vs ordinary lasso

Question

Different implementation software are available for lasso. I know a lot discussed about bayesian approach vs frequentist approach in different forums. My question is very specific to lasso - What are differences or advantages of baysian lasso vs regular lasso?

Here are two example of implementation in the package:

# just example data
set.seed(1233)
X <- scale(matrix(rnorm(30),ncol=3))[,]
set.seed(12333)
Y <- matrix(rnorm(10, X%*%matrix(c(-0.2,0.5,1.5),ncol=1), sd=0.8),ncol=1)

require(monomvn) 
## Lasso regression
reg.las <- regress(X, Y, method="lasso")

## Bayesian Lasso regression
reg.blas <- blasso(X, Y)

So when should I go for one or other methods ? Or they are same ?

Answer 1

The standard lasso uses an L1 regularisation penalty to achieve sparsity in regression. Note that this is also known as Basis Pursuit (Chen & Donoho, 1994).

In the Bayesian framework, the choice of regulariser is analogous to the choice of prior over the weights. If a Gaussian prior is used, then the Maximum a Posteriori (MAP) solution will be the same as if an L2 penalty was used. Whilst not directly equivalent, the Laplace prior (which is sharply peaked around zero, unlike the Gaussian which is smooth around zero), produces the same shrinkage effect to the L1 penalty. Park & Casella (2008) describes the Bayesian Lasso.

In fact, when you place a Laplace prior over the parameters, the MAP solution should be identical (not merely similar) to regularization with the L1 penalty and the Laplace prior will produce an identical shrinkage effect to the L1 penalty. However, due to either approximations in the Bayesian inference procedure, or other numerical issues, solutions may not actually be identical.

In most cases, the results produced by both methods will be very similar. Depending on the optimisation method and whether approximations are used, the standard lasso will probably be more efficient to compute than the Bayesian version. The Bayesian automatically produces interval estimates for all of the parameters, including the error variance, if these are required.

Chen, S., & Donoho, D. (1994). Basis pursuit. In Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers (Vol. 1, pp. 41-44). IEEE. https://doi.org/10.1109/ACSSC.1994.471413

Park, T., & Casella, G. (2008). The bayesian lasso. Journal of the American Statistical Association, 103(482), 681-686. https://doi.org/10.1198/016214508000000337

Answer 2

"Least squares" means that the overall solution minimizes the sum of the squares of the errors made in the results of every single equation.The most important application is in data fitting. The best fit in the least-squares sense minimizes the sum of squared residuals, a residual being the difference between an observed value and the fitted value provided by a model.Least squares problems fall into two categories: linear or ordinary least squares and non-linear least squares, depending on whether or not the residuals are linear in all unknowns.

Bayesian linear regression is an approach to linear regression in which the statistical analysis is undertaken within the context of Bayesian inference. When the regression model has errors that have a normal distribution, and if a particular form of prior distribution is assumed, explicit results are available for the posterior probability distributions of the model's parameters.

In some contexts a regularized version of the least squares solution may be preferable. Tikhonov regularization (or ridge regression) adds a constraint that $\|\beta\|^2$, the L2-norm of the parameter vector, is not greater than a given value. In a Bayesian context, this is equivalent to placing a zero-mean normally distributed prior on the parameter vector.

An alternative regularized version of least squares is Lasso (least absolute shrinkage and selection operator), which uses the constraint that $\|\beta\|_1$, the L1-norm of the parameter vector, is no greater than a given value. In a Bayesian context, this is equivalent to placing a zero-mean Laplace prior distribution on the parameter vector.

One of the prime differences between Lasso and ridge regression is that in ridge regression, as the penalty is increased, all parameters are reduced while still remaining non-zero, while in Lasso, increasing the penalty will cause more and more of the parameters to be driven to zero.

This paper compares regular lasso with Bayesian lasso and ridge regression (see figure 1).

Answer 3

I feel the current answers to this question do not really answer the questions, which were "What are differences or advantages of baysian (sic) lasso vs regular lasso?" and "are they the same?"

First, they are not the same.

The key difference is: The Bayesian lasso attempts to sample from the full posterior distribution of the parameters, under a Laplace prior, whereas lasso is attempting to find the posterior mode (also under a Laplace prior). In practice the full posterior distribution from Bayesian lasso is usually summarized by the posterior mean, so in practice this boils down to this:

The Bayesian lasso attempts to find the posterior mean under a Laplace prior whereas the lasso attempts to find the posterior mode under a Laplace prior

The advantage of the posterior mean vs the posterior mode is that the posterior mean will produce better prediction accuracy (assuming mean squared loss) if the Laplace prior is actually a true reflection of the distribution of the regression coefficients. However, this advantage is dubious in practice since in many applications the Laplace prior is not a true reflection of the distribution of the coefficients (and in general this is difficult to check!)

The advantages of the posterior mode include that it is computationally much easier to find (it is a convex optimization problem).

You may notice that I did not answer "when should I go for one or other methods". That is because this is a hard question to answer in general. My answer would be that generally there are better methods than either of these. But full discussion of this would require a lengthier post.