# Bayesian lasso vs ordinary lasso

Different implementation software are available for lasso. I know a lot discussed about bayesian approach vs frequentist approach in different forums. My question is very specific to lasso - What are differences or advantages of baysian lasso vs regular lasso?

Here are two example of implementation in the package:

# just example data
set.seed(1233)
X <- scale(matrix(rnorm(30),ncol=3))[,]
set.seed(12333)
Y <- matrix(rnorm(10, X%*%matrix(c(-0.2,0.5,1.5),ncol=1), sd=0.8),ncol=1)

require(monomvn)
## Lasso regression
reg.las <- regress(X, Y, method="lasso")

## Bayesian Lasso regression
reg.blas <- blasso(X, Y)


So when should I go for one or other methods ? Or they are same ?

The standard lasso uses an L1 regularisation penalty to achieve sparsity in regression. Note that this is also known as Basis Pursuit.

In the Bayesian framework, the choice of regulariser is analogous to the choice of prior over the weights. If a Gaussian prior is used, then the Maximum a Posteriori (MAP) solution will be the same as if an L2 penalty was used. Whilst not directly equivalent, the Laplace prior (which is sharply peaked around zero, unlike the Gaussian which is smooth around zero), produces the same shrinkage effect to the L1 penalty. This paper describes the Bayesian Lasso..

In fact, when you place a Laplace prior over the parameters, the MAP solution should be identical (not merely similar) to regularization with the L1 penalty and the Laplace prior will produce an identical shrinkage effect to the L1 penalty. However, due to either approximations in the Bayesian inference procedure, or other numerical issues, solutions may not actually be identical.

In most cases, the results produced by both methods will be very similar. Depending on the optimisation method and whether approximations are used, the standard lasso will probably be more efficient to compute than the Bayesian version. The Bayesian automatically produces interval estimates for all of the parameters, including the error variance, if these are required.

• "If a Gaussian prior is used, then the Maximum Likelihood solution will be the same ....". The highlighted phrase should read "Maximum A Posteriori (MAP)" because Maximum Likelihood estimation will just ignore the prior distribution over the parameters, leading to an unregularized solution whereas MAP estimation takes the prior into consideration. – mefathy Sep 28 '15 at 15:04
• When you place a Laplace prior over the parameters, the MAP solution will be identical (not merely similar) to regularization with the L1 penalty and the Laplace prior will produce an identical shrinkage effect to the L1 penalty. – mefathy Sep 28 '15 at 15:06
• @mefathy yes you're right on both counts (can't believe I wrote ML instead of MAP ....), although of course in practise YMMV. I've updated the answer to incorporate both comments. – tdc Oct 12 '15 at 9:16

"Least squares" means that the overall solution minimizes the sum of the squares of the errors made in the results of every single equation.The most important application is in data fitting. The best fit in the least-squares sense minimizes the sum of squared residuals, a residual being the difference between an observed value and the fitted value provided by a model.Least squares problems fall into two categories: linear or ordinary least squares and non-linear least squares, depending on whether or not the residuals are linear in all unknowns.

Bayesian linear regression is an approach to linear regression in which the statistical analysis is undertaken within the context of Bayesian inference. When the regression model has errors that have a normal distribution, and if a particular form of prior distribution is assumed, explicit results are available for the posterior probability distributions of the model's parameters.

In some contexts a regularized version of the least squares solution may be preferable. Tikhonov regularization (or ridge regression) adds a constraint that $\|\beta\|^2$, the L2-norm of the parameter vector, is not greater than a given value. In a Bayesian context, this is equivalent to placing a zero-mean normally distributed prior on the parameter vector.

An alternative regularized version of least squares is Lasso (least absolute shrinkage and selection operator), which uses the constraint that $\|\beta\|_1$, the L1-norm of the parameter vector, is no greater than a given value. In a Bayesian context, this is equivalent to placing a zero-mean Laplace prior distribution on the parameter vector.

One of the prime differences between Lasso and ridge regression is that in ridge regression, as the penalty is increased, all parameters are reduced while still remaining non-zero, while in Lasso, increasing the penalty will cause more and more of the parameters to be driven to zero.

This paper compares regular lasso with Bayesian lasso and ridge regression (see figure 1).