Bayesian Ridge and Bayesian Lasso for GLMs

Question

I had a question about Bayesian ridge and Lasso regression in the context of GLMs (and perhaps Cox models).

A lot of the resources I found online talk about Bayesian ridge and Lasso regression for OLS regression but have not found much on GLMs, unless I'm doing a poor job at Google searching.

I have heard that the MAP estimates from using a Gaussian prior is the same as ridge regression, and likewise with the Laplace prior and Lasso regression. My question was with the tuning parameter.

In a frequentist point of view, we aim to minimize the following penalized log-likelihood: $$\hat{\beta}_{pen} = \arg \min_{\beta} -l(\beta) + \lambda \sum_{j=1}^p p(\beta_j)$$.

For ridge regression, $p(\beta_j) = \beta_j^2$ and for Lasso, $p(\beta_j) = |\beta_j|$.

From my understanding, and please correct me if I'm wrong, the Ridge estimates obtained by minimizing the above function for a fixed $\lambda$ would be similar (exact?) to the MAP estimates from a $N(0, 1/\lambda)$ prior. Is this always true?

And if so, would the same hold for a $Laplace(0, 1/\lambda)$ and the Lasso estimates?

The work I've seen for OLS regression deals with conditioning on $\sigma^2$, but for GLM's this is not the case.

I would appreciate any constructive feedback and any (and all) references that can point me toward the right direction!

Answer 1

The equivalence of MAP estimates and Lasso/Ridge estimates holds by the following argument whenever we're dealing with exponential family likelihoods (which GLMs have).

You can see this by just multiplying the prior and the likelihood. For exponential-family likelihoods, we can write (from Wikipedia) \begin{equation} \pi(\mathcal{D}|\beta) = \exp\left(\sum_i\eta(\beta)T(y_i) - A(\beta) + B(y_i)\right). \end{equation}

And since the parameters are assumed independent, \begin{equation} \pi(\beta) = \prod_{j = 1}^p \pi(\beta_j). \end{equation}

So if we choose $\pi(\beta_j) \propto \exp(-\lambda p(\beta_j))$, the posterior \begin{align} \pi(\beta|\mathcal{D}) &\propto \exp\left(\sum_i\eta(\beta)T(y_i) - A(\beta) + B(y_i)\right) \prod_{j = 1}^p \exp(-\lambda p(\beta_j)) \\ &= \exp\left(\sum_i\eta(\beta)T(y_i) - A(\beta) + B(y_i) - \lambda \sum_{j = 1}^pp(\beta_j)\right) \\ &= \exp\left(\ell(\beta) - \lambda \sum_{j = 1}^pp(\beta_j)\right) \end{align}

is maximised at $\hat{\beta}_{pen}$ by definition (i.e. the MAP is $\hat{\beta}_{pen}$).

Plugging in the different choices for $p$ gets you normal/Laplace priors. Can you take it from here?

Additional note: If you are interested in Bayesian shrinkage, have a look at the horseshoe prior (Carvalho, 2010) and papers that cite that paper.