7
$\begingroup$

So given that the coefficients of a ridge regression with a squared L2 norm penalty corresponds to the maximum a posteriori (MAP) estimate of a Bayesian regression with Gaussian priors on the coefficients and LASSO regression coefficients with L1 norm penalty to the MAP estimate of a Bayesian regression with Laplace priors on the coefficients, is there any known prior distributions in Bayesian regression that would lead to the MAP estimates being equivalent to MCP or SCAD penalized regression? Or could they be approximated by ridge or lasso regressions but with a particular choice of adaptive penalties (in the same way that LASSO can be approximated by adaptive ridge regression with adaptive penalties)? In which case they might still have some empirical Bayesian interpretation?

$\endgroup$
0

1 Answer 1

7
$\begingroup$

I see two distinct questions here:

1) Can MCP, SCAD, or other folded concave penalties with strict debiasing properties be viewed as MAP estimates wrt some prior?

No, or at least, not with respect to a proper prior. Intuitively, this is because in order to get unbiasedness, we require that the penalty be constant for large parameter values, which translates to placing constant probability density arbitrarily far from the origin, and consequently to a non-integrable (i.e. improper) prior.

To state the same information more precisely, recall that to go from a penalty $r(\beta)$ to the prior which would induce it, we exponentiate its negation $P(\beta) = e^{-r(\beta)}$. Let's consider the penalties which achieve the oracle rates described by [1]. Their unbiasedness condition on the penalty (bottom of page 3; I'll call it strict unbiasedness) is that $r'(\beta) = 0$ for $|\beta| \geq M$ for some $M$. Under suitable regularity conditions, this means that $r(|\beta|) = c$ for some constant $c$ for $|\beta|>M$. Thus:

$$\int_{\mathbb{R}} e^{-r(\beta)} d\beta\geq \int_{|\beta|>M} e^{-r(\beta)}d\beta = \int_{|\beta|>M} e^{-c}d\beta=\infty .$$

Now, just because the prior is improper doesn't necessarily stop us from using it in practice. Indeed, [3] show the results of running MCMC with a SCAD prior. So far as I can tell, they do not comment on the impropriety of the prior, nor establish the propriety of the posterior. But the impropriety certainly does mean that we lose some nice generative interpretations that proper Bayesian analysis carries.

2) Can we approximate the cost functions induced by these folded concave penalties using reweighted Lassos?

Yes, there have been proposed algorithms for fitting folded concave penalties via iteratively reweighted Lasso regressions, where each coefficient $\beta_p$ is endowed with its own penalty coefficient $\lambda_p^t$ which changes at each iteration. The working penalty at a given iteration is thus $\sum_{p}^P \lambda_p |\beta_p|$. If we wanted to match the gradient of the true penalty, we would set $\lambda_p^t = r'(\beta^{t-1})$. [2] proposes doing just one step of this, and shows that this procedure has the right asymptotics. I'm sure there's an article describing an iterative version of this with more than one step, but I can't for the life of me find it at the moment.

References

[1] https://www.jstor.org/stable/3085904

[2] https://projecteuclid.org/journals/annals-of-statistics/volume-36/issue-4/One-step-sparse-estimates-in-nonconcave-penalized-likelihood-models/10.1214/009053607000000802.full

[3] https://www.tandfonline.com/doi/abs/10.1080/03610918.2017.1280830

$\endgroup$
4
  • $\begingroup$ Would you also happen to know if the MCP penalty could be approximated by a single step ridge regression, and if so what the appropriate penalty weights would be? (Just that I am approximating L0 penalized GLMs using an iterative adaptive ridge procedure but I found that for high dimensional problems my method works better if I initialise my coefficients with the MCP solution - which now led me to look for a way to approximate that with my adaptive ridge penalties in the 1st step). $\endgroup$ Aug 10, 2023 at 15:36
  • 1
    $\begingroup$ @TomWenseleers may be more appropriate to discuss your specific case offline; I shot you an email. $\endgroup$ Aug 11, 2023 at 13:09
  • 1
    $\begingroup$ Thanks - yes I'll get back to you next week. I'll clean up my R code a bit to show you what I was working on... $\endgroup$ Aug 11, 2023 at 20:33
  • $\begingroup$ Just to come back to this - so given that the SCAD or MCP penalty can be approximated via a LASSO fit with a particular choice of adaptive penalties - I suppose that would mean that even if SCAD or MCP have improper priors & hence no strict Bayesian interpretation, an empirical Bayesian interpretation or 2 step Bayesian interpretation would still be possible right? I.e. where you start with Laplace priors on the coefs, but then do a 2nd fit with Laplace priors where the scale of the Laplace priors depend on the results of that 1st fit? $\endgroup$ Aug 13, 2023 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.