Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 3 down vote favorite
1

I'm working on a Bayesian Regression problem where I would like to estimate the beta coefficients subject to this constraint (penalty):

$\sum|\beta_i|<C$ or similarly $\sum \beta_i^2<C$

Which is basically a Lasso or L2 Penalty.

Now, if I understand correctly, we constrain the coefficients through the prior in Bayesian analysis. Therefore my question is what would an appropriate prior be for the Betas? I should note, that for my case, the betas are restricted to be positive, they cannot be negative.

share|improve this question

4 Answers

up vote 4 down vote accepted

L2 penalty penalizes the sum of squared betas but not via a constraint such as $< C$. The L1 penalty is the lasso. For the Bayesian lasso see the 2008 JASA paper by Trevor Park and George Cassella.

share|improve this answer
Thanks, I wasn't sure what modifications I need however if I want to restrict the prior so only positive values are possible. – Glen Feb 16 '12 at 4:46
@Frank: Do you mind explaining your first comment? For each value of the regularization parameter $\lambda$, there is, indeed a constant $C$ corresponding to the constrained optimization problem of the OP. And vice versa. (Just think about the Lagrangian formulation.) – cardinal Feb 16 '12 at 15:13
My understanding of $L_2$ penalty is that large values of sum of squared $\beta$s are penalized against but there is no cutoff on how large this sum can be. – Frank Harrell Feb 17 '12 at 3:22
@FrankHarrell - this is true, but characterising the maximum $\beta$ is 1-to-1 between $C$ and $\lambda$. For any $C$ which leads to a maximum $\hat{\beta}_C$ there is a corresponding $\lambda(C)$ which leads to the same maximum $\hat{\beta}_{\lambda(C)}=\hat{\beta}_{C}$ – probabilityislogic May 6 '12 at 0:12
@prob: The correspondence is not quite bijective, unless restricted to subintervals of the positive half-line. See my previous comment to this effect above. – cardinal May 6 '12 at 0:16
up vote 3 down vote

For the lasso penalty this corresponds to a double exponential prior - so long as you are taking the posterior mode as your estimate. If you constraint the betas to be positive then you have an exponential prior. The parameter of the exponential distribution $\lambda$ has a correspondence with your $C$ in that you can choose a value of each such that the same naximum is achieved. The bayesian prior is the langrangian form (on log scale) of the constraint. For the ridge penalty constraning the parameter to be positive is just a truncated normal distribution.

share|improve this answer
up vote 2 down vote

The $L_2$ constraint on the coefficients is Tikhonov regularization.

It turns out that for the case where the prior is multivariate normal and the model is linear, the posterior is also multivariate normal. It turns out that the mean of the posterior distribution occurs at a point in parameter space that can also be obtained by Tikhonov regularization and the relationship between the Tikhonov regularization parameter and the equivalent multivariate normal prior is pretty simple.

This is textbook material that can be found in Tarantola's textbook among other places.

When the model being fit is nonlinear, or there are additional constraints on the parameters (such as your nonnegativity constraint), or the prior isn't MVN, it all becomes much more complicated and this simple equivalence breaks down.

share|improve this answer
up vote 1 down vote

If you want your $\beta$s to be non-negative and sum to a given value then it seems a scaled Dirichlet prior would make sense.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.