How do you apply constrains on parameters in Bayesian modeling?

Question

In Frequentist modeling, I know how to fit and interpret models with constraints on the parameter space.

Let's say I'm fitting a $N(\mu, \sigma^2)$ distribution to my data $x_1, \dots, x_n$, and I know my $\sigma^2 > 2$. I could get "maximum likelihood" estimates by maximizing $L_n(\mu, \sigma^2)$ subject to $\sigma^2 > 2$. In R, I could get parameter estimates with optim(par, fn = normal_likelihood, lower = c(-Inf, 2), upper = c(Inf, Inf).

You can imagine more complicated constraints with equality or inequality that can be fit using maximum likelihood, but the process is the same; use constrained optimization on the likelihood.

How would you fit such constraints in a Bayesian model? For the simple inequality constraint on the variance parameter, you can set a prior which excludes 2. Setting $\sigma \sim U[\sqrt{2}, \infty]$ would fit the criteria. Is this roughly the same as the constraint in the frequentist model?

What if the constraint is more complicated? How would you fit a model with likelihood $L_n(\Theta)$, where $\Theta = (\theta_1, \theta_2, \dots, \theta_P$) and you need $\theta_1 + \theta_2 + \theta_3 \leq 0$. Would you have to come up with a prior that enforces this?

Answer 1

In Bayesian setting we are dealing with posterior distribution, that is defined in terms of likelihood and priors

$$ p(\theta | X) \propto p(X | \theta) \, p(\theta) $$

If you need to constrain the parameters, you can do this by constraining the priors, or by transforming them. If you'd assume a prior that is zero for some region of the possible values of the parameters, multiplying likelihood by it would also zero-out the posterior probability over this range. Alternatively, in some cases you can transform the parameters. For example, if you have unconstrained parameters $\theta_1, \theta_2, \theta_3$, and need $\eta = \theta_1 + \theta_2 + \theta_3 \le 0$, you can transform it by taking something like $\eta = -\exp(\theta_1 + \theta_2 + \theta_3)$ would make it negative, or course there is an infinite number of other possible functions. This is easier discussed on practical, rather then abstract, examples, since in such cases it is often easier to come up with reasonable reparametrizations of the model.