15
$\begingroup$

In mixed model, we assume the random effects (parameters) are random variables that follow normal distributions. It looks very similar to the Bayesian method, in which all the parameters are assumed to be random.

So is the random effect model kind of special case of Bayesian method?

Improve this question
$\endgroup$

4 Answers 4

Reset to default
11
$\begingroup$

This is a good question. Strictly speaking, using a mixed model does not make you Bayesian. Imagine estimating each random effect separately (treating it as a fixed effect) and then looking at the resulting distribution. This is "dirty," but conceptually you have a probability distribution over the random effects based on a relative frequency concept.

But if, as a frequentist, you fit you model using full maximum likelihood and then wish to "estimate" the random effects, you've got a little complication. These quantities aren't fixed like your typical regression parameters, so a better word than "estimation" would probably be "prediction." If you want to predict a random effect for a given subject, you're going to want to use that subject's data. You'll need to resort to Bayes' rule, or at least the notion that $$f(\beta_i | \mathbf{y}_i) \propto f(\mathbf{y}_i | \beta_i) g(\beta_i).$$ Here the random effects distribution $g()$ works essentially like a prior. And I think by this point, many people would call this "empirical Bayes."

To be a true Bayesian, you would not only need to specify a distribution for your random effects, but distributions (priors) for each parameter that defines that distribution, as well distributions for all fixed effects parameters and the model epsilon. It's pretty intense!

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Really clear, straightforward answer. $\endgroup$
    – D L Dahly
    Commented Dec 15, 2013 at 10:46
  • 1
    $\begingroup$ @baogorek - a fairly robust default is Cauchy priors for fixed effects and half cauchy for variance parameters - not that "intense" - it just looks like penalised likelihood $\endgroup$
    – probabilityislogic
    Commented Dec 15, 2013 at 13:12
  • $\begingroup$ I realise this was answered almost 10 years ago, I was just a bit unsure about something from this answer. I think I'm misunderstanding something, but why, in a frequentist setting, aren't you able to estimate the random effects by estimating the parameters of the assumed distribution of the effect with ML? $\endgroup$
    – Geoff
    Commented Aug 14, 2023 at 17:39
  • 1
    $\begingroup$ @Geoff Time flies! By parameters, I assume you mean the beta_i's and not a fixed effect parameter like beta_bar (the average) and ML is maximum likelihood. If you treat the beta_i's like parameters and estimate them, then you're out of a random effects framework and back into fixed effects, aka, just a bunch of regressions. You get more flexibility by doing that but for i's with small sample sizes the beta_i estimates will be all over the place. You could add a penalty term to the likelihood to stabilize. Actually I think lme4 "estimates" the random effects in this way & backs into the rest. $\endgroup$
    – Ben Ogorek
    Commented Aug 14, 2023 at 20:14
  • 1
    $\begingroup$ @Geoff In the parameter estimation of the fixed effects parameters, you're right that the RE distributions go in the LL but then REs themselves are integrated out to get a marginal distribution (likelihood) w/o them. The "prediction" of the REs is a second step because in the LL, they weren't parameters that had to be estimated (only their mean and variance). Their predictions are interesting secondary quantities. I'm not fully following the second part of your question, but I think my throwing in the alternate estimation methods was a bit of a red herring. $\endgroup$
    – Ben Ogorek
    Commented Aug 17, 2023 at 2:39
4
$\begingroup$

Random effects are a way to specify a distributionial assumption by using conditional distributions. For example, the random one-way ANOVA model is: $$(y_{ij} \mid \mu_i) \sim_{\text{iid}} {\cal N}(\mu_i, \sigma^2_w), \quad j=1,\ldots,J, \qquad \mu_i \sim_{\text{iid}} {\cal N}(\mu, \sigma^2_b), \quad i=1,\ldots,I.$$ And this distributional assumption is equivalent to $$\begin{pmatrix} y_{i1} \\ \vdots \\ y_{iJ} \end{pmatrix} \sim_{\text{iid}} {\cal N}\left(\begin{pmatrix} \mu \\ \vdots \\ \mu \end{pmatrix}, \Sigma\right), \quad i=1,\ldots,I$$ where $\Sigma$ has an exchangeable structure (with diagonal entry $\sigma^2_b+\sigma^2_w$ and covariance $\sigma^2_b$). To Bayesianify the model, you need to assign prior distributions on $\mu$ and $\Sigma$.

Improve this answer
$\endgroup$
0
3
$\begingroup$

If you're talking in terms of reproducing the same answers, then the answer is yes. The INLA (google "inla bayesian") computational method for bayesian GLMMs combined with a uniform prior for the fixed effects and variance parameters, basically reproduces the EBLUP/EBLUE outputs under the "simple plug in" gaussian approximation, where the variance parameters are estimated via REML.

Improve this answer
$\endgroup$
0
1
$\begingroup$

I don't think so, I consider it part of the likelihood function. It's similar to specifying the error term follows a Normal distribution in a regression model, or a certain binary process can be modeled using a logistic relationship in a GLM.

Since no prior information, or distributions, are used I do not consider it Bayesian.

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ No prior information used hey? How did you specify the functional form for the likelihood function then? :-D $\endgroup$
    – probabilityislogic
    Commented Dec 15, 2013 at 12:18
  • $\begingroup$ Some people argue that the distinction between likelihood and prior is somewhat artificial. $\endgroup$
    – Christoph Hanck
    Commented Apr 7, 2015 at 4:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.