Assumptions of mixed effects model

Question

I am reading a book which has a chapter on mixed effects models, but I am a little confused by both the model and the underlying assumptions.

Suppose we have the following basic mixed effects model:

$$y_{ij}=\mu+\alpha_{i}+\epsilon_{ij},$$

for levels $i=1,...,a$, observations $j=1,...,n$, and where the $\alpha_{i} \sim N(0,\sigma^2_{\alpha})$ and $\epsilon_{ij}\sim N(0,\sigma^2_{\epsilon})$ are independent and identically distributed random effects, and the $\mu$ is a fixed effect.

My questions are as follows...

1. The random effects for different levels, $\alpha_i$ and $\alpha_j$, $i \neq j$, are modelled as identically distributed i.e. they share a common mean and variance. The goal of inference is to estimate the common variance parameter $\sigma_{a}^2$. Does this mean that we are not interested in the effect of any in particular level, but only how the effects of the levels vary?
2. In the book they talk about correlation between levels, and claim that "when there is no variation between the levels, $\sigma_{a}^2=0$, and when the variation between the levels is much larger than within the levels, $\sigma_{a}^2$ is large". I don't think I understand how there can be any variation between the levels, when both the effects $\alpha_i$ and the errors $\epsilon_{ij}$ are modelled as independent?

I don't think I understand the underlying assumptions of the model. Namely, are the distributions for the different random effects identically distributed? Are the effects for different levels independent? If I were to sample many observations from the same level, would they only vary through the $\epsilon_{ij}$ term, because $\alpha_i$ isn't indexed by $j$?

Thanks. If anyone can recommend a thorough introduction to mixed effects modelling I would be grateful.

Answer 1

Random effects model in my opinion is easiest to understand when you write down the probabilistic model behind it:

$$\begin{align} a_i &\sim \mathcal{N}(0, \, \sigma^2_a) \\ y_{ij} &\sim \mathcal{N}(\mu + a_i, \, \sigma^2_\epsilon) \end{align}$$

As you immediately see, $a_i$ and $y_{ij}$ are both random variables. In fixed effects model $a_i$ would be just "fixed" parameters like $\mu$ in the above example.

Your first question relates more to the nuances of the frequentist setting, because in Bayesian setting all the parameters are considered as random variables, so there is no distinction between fixed and random effects. Hopefully, we already have many good answers that explain it in detail, see What is the mathematical difference between random- and fixed-effects? or What is a difference between random effects-, fixed effects- and marginal model?. The biggest difference is that in fixed effects model $a_i$ are independent, "fixed" parameters, while in random effects model they are assumed to be random variables with the same distribution. In this sense we care about the variance of the distribution of those parameters. When fitting random effects models we use special methods that help us to precisely estimate the variance. You may also want to read What is the upside of treating a factor as random in a mixed model? or Fixed effect vs random effect when all possibilities are included in a mixed effects model for deeper insight on the differences between fixed and random effects, because it is quite subtle.

Answering your second question, what is meant is that $a_i$ are independent and identically distributed random variables, with variance $\sigma^2_a$. If the variance $\sigma^2_a=0$ this means that the distribution is degenerate, because it is constant, so all the $a_i$ are the same. With large $\sigma^2_a$ variance, the $a_i$ would vary a lot. It has nothing to do with $\epsilon_{ij}$ errors.

Answer 2

1. As far as I understand, we may be interested in the effect of any particular level, but we may not. I.e. it is possible that we care about the $\alpha_i$s, but it is also possible the $\alpha_i$s are nuisance parameters and we only care about $\sigma_{\alpha}^2$. This depends on the situation.

2. As you say, the distributions for different random effects are indeed (at least in the basic model) identically distributed. The effects for different levels are independent.
   And as you further note, if you sample different observations from the same level, they would only vary through $\epsilon_{ij}$. The point the book is making, is about how much observations in different groups will vary. Say we have two observations, $y_{k}$ and $y_{l}$. If they are in the same group (say $i$), then $$y_k - y_l = \alpha_i + \epsilon_k - \alpha_i - \epsilon_l = \epsilon_k - \epsilon_l \sim N(0, 2\sigma^2_{\epsilon})$$ If they are in different groups (say $i$ and $j$ respectively), then $$y_k - y_l = \alpha_i + \epsilon_k - \alpha_j - \epsilon_l \sim N(0, 2(\sigma^2_{\alpha} + \sigma^2_{\epsilon}))$$ So individuals in different groups vary more than individuals in the same group.

I find it helpful to think of in terms of how $\sigma^2_{\alpha}$ affects the variation between individuals in different groups. First let's rewrite the model in an equivalent form. Note you could equally define the model as

\begin{equation} y_{ij} = \alpha_j + \epsilon_{ij} \end{equation} where $\alpha_j \sim N(\mu,\sigma_{\alpha}^2)$ (note these $\alpha$ are different to yours as they have mean $\mu$ rather than 0). You can show that the estimate for the mean $\hat{\alpha_i}$ is given by a weighted sum of the mean of the $i$th group $\bar{y_i}$ and the population mean for the $\alpha$, which is $\mu$. So

• As $\sigma^2_{\alpha} \to 0$, the $\alpha_i \to \mu$ are all the same and individuals in different groups differ only by their error $\epsilon$, so there is no between-group variation, as if we had not included the $\alpha_i$s but set all equal to $\mu$.
• As $\sigma^2_{\alpha} \to \infty$, $\alpha_i \to \bar{y_i}$, the mean of the $i$th group. This is as if we had not modelled the $\alpha_i$s by assuming them to be normally distributed, but treated them as separate parameters in a fixed effects model.

I would recommend reading "Data Analysis Using Regression and Multilevel/Hierarchical Models" by Gelman and Hill.