Why does Partial Pooling occur in Mixed Effects Regressions?

Question

I am trying to create an intuitive example which shows why Mixed Effects Regression models perform Partial Pooling in the background. I previously tried to demonstrate that Mixed Effects Regressions perform Shrinkage/Regularization in the background (Why do Mixed Effects Regression models Shrink Parameter Estimates?) - now, I am trying to show that they perform Partial Pooling.

Here is my attempt to do this:

For a simple Mixed Effects Regression model:

$$y_{ij} = \alpha_j + \beta x_{ij} + \epsilon_{ij},$$

where $\alpha_j \sim N(\mu_\alpha, \sigma_\alpha^2)$

Part 1: Mixed Effects Estimation: Taking the Likelihood, we see:

$$L(\mu_\alpha, \sigma_\alpha^2, \beta, \sigma_y^2 | y, x) = \prod_{j=1}^{J} \prod_{i=1}^{n_j} f(y_{ij} | \alpha_j, \beta, x_{ij}, \sigma_y^2) g(\alpha_j | \mu_\alpha, \sigma_\alpha^2),$$

$$\log L = \sum_{j=1}^{J} \sum_{i=1}^{n_j} \left[ -\frac{1}{2\sigma_y^2} (y_{ij} - \alpha_j - \beta x_{ij})^2 - \frac{1}{2\sigma_\alpha^2} (\alpha_j - \mu_\alpha)^2 \right] + \text{constant},$$

Taking the derivative with respect to the parameters:

$$\frac{\partial \log L}{\partial \alpha_j} = \frac{1}{\sigma_y^2} \sum_{i=1}^{n_j} (y_{ij} - \alpha_j - \beta x_{ij}) - \frac{1}{\sigma_\alpha^2} (\alpha_j - \mu_\alpha) = 0$$

$$\frac{\partial \log L}{\partial \sigma_\alpha^2} = -\frac{1}{2\sigma_\alpha^4} \sum_{j=1}^{J} (\alpha_j - \mu_\alpha)^2 + \frac{J}{2\sigma_\alpha^2} = 0 $$

$$\frac{\partial \log L}{\partial \sigma_y^2} = -\frac{1}{2\sigma_y^4} \sum_{j=1}^{J} \sum_{i=1}^{n_j} (y_{ij} - \alpha_j - \beta x_{ij})^2 + \frac{Jn_j}{2\sigma_y^2} = 0$$

Solving these, we get:

$$\hat{\alpha}_j = \frac{\sigma_\alpha^2}{n_j \sigma_y^2 + \sigma_\alpha^2} \bar{y}_j + \frac{n_j \sigma_y^2}{n_j \sigma_y^2 + \sigma_\alpha^2} \mu_\alpha.$$

$$\mu_\alpha = \frac{\hat{\alpha}_j - \frac{\sigma_\alpha^2}{n_j \sigma_y^2 + \sigma_\alpha^2} \bar{y}_j}{\frac{n_j \sigma_y^2}{n_j \sigma_y^2 + \sigma_\alpha^2}}.$$

$$\hat{\sigma}_\alpha^2 = \frac{1}{J} \sum_{j=1}^{J} (\alpha_j - \mu_\alpha)^2.$$

$$\hat{\sigma}_y^2 = \frac{1}{Jn_j} \sum_{j=1}^{J} \sum_{i=1}^{n_j} (y_{ij} - \alpha_j - \beta x_{ij})^2.$$

(note: I read that $\hat{\mu}_\alpha = \frac{1}{J} \sum_{j=1}^{J} \hat{\alpha}_j$, but I am not sure why this is true)

Thus, for this simple case (assuming I did the math correctly), we can see that all mixed effect parameters "borrow" information from one another via the population mean (i.e. the population mean is calculated from information across all groups), allowing mixed effect parameters to indirectly influence each other.

Part 2: Fixed Effects Estimation: However, it seems that some partial pooling is even occurring within the Fixed Effects parameter estimation (seeing as they reference the Mixed Effects parameters):

$$\frac{\partial \log L}{\partial \beta} = \frac{1}{\sigma_y^2} \sum_{j=1}^{J} \sum_{i=1}^{n_j} x_{ij} (y_{ij} - \alpha_j - \beta x_{ij}) = 0 $$

$$\hat{\beta} = \frac{\sum_{j=1}^{J} \sum_{i=1}^{n_j} x_{ij} (y_{ij} - \hat{\alpha}_j)}{\sum_{j=1}^{J} \sum_{i=1}^{n_j} x_{ij}^2},$$

$$\hat{\sigma}_\beta^2 = -\left[ \frac{\partial^2 \log L}{\partial \beta^2} \right]^{-1}_{\beta=\hat{\beta}} = \frac{\sigma_y^2}{\sum_{j=1}^{J} \sum_{i=1}^{n_j} x_{ij}^2}.$$

Thus, for a simple Mixed Effects Regression model, we can see that the Mixed Effects Parameter Estimates do in fact use Partial Pooling in the background.

Is this the correct interpretation to understand why Mixed Effects Regression models perform Partial Pooling in the background?

Answer 1

The logic of your question is correct, but to make things more explicit, partial pooling usually refers to the "mixing" of information from the population and of the group for the group-level parameters.

This implies that

1. each group has its own parameter, $\alpha_j$ in the case you present.
2. the parameter mixes the population + group level of the same parameter. In our example its the following.

$$\hat{\alpha}_j = \frac{\sigma_\alpha^2}{n_j \sigma_y^2 + \sigma_\alpha^2} \bar{y}_j + \frac{n_j \sigma_y^2}{n_j \sigma_y^2 + \sigma_\alpha^2} \mu_\alpha.$$

The thing is, we know that intercepts are closely related to the response mean, which is why we see it there.

But for the $\beta$ parameter, you don't have a notion of "group/population" coefficient (you could, with a different model), you just have the one coefficient. So there is no pooling there, you are just centering with regards to an intercept just as you normally would.