# How can I marginalize $\boldsymbol{\alpha}$ out of my hierarchical model?

Suppose I have the following hierarchical distribution:

$$\mathbf{y} \sim \text{Normal}(\mathbf{X}\boldsymbol{\beta} + \mathbf{K}\boldsymbol{\alpha}, \sigma^2\boldsymbol{\Sigma}_y),$$ $$\boldsymbol{\alpha} \sim \text{Normal}(\boldsymbol{0}, \sigma^2_\alpha\mathbf{I}),$$ $$\boldsymbol{\beta} \sim \text{Normal}(\boldsymbol{\mu}_\beta, \boldsymbol{\Sigma}_\beta),$$ $$\sigma^2_\alpha \sim \text{IG}(\alpha_\alpha, \beta_\alpha),$$ $$\sigma^2 \sim \text{IG}(\alpha_\sigma, \beta_\sigma),$$ where $$\boldsymbol{\Sigma}_y$$ is a diagonal matrix, $$\mathbf{X}$$ and $$\mathbf{K}$$ are known, and $$\boldsymbol{\mu}_\beta$$, $$\boldsymbol{\Sigma}_\beta$$, $$\alpha_\alpha$$, $$\beta_\alpha$$, $$\alpha_\sigma$$, and $$\beta_\sigma$$ are fixed hyperparameters. Further, suppose that $$\mathbf{K}$$ acts as a mapping matrix, where it selects two values in $$\boldsymbol{\alpha}$$ for each response $$y_i$$ (i.e., $$y_i$$ may have mean $$\mathbf{X}_i^T\boldsymbol{\beta} + \alpha_3 + \alpha_8$$).

I would like to marginalize over $$\boldsymbol{\alpha}$$. Is the resulting hierarchical model simply $$\mathbf{y} \sim \text{Normal}(\mathbf{X}\boldsymbol{\beta}, 2\sigma^2_\alpha\mathbf{I} + \sigma^2\boldsymbol{\Sigma}_y),$$ $$\boldsymbol{\beta} \sim \text{Normal}(\boldsymbol{\mu}_\beta, \boldsymbol{\Sigma}_\beta),$$ $$\sigma^2_\alpha \sim \text{IG}(\alpha_\alpha, \beta_\alpha),$$ $$\sigma^2 \sim \text{IG}(\alpha_\sigma, \beta_\sigma)?$$

This seems a bit too trivial, so I'm wondering if I messed up somewhere. My thinking is that, regardless of which response we have, there will be an addition of two normally distributed random variables with variance $$\sigma^2_\alpha$$.

## 1 Answer

Your solution does not account for the fact that responses $$y_i$$ and $$y_j$$ will be positively correlated if they depend on the same value $$\alpha_k$$.

The correct variance for $$\boldsymbol{y}$$ in the marginalised model is $$\sigma^2_\alpha\mathbf{K}\mathbf{K}^T + \sigma^2\boldsymbol{\Sigma}_y$$.

As the design matrix $$\mathbf{K}$$ always picks out two values from $$\boldsymbol{\alpha}$$, the diagonal terms in $$\mathbf{K}\mathbf{K}^T$$ are all equal to 2.

• So, is it ok to write $2\sigma^2_\alpha\mathbf{I} = \sigma^2_\alpha\mathbf{K}\mathbf{K}^T$, or is that not always the case? Apr 27, 2023 at 22:18
• Not always, no. It would only be the case if no element of $\boldsymbol{\alpha}$ appears in more than one value of the response variable. Apr 27, 2023 at 22:29
• Try writing out $\mathbf{K}$ and evaluating $\mathbf{K} \mathbf{K}^T$ in the case where you have only two observations: $y_1$ with mean $\mathbf{X}_1^T\boldsymbol{\beta} + \alpha_1+ \alpha_2$, and $y_2$ with mean $\mathbf{X}_2^T\boldsymbol{\beta} + \alpha_2+ \alpha_3$. Apr 27, 2023 at 22:33
• I am beginning to see why this is the case. To obtain my (wrong) answer, I was using the addition of (not!) independent random variables. What is the math that you execute to get the correct specification? I see why it's right, but I'd like to understand how I could derive that myself. Apr 28, 2023 at 14:48
• I sent that message too early. I just used iterated expectation and variance properties because it's normal. I was thinking we'd need to do some cheeky integration technique, but that's not necessary here. Thank you for all of your help! Apr 28, 2023 at 14:55