2
$\begingroup$

Suppose I have the following hierarchical distribution:

$$\mathbf{y} \sim \text{Normal}(\mathbf{X}\boldsymbol{\beta} + \mathbf{K}\boldsymbol{\alpha}, \sigma^2\boldsymbol{\Sigma}_y),$$ $$\boldsymbol{\alpha} \sim \text{Normal}(\boldsymbol{0}, \sigma^2_\alpha\mathbf{I}),$$ $$\boldsymbol{\beta} \sim \text{Normal}(\boldsymbol{\mu}_\beta, \boldsymbol{\Sigma}_\beta),$$ $$\sigma^2_\alpha \sim \text{IG}(\alpha_\alpha, \beta_\alpha),$$ $$\sigma^2 \sim \text{IG}(\alpha_\sigma, \beta_\sigma),$$ where $\boldsymbol{\Sigma}_y$ is a diagonal matrix, $\mathbf{X}$ and $\mathbf{K}$ are known, and $\boldsymbol{\mu}_\beta$, $\boldsymbol{\Sigma}_\beta$, $\alpha_\alpha$, $\beta_\alpha$, $\alpha_\sigma$, and $\beta_\sigma$ are fixed hyperparameters. Further, suppose that $\mathbf{K}$ acts as a mapping matrix, where it selects two values in $\boldsymbol{\alpha}$ for each response $y_i$ (i.e., $y_i$ may have mean $\mathbf{X}_i^T\boldsymbol{\beta} + \alpha_3 + \alpha_8$).

I would like to marginalize over $\boldsymbol{\alpha}$. Is the resulting hierarchical model simply $$\mathbf{y} \sim \text{Normal}(\mathbf{X}\boldsymbol{\beta}, 2\sigma^2_\alpha\mathbf{I} + \sigma^2\boldsymbol{\Sigma}_y),$$ $$\boldsymbol{\beta} \sim \text{Normal}(\boldsymbol{\mu}_\beta, \boldsymbol{\Sigma}_\beta),$$ $$\sigma^2_\alpha \sim \text{IG}(\alpha_\alpha, \beta_\alpha),$$ $$\sigma^2 \sim \text{IG}(\alpha_\sigma, \beta_\sigma)?$$

This seems a bit too trivial, so I'm wondering if I messed up somewhere. My thinking is that, regardless of which response we have, there will be an addition of two normally distributed random variables with variance $\sigma^2_\alpha$.

$\endgroup$

1 Answer 1

2
$\begingroup$

Your solution does not account for the fact that responses $y_i$ and $y_j$ will be positively correlated if they depend on the same value $\alpha_k$.

The correct variance for $\boldsymbol{y}$ in the marginalised model is $\sigma^2_\alpha\mathbf{K}\mathbf{K}^T + \sigma^2\boldsymbol{\Sigma}_y$.

As the design matrix $\mathbf{K}$ always picks out two values from $\boldsymbol{\alpha}$, the diagonal terms in $\mathbf{K}\mathbf{K}^T$ are all equal to 2.

$\endgroup$
6
  • $\begingroup$ So, is it ok to write $2\sigma^2_\alpha\mathbf{I} = \sigma^2_\alpha\mathbf{K}\mathbf{K}^T$, or is that not always the case? $\endgroup$
    – Ron Snow
    Apr 27, 2023 at 22:18
  • 1
    $\begingroup$ Not always, no. It would only be the case if no element of $\boldsymbol{\alpha}$ appears in more than one value of the response variable. $\endgroup$ Apr 27, 2023 at 22:29
  • 2
    $\begingroup$ Try writing out $\mathbf{K}$ and evaluating $\mathbf{K} \mathbf{K}^T$ in the case where you have only two observations: $y_1$ with mean $\mathbf{X}_1^T\boldsymbol{\beta} + \alpha_1+ \alpha_2$, and $y_2$ with mean $\mathbf{X}_2^T\boldsymbol{\beta} + \alpha_2+ \alpha_3$. $\endgroup$ Apr 27, 2023 at 22:33
  • $\begingroup$ I am beginning to see why this is the case. To obtain my (wrong) answer, I was using the addition of (not!) independent random variables. What is the math that you execute to get the correct specification? I see why it's right, but I'd like to understand how I could derive that myself. $\endgroup$
    – Ron Snow
    Apr 28, 2023 at 14:48
  • 1
    $\begingroup$ I sent that message too early. I just used iterated expectation and variance properties because it's normal. I was thinking we'd need to do some cheeky integration technique, but that's not necessary here. Thank you for all of your help! $\endgroup$
    – Ron Snow
    Apr 28, 2023 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.