BLUP formula for random coefficient models, in form that allows for convenient derivatives WRT loss function

In Semiparametric Regression, Ruppert et al assert that the BLUP for $U_i$ in the model

$$\begin{equation} y_{it} = \beta_0 + U_i + \beta_1x_{it} + \epsilon_{it} \end{equation}$$

reduces to $$\tilde{U}_i = \frac{n_i\sigma_u^2}{\sigma_\epsilon^2+n_i\sigma_u^2}\left(\bar{y}_i-\bar{x}_i\beta\right)$$ They cite this book, which I don't have access to...

How is this derived? I can see how it simplifies to the econometric fixed effects estimator when $\sigma_u \rightarrow \infty$, because the fraction would approach 1, which would make the individual intercept equal to the group-wise average residual.

This formulation has a convenient derivative of the variance terms WRT to a (L2-penalizable) squared-error loss function. Assuming I remember my calculus, it is $$\frac{\partial L}{\partial \sigma^2_u} = -2\hat\epsilon(\bar{y} - \bar{x}_i\beta)\left(\frac{n_i}{\sigma^2_\epsilon + n_i\sigma_u^2}+\sigma^2_u\right)$$

Now consider a simple random coefficients model: $$y_{it} = \beta_0 + U_{1i} + \beta_1x_{it} + U_{2i}x_{it} + \epsilon_{it}$$ where $$\left[\begin{array}{c} U_1\\ U_2\\ \epsilon \end{array}\right]\sim\mathcal{N}\left(\left[\begin{array}{c} 0\\ 0\\ 0\\ \end{array}\right],\left[\begin{array}{ccc} \sigma_{U_1}^2 & \rho_{12} & 0\\ \rho_{12} &\sigma_{U_2}^2 & 0\\ 0&0& \sigma^2_\epsilon\\ \end{array}\right]\right)$$

and where $U_{2i}$ is $N \times 1$ and is multiplied element-wise by $x_{it}$

What is the equivalent expression for predicting the random effects $\tilde U_1$ and $\tilde U_2$ (given estimated parameters) and how is it derived?

There is a matrix form for this. Following Simon Wood, the model is first formulated as $$\mathbf{y = X\beta+Z}b+\epsilon$$ While the book doesn't make it super-explicit, I'm taking it that the dimension of $\mathbf{Z}$ would be $N \times gk$, where $g$ is the number of groups, and $k$ is the number of random terms. In my example, it'd be the Khatri-Rao product of the group indicator on a vector of ones, concatenated to a KR product of the group indicator on the regressor $x_{it}$.

$b$ is thus a $gk\times 1$ vector, with the BLUPs for different terms stacked on each other. (Is this right? would appreciate confirmation)

In this formulation, the BLUP is then $$\hat{b} = \mathbf{\left(\tilde{Z}^T\tilde{Z}\right)^{-1}\tilde{Z}^T\left(\tilde y-\tilde X\hat\beta\right)}$$ Where $$\tilde{y} = \left[\begin{array}{c}\mathbf{y}\\\mathbf{0}\end{array}\right], \tilde{X} = \left[\begin{array}{c}\mathbf{X}\\\mathbf{0}\end{array}\right], \tilde{Z} = \left[\begin{array}{c}\mathbf{Z}\\\mathbf{B}\end{array}\right]$$ Where $\mathbf{B^TB} = \psi^{-1}\sigma^2$, where $\psi$ is the covariance of the random effects and $\sigma^2$ is the variance of $\epsilon$.

Since $\psi$ is $k\times k$ however, $\mathbf{B}$ can't be conformable with $\mathbf{Z}$, which is $n\times gk$. So there is a place where I am lost. Any help?

Anyway, I'm not sure that this formulation of the problem is helpful, becuase ultimately I'm thinking about mixed models in the context of estimating them by gradient descent.

Is there a clean way to compute $\frac{\partial y}{\partial \psi}$ (and ultimately the gradient with respect to the loss function) from this matrix formulation? Or can I derive an expression for computing each $g\times 1$ subset of $\mathbf{b}$ individually, for each random term, as in the second equation above?

• Forgive my ignorance, but what does BLUP stand for? Best Linear Unbiased <something>? – shadowtalker Dec 10 '17 at 19:02
• – generic_user Dec 10 '17 at 22:33