# Deriving the maximum likelihood of REML for linear mixed model

Consider the linear model $Y = X \beta + e$, $e \sim N(0, V(\theta))$, where $Y$ is a $n \times 1$ vector, $X$ is the $n \times p$ full rank design matrix, $V(\theta)$ is the covariance matrix. I drop $\theta$ below for convenient expression.

B is an $n \times (n-p)$ matrix whose columns are orthonormal basis of $C(X)^{\perp}$. REML maximizes the likelihood of $B^T Y$, which can be expressed as

$$L(\theta) = (2 \pi)^{-(n-p)/2} |B^T V B|^{-1/2} \text{exp} \{- \frac{1}{2} Y^T B (B^T V B)^{-1} B^T Y \} \tag{1}$$ where $|A|$ is the determinant of matrix $A$.

This can be proved to be equivalent as

$$L(\theta) = (2 \pi)^{-(n-p)/2} |X^T X|^{1/2} |V|^{-1/2} |X^T V^{-1} X|^{-1/2} \text{exp} \{- \frac{1}{2} Y^T V^{-1} (I - Q) Y \} \tag{2}$$ where $Q = X (X^T V^{-1} X)^{-1}X^T V^{-1}$.

I am able to prove that $$I - Q = V B (B^T V B)^{-1} B^T \tag{3}$$ by showing that they are the same projection operator onto $C(VB)$ along $C(X)$, and thus prove the $\text{exp}$ part.

However, I don't know how to prove

$$|B^T V B| = |X^T X|^{-1} |V| |X^T V^{-1} X| \tag{4}$$

Any suggestion would help.

• Can you please add the source of these derivations? Dec 28, 2017 at 0:17
• I use my notation in the derivation, but there are some sources. For example, page 20 of this paper. Also, this is another relevant post, citing "Harville 1974 showed that". Dec 28, 2017 at 3:02
• Note that it is obvious when $V=$ identity since $B^\top V B = B^\top B = I_{n-p}$ then. So this is a question about dot product.
– Yves
Dec 28, 2017 at 19:56
• @Yves Sorry I still fail to get the point. Could you please elaborate on how to use dot product to show the equality? Thanks. Dec 28, 2017 at 20:53
• Sorry I thought that orthonormality related to the dot product $<y, y'> := y^\top V^{-1}y'$ (which is not the case) and that the result would come by transformation. It seems to be quite tricky, and I will try to understand Harville 1974'' or later references that might be more detailed.
– Yves
Dec 29, 2017 at 11:11

Because $\begin{pmatrix} X & B \end{pmatrix}$ is a n by n matrix with $B^T X = 0$, we have
where (5) and (7) follows from rule of determinant of block matrix, (6) and (9) follows from the rule of determinant of product of square matrix, and (8) follows from (3). Finally, using the fact that $B^T B = I$ (orthonormal basis), (4) is proved.