# Fitting special variance structure in mixed model

I'm interested in fitting a linear mixed model with this special variance structure on the random effects $\mathbf{u}$:

$\begin{eqnarray*} \mathbb{V}\left(\mathbf{u}\right) & = & \mathbf{A}\mathbf{G}\mathbf{A}^{\prime} \end{eqnarray*}$

This variance structure is very similar to Cholesky and Antedependence structures except few differences:

1. $\mathbf{A}$ is not a unit lower triangle or unit upper triangle matrix
2. $\mathbf{G}$ is not necessarily a diagonal matrix

I can specify initial values of $\mathbf{A}$. I'd highly appreciate if someone give me some hints to fit this variance covariance structure in R. Thanks in advance for your help and time.

With no restrictions on $A$, your equation doesn't place any restrictions whatsoever on the form of the variance-covariance matrix, even if you do specify that $G$ is diagonal. A variance-covariance matrix is a real symmetric matrix, and any such matrix can be diagonalized by an orthogonal matrix.
• Thanks @onestop for your nice answer. Actually $\mathbf{A}$ is kind of a relationship matrix and I want to use it in model fitting and interested in its value after model fitting. Any suggestion. Thanks Dec 15, 2011 at 20:24