I have been reading about alternatives to assuming an AR(1) covariance structure for mixed models with repeated measures (in time). In particular I have heard about Toeplitz and Ante-dependence structures (that are both available in SAS but not too many others as far as I know), as well as homogeneous/heterogeneous versions of all three, which can apparently be seen as generalisations of AR(1).
What are the advantages/disadvantages of these and in what situations would we use them ?
When we encounter models with repeated measures in time, we naturally want to account for the correlations within subjects and also the possibility of heterogeneous variances across time points. When we constrain the variances across time points to be equal, this is a homogeneous structure, and when we allow variances to be different, this is a heterogeneous version of the structure. Homogeneous and heterogeneous structures share the same correlation structure.
Description: The AR(1) structure models a scenario where the correlation between observations decays as the time interval between them increases. This structure is based on the principle that the influence of a measurement on subsequent measurements decreases as the time gap widens.
Use Case: AR(1) is suitable for data where the correlation between measurements decreases as the time between them increases. It's often used in studies where the process has a "memory" of the previous value, but that influence diminishes over time. In most implementations I have seen, this structure requires equally spaced time points. One exception I have found is JMP:
Many statistical software packages require the time points to be at equal intervals, but JMP allows unequal spacing in the time points."
Correlation Matrix (P): $$ P_{\text{AR1}} = \begin{bmatrix} 1 & \rho & \rho^2 & \rho^3 \\ \rho & 1 & \rho & \rho^2 \\ \rho^2 & \rho & 1 & \rho \\ \rho^3 & \rho^2 & \rho & 1 \\ \end{bmatrix} $$
Homogeneous AR(1) Covariance Matrix: $$ \Sigma_{\text{AR1}} = \sigma^2 \cdot P_{\text{AR1}} = \begin{bmatrix} \sigma^2 & \sigma^2\rho & \sigma^2\rho^2 & \sigma^2\rho^3 \\ \sigma^2\rho & \sigma^2 & \sigma^2\rho & \sigma^2\rho^2 \\ \sigma^2\rho^2 & \sigma^2\rho & \sigma^2 & \sigma^2\rho \\ \sigma^2\rho^3 & \sigma^2\rho^2 & \sigma^2\rho & \sigma^2 \\ \end{bmatrix} $$ Number of parameters to be estimated: $2$ (1 variance and 1 correlation)
Heterogeneous AR(1) Covariance Matrix: $$ \Sigma_{\text{AR1h}} = D \cdot P_{\text{AR1}} \cdot D = \begin{bmatrix} \sigma_1^2 & \sigma_2\sigma_1\rho & \sigma_3\sigma_1\rho^2 & \sigma_4\sigma_1\rho^3 \\ \sigma_2\sigma_1\rho & \sigma_2^2 & \sigma_3\sigma_2\rho & \sigma_4\sigma_2\rho^2 \\ \sigma_3\sigma_1\rho^2 & \sigma_3\sigma_2\rho & \sigma_3^2 & \sigma_4\sigma_3\rho \\ \sigma_4\sigma_1\rho^3 & \sigma_4\sigma_2\rho^2 & \sigma_4\sigma_3\rho & \sigma_4^2 \\ \end{bmatrix} $$ Number of parameters to be estimated: $t + 1$ ($t$ variances and 1 correlation)
Description: The Toeplitz structure accommodates different correlations for different lags. The homogeneous version maintains constant variance across all time points, while the heterogeneous version permits different variances at each time point, offering more flexibility in modeling correlation patterns.
Use Case: Toeplitz matrices are diagonal-constant matrices. These are often used when the correlation pattern does not strictly follow the pattern of an AR(1) structure, particularly useful in scenarios where the correlation changes in a patterned way based on lag but not necessarily in a monotonically decreasing manner.
Correlation Matrix (P): $$ P_{\text{Toep}} = \begin{bmatrix} 1 & \rho_1 & \rho_2 & \rho_3 \\ \rho_1 & 1 & \rho_1 & \rho_2 \\ \rho_2 & \rho_1 & 1 & \rho_1 \\ \rho_3 & \rho_2 & \rho_1 & 1 \\ \end{bmatrix} $$
Homogeneous Toeplitz Covariance Matrix: $$ \Sigma_{\text{Toep}} = \sigma^2 \cdot P_{\text{Toep}} = \begin{bmatrix} \sigma^2 & \sigma^2\rho_1 & \sigma^2\rho_2 & \sigma^2\rho_3 \\ \sigma^2\rho_1 & \sigma^2 & \sigma^2\rho_1 & \sigma^2\rho_2 \\ \sigma^2\rho_2 & \sigma^2\rho_1 & \sigma^2 & \sigma^2\rho_1 \\ \sigma^2\rho_3 & \sigma^2\rho_2 & \sigma^2\rho_1 & \sigma^2 \\ \end{bmatrix} $$ Number of parameters to estimate: $k+1$ ($k$ covariance parameters (one for each lag) plus one variance parameter.)
Heterogeneous Toeplitz Covariance Matrix: $$ \Sigma_{\text{Toeph}} = D \cdot P_{\text{Toep}} \cdot D = \begin{bmatrix} \sigma_1^2 & \sigma_2\sigma_1\rho_1 & \sigma_3\sigma_1\rho_2 & \sigma_4\sigma_1\rho_3 \\ \sigma_2\sigma_1\rho_1 & \sigma_2^2 & \sigma_3\sigma_2\rho_1 & \sigma_4\sigma_2\rho_2 \\ \sigma_3\sigma_1\rho_2 & \sigma_3\sigma_2\rho_1 & \sigma_3^2 & \sigma_4\sigma_3\rho_1 \\ \sigma_4\sigma_1\rho_3 & \sigma_4\sigma_2\rho_2 & \sigma_4\sigma_3\rho_1 & \sigma_4^2 \\ \end{bmatrix} $$ Number of parameters to estimate: $k + t$, ($k$ covariance parameters (one for each lag) plus $t$ variance parameter)
Description: The ante-dependence structure models a scenario where the correlation between two nonadjacent observations is the product of correlations between the intervening observations, allowing for heterogeneous variances and correlations. This structure can capture more complex dependency patterns.
Use Case: Ante-dependence is suitable for studies where the correlation between two time points depends on the product of intermediate correlations, ideal for modeling processes where the influence of past observations diminishes over time or distance in a non-linear fashion. This structure is also suitable for data with unequal time points.
Correlation Matrix (P): $$ P_{\text{AD1}} = \begin{bmatrix} 1 & \rho_1 & \rho_1\rho_2 & \rho_1\rho_2\rho_3 \\ \rho_1 & 1 & \rho_2 & \rho_2\rho_3 \\ \rho_1\rho_2 & \rho_2 & 1 & \rho_3 \\ \rho_1\rho_2\rho_3 & \rho_2\rho_3 & \rho_3 & 1 \\ \end{bmatrix} $$
Covariance Matrix:
$$ \Sigma_{\text{AD1}} = D \cdot P_{\text{AD1}} \cdot D = \begin{bmatrix} \sigma_1^2 & \sigma_2\sigma_1\rho_1 & \sigma_3\sigma_1\rho_1\rho_2 & \sigma_4\sigma_1\rho_1\rho_2\rho_3 \\ \sigma_2\sigma_1\rho_1 & \sigma_2^2 & \sigma_3\sigma_2\rho_2 & \sigma_4\sigma_2\rho_2\rho_3 \\ \sigma_3\sigma_1\rho_1\rho_2 & \sigma_3\sigma_2\rho_2 & \sigma_3^2 & \sigma_4\sigma_3\rho_3 \\ \sigma_4\sigma_1\rho_1\rho_2\rho_3 & \sigma_4\sigma_2\rho_2\rho_3 & \sigma_4\sigma_3\rho_3 & \sigma_4^2 \\ \end{bmatrix} $$ Number of parameters to estimate: In a first-order ante-dependence model, the number of parameters is similar to an AR(1) model. However, in higher-order models, the number of parameters increases significantly. For a $k$-th order ante-dependence model, you would generally estimate $k$ parameters for the covariances and one variance parameter, totalling $k+1$. However, the exact number can vary depending on whether the model is structured to have different parameters for different lags.
It is usually quite difficult to determine which structure to use. There can be a trade-off between model parsimony where a small number of parameters are needed (eg AR1) and a "better" model fit. One approach is to start with the least parsimonious, which will usually be fully unstructured, and if this fits the data (sometimes such a model cannot be fitted due to numerical problems) then proceed with more parsimonious structures, and compare the fits with, for example, AIC or BIC)
The choice of the correlation pattern to fit is a crucial one that affects the efficiency and bias of estimates of the main effects of interest. AR(1) tends to be a very competitive fit in terms of AIC, i.e., on the likelihood of predicting future observations well. One can use graphical displays of correlation matrices to choose a correlation pattern. Usually better, because it handles the continuous time case, is a semivariogram, where one plots the covariance vs. the time gap, with the theoretical semivariogram (say, assuming AR(1)) superimposed. An example is here.
