Will off-diagonal elements of Cov-Matrix contribute to number of parameters for AIC?

Question

This question was originally asked by a colleague with regards to calculation of AIC for a non-linear mixed effects model (nlme) where off-diagonal elements of the covariance matrix are estimated. The equations for AIC (from Wikipedia):

\begin{align} AIC &= OFV + 2k \\ BIC &= OFV + k \ln(n) \end{align}

with $k$ the number of (non-fixed) parameters, and $n$ the number of observations (and $OFV = -2LL$ for a nlme model using maximum likelihood).

Question: Would only the diagonal elements of the random effects matrix count in k, or also the off-diagonal elements, if estimated? Would the off-diagonal elements really count as a full degree of freedom?

My thoughts after reading this paper¹ are that AIC in its native expression can be used mostly for linear models. AIC also depends on the estimation method used, marginal vs conditional likelihood. Even if we have to use the "conditional AIC" (cAIC) that the authors suggest in the paper, how would we treat the off-diagonal elements of the random effect matrix: would they contribute to the degrees of freedom and by what extent?

Would any of you consider using AIC for complex nlme model with many random effects and possible correlations?

_{1.Vaida, F. and S. Blanchard, Conditional Akaike information for mixed-effects models. Biometrika, 2005. 92(2): p. 351-370.}

Answer 1

The off diagonal elements are unknown and would count because they are not fixed and are estimated from the data. But keep in mind that the mixed model usually specifies a structured covariance matrix which means you are estimating only a few parameters that determine the covariances and not all of of the upper half off diagonal elements. For example, if you specify AR(1) structure all the covariances are determined based on the one autoregressive parameter.

Answer 2

If you are using non-linear mixed effects models, you should probably not be using AIC to begin with. Parameters in multi-level models cannot vary independently, and simply counting them is not a great way to get an effective parameter size estimate.

Andrew Gelman and the people at Stan have a paper on WAIC. Check it out.