How to perform model selection with the BIC for correlated observations

Question

How is it possible to use the Bayesian Information Criterion (and more generally, to perform model selection) when observations are correlated ?

I would like to compare the BIC of different models given a data set of observations. The simplest models assume that my observations are IID; the more complicated ones assume a correlation between them (namely, the value of observation $x_i$ will depend on $x_{i-1}$).

The derivation and definition of the BIC are only valid under the assumption that data are IID. In this case, after integrating using Laplace's method, we are left with $-2log(p(x|M)) \approx -2log(p(x|\hat{\theta},M)) -2log(n^{-k/2}) -2log(|I(\hat{\theta})|^{-1/2}) - 2log(\pi(\hat{\theta}|M))-2log((2\pi)^{k/2})$ (where k is the number of parameters), which finally yields $BIC = -2log(\hat{L}) + klog(n)$, where $n$ is simply the number of data points.

Is it possible to extend this derivation for the case of correlated observations ? In this case, after integrating using Laplace's method, we are left with $-2log(p(x|M)) \approx -2log(p(x|\hat{\theta},M)) - 2log(\pi(\hat{\theta}|M))-2log((2\pi)^{k/2}) + log(|-D^2 log(p(x|\hat{\theta},M))|)$.

The last term is the determinant of the Hessian matrix of the log-likelihood function, and does not seem to simplify as nicely as in the IID case.

I would have guessed that it was a fairly common problem in model comparison, but I could not find relevant papers on how to use the BIC with correlated data. Is it possible, and if so, how would you do it ?

Answer 1

To the best of my knowledge, it turns out there are 3 ways to obtain a BIC-like proxy for $p(x|M)$ when $M$ is a model with correlated data:

1. One can use the previous approximation $-2 \log p(x|M) \approx -2 \log \pi(\hat{\theta} |M) -2 \log p(x|\hat{\theta},M) -k \log (2\pi) + \log(|-D^2 \log p(x|\hat{\theta},M)|)$

and explicitly compute the Hessian $D^2 \log p(x|\hat{\theta},M)$;

2. In case this computation is impossible (e.g. if the model is too complicated), a numerical approximation of the Hessian can be obtained through MCMC method, event without an explicit expression for the gradient, see for example Spall, J. C. (2005). Monte Carlo computation of the Fisher information matrix in nonstandard settings. Journal of Computational and Graphical Statistics, 14(4), 889-909.

3. Data can be "decorrelated" using for instance a mean-field approach, which allows to use the classical definition of the BIC for IID data.