I am working with some two-regime autoregressive models first introduced by Hamilton in 1989. The specific models is of no great concern to my question, but some variables within my autoregressive models are latent binary variables - non-observable binary variables, that is. I have a dataset and use MCMC to find posterior densities of all the parameters of my models, including all the latent binary variables.
I have a dataset of about 1000 observations, and for every observation I have a latent variable in my models, which can wither be 1 or 0. If my data suddendly changes, I can assume that a switch has been turned on and the latent variable has changed. So if the data, at a specific time, is (most likely) best described by the latent variable being 1, my sampler will produce a posterior distribution (for this specific latent variable) with high probability of 1 and low probability of 0.
So this is all good, I have managed to do this. The question is, how do I validate my model? AIC/BIC is just for comparing models. What I am thinking is that I have to be able to plot the residuals somehow, like in normal regression where you just say that the residual is the difference of observed and predicted result. I understand that for all other parameters than the latent parameters/variables, i can just use the mean of each posterior distribution and treat it like a Maximum Likelihood Estimator. But I cant take the mean of my posterior distributions for all the latent variables, because they are either 0 or 1. It does not make any sense using, for example, 0.8 for one of my latent variables. This is no option.
So how should I go on to validate my model? I am really stuck here. If I have done a bad job explaining my problem, I apologize and will try to explain better.
So I have been thinking. Can I use a predictive posterior distribution to validate my model? And if too many observations, lies above, say, the 95 percentile of the predictive posterior distribution, I throw away my model?
Generally, for any model whose parameters are called theta, observations called x, and new observation called x_new, one calculate the predictive posterior distribution as follows: For i = 1, . . . , n (samples), we sample: 1. theta[i] from p(theta|x) and 2. x_new[i] from p(x_new|theta[i]) Then x_new, . . . ,x_new[n] are a sample from the predictive posterior distribution. But this method must surely only work when we have a iid sample. I am working with an autoregressive model. So I am stuck...