Difference between sequential and one-batch Bayesian update I learnt that sequential Bayesian update and one batch (all at once)update will give the same result if the observations are i.i.d. I tried to test this theory using my model which contains 4 parameters (2 of them are highly correlated).
Here is how I implement it:

*

*I divide 500 observations into 10 sets of 50data for sequential update

*Define the prior distributions of the parameters. (assume all follows N(ai, bi))

*observe one set of 50 data, the conditional posterior samples of each parameter are obtained by Gibbs sampling

*obtain the mean and std from the posterior samples and fit into normal distribution as new priors for the next set of data

*repeat steps 2-4 10 times

I found that the results between sequential and batch differ a lot. Can anyone please suggest to me what causes the difference?
 A: Assuming that you have excluded simple variation in MCMC results, I would assume that your problem is that the posterior after each sequential step cannot be adequately approximated by independent normal distributions for each parameter. Generally (of course with some exceptions), the posterior samples for different parameters will be correlated a-posteriori, even if the prior distributions were specified as independent. I.e. you need to describe the posterior by some joint distribution (perhaps a multivariate normal can capture the situation, but perhaps not).
If you reduce things to a super simple example, your approach should work though. E.g. it should work for the case where you only unknown parameter is the mean of a normal distribution (with known SD) as you can see here. In that simple case, you can do this both analytically using conjugate updating (see first row of this table), or numerically using conjugate updating (the RBesT R package is a nice tool for letting the computer do this for you), or in the way you described. However, once you assume the standard deviation is unknown, you already end up with a normal distribution being wrong even as the marginal posterior distribution for the mean (too short-tailed; additionally, of course you need a joint posterior for both parameters).
