I'm new in Bayesian inference and I can't found the answer to this:
In real life scenario people use MCMC to compute the posterior distribution given the likelihood and the prior. Analytical solutions are not possibles. Bayesian people often say "we update our prior believe given some data to have the posterior". But something is not ok to me here: the posterior is never in the same form than the prior, right ? Unless you have a conjugate prior, which is really rare.
So what does it mean ? Your prior is a gamma distribution, and you end-up with a posterior with a completely different shape. Did you really update the prior distribution ? certainly no. We can't compare apples and oranges.
Does it means that we had a prior belief with a certain shape (gamma distribution), then we update this belief so we have a new shape (not even described analytically) as the output of the MCMC.
I'm very confused with this idea of "Bayesian update", because in practice, if you end-up with a complete new kind of distribution for the posterior, you can't reuse it as a new prior for the next batch of data, right ? So it means that this is just "a one shot update" of the prior belief.
It seems to me that the Bayesian update mean you update your belief in the sense of you change the prior distribution to something else. It's like saying "I've changed my mind, it is no longer a gamma distribution".
On the other hand when I follow some lectures, they never say that. They talk about Bayesian update related to the use of conjugate prior. In this case the math are nice so the posterior can be used as a prior. But this never happen in real life, right ? I mean, you don't use MCMC if you know that the posterior will be of the same family of the prior ?