Sequential Update of Bayesian I am currently reading Murphy's ML: A Probabilistic Perspective. In CH 3 he explains that a batch update of the posterior is equivalent to a sequential update of the posterior, and I am trying to understand this in the context of his example.
Suppose $D_a$ and $D_b$ are two data sets and $\theta$ is the parameter to our model. We are trying to update the posterior $P(\theta \mid D_a, D_b)$. In a sequential update, he states that,
$$
(1) \ \ \ \ \ \ \ \  P(\theta \mid D_{a}, D_{b}) \propto P(D_b \mid \theta) P(\theta \mid D_a)
$$
However, I am slightly confused as to how he got this mathematically. Conceptually, I understand that he is saying the posterior $P(\theta \mid D_a)$ is now a prior used to update the new posterior, which includes the new data $D_b$, and is multiplying this prior with the likelihood $P(D_b \mid \theta)$. Expanding the last statement out, I have,
$$ 
P(D_b \mid \theta) P(\theta \mid D_a) = P(D_b \mid \theta) P(D_a \mid \theta) P(\theta)
$$
but are we allowed to say $P(D_a \mid \theta) P(D_b \mid \theta) = P(D_a, D_b \mid \theta)$ in order to make the connection in (1)?
 A: Indeed - you can update sequentially or in a batch fashion so long as you assume exchangeability.  It's analogous to the iid assumption typically made in frequentist models.  
In this case, $D_{a}$ and $D_{b}$ exchangeable implies that $P(D_{a}, D_{b} \, | \, \theta) = P(D_{a} \, | \, \theta) P(D_{b} \, | \, \theta)$ for some $\theta$, which is exactly what you need to make the connection.
You can see a proof of equivalence between a single $n$-large batch update and $n$ sequential updates in an answer I wrote to a similar question.
A: Actually, the general formula of sequential Bayesian updating is:
$$
P(\theta \mid D_{a}, D_{b}) \propto P(D_b \mid \theta, D_a) P(\theta \mid D_a). 
\,\,\,(*)
$$
However, for most machine learning models, 
 $D_a$ and $D_b$ are conditionally independent given $\theta$, i.e.,
$$ 
P(D_a \mid \theta) P(D_b \mid \theta) = P(D_a, D_b \mid \theta),
$$
then, $P(D_b \mid \theta, D_a)$ in $(*)$ naturally equals to $P(D_b \mid \theta), $
so the $(*)$ becomes:
$$
(1)\,\,\,\,\,\,P(\theta \mid D_{a}, D_{b}) \propto P(D_b \mid \theta) P(\theta \mid D_a),
$$
which is exactly what Murphy's ML book talks about.
