Sequential updating of a Bayesian regression model

Question

I'd like to perform Bayesian regression of $Y$ on $X$. i.e. estimate a posterior $p\left(\theta\mid\mathcal{D}\right)$ of some regression parameter $\theta$ given a dataset $\mathcal{D}$ of pairs of points $(x_i, y_i)$. Suppose I have a dataset, $\mathcal{D}_1$, I estimate $p\left(\theta\mid\mathcal{D}_1\right)$, and then I get another dataset $\mathcal{D}_2$, and I want $p\left(\theta\mid\mathcal{D}_1, \mathcal{D}_2\right)$.

Intuitively, I want to use the posterior after the first update as the prior for the second. e.g. in linear regression with a Gaussian prior $\mu_{\theta},\Sigma_{\theta}$. I have a formula to which I insert $\mu_{\theta},\Sigma_{\theta}$ and $\mathcal{D}_1$ and it gives me $\mu_{\theta\mid\mathcal{D}_{1}},\Sigma_{\theta\mid\mathcal{D}_{1}}$. So when I subsequently input $\mu_{\theta\mid\mathcal{D}_{1}},\Sigma_{\theta\mid\mathcal{D}_{1}}$ and $\mathcal{D}_2$ it should give $\mu_{\theta\mid\mathcal{D}_{1}, \mathcal{D}_{2}},\Sigma_{\theta\mid\mathcal{D}_{1}, \mathcal{D}_{2}}$

I know how to justify this for sequential binary hypothesis testing, and I could probably show it for the specific example I gave above. But I know (maybe "feel" is a better word) that this is a general principle, just not sure how to prove it.

Answer 1

Using Bayes' Theorem, we have: \begin{equation} p({\theta}|{y})=\dfrac{L({\theta}|{y})p({\theta})}{p({y})}. \end{equation} where $p({y})=\int{L({\theta}|{y})p({\theta})}d{\theta}$. Since $p({y})$ does not depend on ${\theta}$, the above expression can be rewritten as: \begin{equation*} p({\theta}|{y})\propto L({\theta}|{y})p({\theta}). \end{equation*}

From this, we obtain: $$p(\theta\mid D_1, D_2) = L(\theta\mid D_1, D_2)\times p(\theta)=L(\theta\mid D_1)\times L(\theta\mid D_2)\times p(\theta),$$ where $p(\theta)$ is a prior for $\theta$ before knowing $D_1$. This expression can be rewritten as: $$L(\theta\mid D_2)\times (L(\theta\mid D_1)\times p(\theta))=L(\theta\mid D_2)\times p(\theta\mid D_1),$$ where $L(\theta\mid D_2)\times p(\theta\mid D_1)$ is actually the posterior of $\theta$ (after $D_2$) given prior $p(\theta\mid D_1)$.

Answer 2

Assuming that the data are conditionally independent given the parameter $\theta$,

$$p\left(\theta\mid\mathcal{D}_{1},\mathcal{D}_{2}\right) =p\left(\theta\right)\frac{p\left(\mathcal{D}_{1},\mathcal{D}_{2}\mid\theta\right)}{p\left(\mathcal{D}_{1},\mathcal{D}_{2}\right)} \\=p\left(\theta\right)\frac{p\left(\mathcal{D}_{1},\mathcal{D}_{2}\mid\theta\right)}{\int p\left(\mathcal{D}_{1},\mathcal{D}_{2},\theta'\right)d\theta'} \\=p\left(\theta\right)\frac{p\left(\mathcal{D}_{1}\mid\theta\right)p\left(\mathcal{D}_{2}\mid\theta\right)}{\int p\left(\mathcal{D}_{1},\theta'\right)p\left(\mathcal{D}_{2},\theta'\right)d\theta'} \\=p\left(\theta\right)\frac{p\left(\mathcal{D}_{1}\mid\theta\right)}{\int p\left(\mathcal{D}_{1},\theta'\right)d\theta'}\frac{p\left(\mathcal{D}_{2}\mid\theta\right)}{\int p\left(\mathcal{D}_{2},\theta'\right)d\theta'} \\=p\left(\theta\right)\frac{p\left(\mathcal{D}_{1}\mid\theta\right)}{p\left(\mathcal{D}_{1}\right)}\frac{p\left(\mathcal{D}_{2}\mid\theta\right)}{p\left(\mathcal{D}_{2}\right)}$$