# Sequential updating of a Bayesian regression model

I'd like to perform Bayesian regression of $$Y$$ on $$X$$. i.e. estimate a posterior $$p\left(\theta\mid\mathcal{D}\right)$$ of some regression parameter $$\theta$$ given a dataset $$\mathcal{D}$$ of pairs of points $$(x_i, y_i)$$. Suppose I have a dataset, $$\mathcal{D}_1$$, I estimate $$p\left(\theta\mid\mathcal{D}_1\right)$$, and then I get another dataset $$\mathcal{D}_2$$, and I want $$p\left(\theta\mid\mathcal{D}_1, \mathcal{D}_2\right)$$.

Intuitively, I want to use the posterior after the first update as the prior for the second. e.g. in linear regression with a Gaussian prior $$\mu_{\theta},\Sigma_{\theta}$$. I have a formula to which I insert $$\mu_{\theta},\Sigma_{\theta}$$ and $$\mathcal{D}_1$$ and it gives me $$\mu_{\theta\mid\mathcal{D}_{1}},\Sigma_{\theta\mid\mathcal{D}_{1}}$$. So when I subsequently input $$\mu_{\theta\mid\mathcal{D}_{1}},\Sigma_{\theta\mid\mathcal{D}_{1}}$$ and $$\mathcal{D}_2$$ it should give $$\mu_{\theta\mid\mathcal{D}_{1}, \mathcal{D}_{2}},\Sigma_{\theta\mid\mathcal{D}_{1}, \mathcal{D}_{2}}$$

I know how to justify this for sequential binary hypothesis testing, and I could probably show it for the specific example I gave above. But I know (maybe "feel" is a better word) that this is a general principle, just not sure how to prove it.

• You may find this paper on 'Predictive Updating Methods with Application to Bayesian Classification' by Rong Chen useful. It is fully available at researchgate.net/profile/Jun_Liu54/publication/… . Commented Nov 30, 2020 at 19:36

Using Bayes' Theorem, we have: $$$$p({\theta}|{y})=\dfrac{L({\theta}|{y})p({\theta})}{p({y})}.$$$$ where $$p({y})=\int{L({\theta}|{y})p({\theta})}d{\theta}$$. Since $$p({y})$$ does not depend on $${\theta}$$, the above expression can be rewritten as: $$\begin{equation*} p({\theta}|{y})\propto L({\theta}|{y})p({\theta}). \end{equation*}$$
From this, we obtain: $$p(\theta\mid D_1, D_2) = L(\theta\mid D_1, D_2)\times p(\theta)=L(\theta\mid D_1)\times L(\theta\mid D_2)\times p(\theta),$$ where $$p(\theta)$$ is a prior for $$\theta$$ before knowing $$D_1$$. This expression can be rewritten as: $$L(\theta\mid D_2)\times (L(\theta\mid D_1)\times p(\theta))=L(\theta\mid D_2)\times p(\theta\mid D_1),$$ where $$L(\theta\mid D_2)\times p(\theta\mid D_1)$$ is actually the posterior of $$\theta$$ (after $$D_2$$) given prior $$p(\theta\mid D_1)$$.
Assuming that the data are conditionally independent given the parameter $$\theta$$,
$$p\left(\theta\mid\mathcal{D}_{1},\mathcal{D}_{2}\right) =p\left(\theta\right)\frac{p\left(\mathcal{D}_{1},\mathcal{D}_{2}\mid\theta\right)}{p\left(\mathcal{D}_{1},\mathcal{D}_{2}\right)} \\=p\left(\theta\right)\frac{p\left(\mathcal{D}_{1},\mathcal{D}_{2}\mid\theta\right)}{\int p\left(\mathcal{D}_{1},\mathcal{D}_{2},\theta'\right)d\theta'} \\=p\left(\theta\right)\frac{p\left(\mathcal{D}_{1}\mid\theta\right)p\left(\mathcal{D}_{2}\mid\theta\right)}{\int p\left(\mathcal{D}_{1},\theta'\right)p\left(\mathcal{D}_{2},\theta'\right)d\theta'} \\=p\left(\theta\right)\frac{p\left(\mathcal{D}_{1}\mid\theta\right)}{\int p\left(\mathcal{D}_{1},\theta'\right)d\theta'}\frac{p\left(\mathcal{D}_{2}\mid\theta\right)}{\int p\left(\mathcal{D}_{2},\theta'\right)d\theta'} \\=p\left(\theta\right)\frac{p\left(\mathcal{D}_{1}\mid\theta\right)}{p\left(\mathcal{D}_{1}\right)}\frac{p\left(\mathcal{D}_{2}\mid\theta\right)}{p\left(\mathcal{D}_{2}\right)}$$
• In your derivation you have $p(\mathcal{D}_1, \mathcal{D}_2) = \int p(\mathcal{D}_1, \mathcal{D}_2 | \theta') d\theta'$. However, isn't it the case that $p(\mathcal{D}_1, \mathcal{D}_2) = \int p(\mathcal{D}_1, \mathcal{D}_2 | \theta') p(\theta')d\theta'$ ? Commented Sep 13, 2023 at 6:48