# Deriving bayes formulas from "Overcoming catastrophic forgetting in neural networks"

I am trying to understand the formulas from the paper "Overcoming catastrophic forgetting in neural networks" and am wondering if someone could help explain how they derive these formulas. The paper explains a method for overcoming forgetting of task A when training the same network for task B.

The paper first defines the training of a neural network in a probabilistic way as follows:

$$\log p(\theta \mid D)=\log p(D \mid\theta)+\log p(\theta)-\log p(D)$$

The reasoning they provide for this definition is as follows:

"optimizing the parameters is tantamount to finding their most probable values given some data $$D$$. We can compute this conditional probability $$p(θ \mid D)$$ from the prior probability of the parameters $$p(θ)$$ and the probability of the data $$p(D \mid θ)$$ by using Bayes’ rule"

But since I am not too familiar with Bayes rule this definition is slightly confusing to me how they get this definition. The second formulation they make is to consider $$D$$ being composed of 2 independent parts, one for task A $$(D_A)$$ and one for task B $$(D_B)$$.

$$\log p(\theta \mid D)=\log p(D_B\mid \theta)+\log p(\theta \mid D_A)-\log p(D_B)$$

If anyone could help me fill in the steps / logic behind these formulas that would be very helpful

They got the first equation by taking the log of the normal Bayes rule expression, $$p(\theta \mid D) = \frac{p(D \mid \theta) p(\theta)}{p(D)} \text{.}$$
To make it clear that this is a general claim (not specific to $$D$$ and $$\theta$$), I could rewrite this as $$p(B\mid A) = \frac{p(A \mid B) p(B)}{p(A)} \text{.}$$
If this is unfamiliar, rearrange these statements about the joint probability drawn from the chain rule $$p(A, B) = p(A \mid B) p(B) = p(B\mid A)p(A)$$.
Now, we derive the second equation. $$D_a$$ and $$D_b$$ are considered independent of each other, so we may write $$p(D) = p(D_a)p(D_b)$$. They are also conditionally independent given $$\theta$$, so we may write $$p(D \mid \theta) = p(D_a \mid \theta) p(D_b \mid \theta)$$.
\begin{align} \log p(\theta \mid D) &= \log p(D \mid \theta) + \log p(\theta) - \log p(D) & \text{Bayes rule} \\ &= \log p(D_a \mid \theta) + \log p(D_b \mid \theta) + \log p(\theta) - \log p(D_a) - \log p(D_b) &\text{Independence} \\ &= \log p(D_b \mid \theta) + \left[\log p(D_a \mid \theta) + \log p(\theta) - \log p(D_a) \right] - \log p(D_b) &\text{Rearrange terms} \\ &= \log p(D_b \mid \theta) + \log p(\theta \mid D_a) - \log p(D_b) &\text{Bayes rule again} \\ \end{align}
What this tells us is that $$D_a$$ has updated our prior about $$\theta$$; we now treat the posterior given $$D_a$$ as our new prior when we work with $$D_b$$.