I am trying to understand the formulas from the paper "Overcoming catastrophic forgetting in neural networks" and am wondering if someone could help explain how they derive these formulas. The paper explains a method for overcoming forgetting of task A when training the same network for task B.

The paper first defines the training of a neural network in a probabilistic way as follows:

$$\log p(\theta \mid D)=\log p(D \mid\theta)+\log p(\theta)-\log p(D)$$

The reasoning they provide for this definition is as follows:

"optimizing the parameters is tantamount to finding their most probable values given some data $D$. We can compute this conditional probability $p(θ \mid D)$ from the prior probability of the parameters $p(θ)$ and the probability of the data $p(D \mid θ)$ by using Bayes’ rule"

But since I am not too familiar with Bayes rule this definition is slightly confusing to me how they get this definition. The second formulation they make is to consider $D$ being composed of 2 independent parts, one for task A $(D_A)$ and one for task B $(D_B)$.

$$\log p(\theta \mid D)=\log p(D_B\mid \theta)+\log p(\theta \mid D_A)-\log p(D_B)$$

If anyone could help me fill in the steps / logic behind these formulas that would be very helpful


They got the first equation by taking the log of the normal Bayes rule expression, $$ p(\theta \mid D) = \frac{p(D \mid \theta) p(\theta)}{p(D)} \text{.} $$

To make it clear that this is a general claim (not specific to $D$ and $\theta$), I could rewrite this as $$ p(B\mid A) = \frac{p(A \mid B) p(B)}{p(A)} \text{.} $$

If this is unfamiliar, rearrange these statements about the joint probability drawn from the chain rule $p(A, B) = p(A \mid B) p(B) = p(B\mid A)p(A)$.

Now, we derive the second equation. $D_a$ and $D_b$ are considered independent of each other, so we may write $p(D) = p(D_a)p(D_b)$. They are also conditionally independent given $\theta$, so we may write $p(D \mid \theta) = p(D_a \mid \theta) p(D_b \mid \theta)$.

$$ \begin{align} \log p(\theta \mid D) &= \log p(D \mid \theta) + \log p(\theta) - \log p(D) & \text{Bayes rule} \\ &= \log p(D_a \mid \theta) + \log p(D_b \mid \theta) + \log p(\theta) - \log p(D_a) - \log p(D_b) &\text{Independence} \\ &= \log p(D_b \mid \theta) + \left[\log p(D_a \mid \theta) + \log p(\theta) - \log p(D_a) \right] - \log p(D_b) &\text{Rearrange terms} \\ &= \log p(D_b \mid \theta) + \log p(\theta \mid D_a) - \log p(D_b) &\text{Bayes rule again} \\ \end{align} $$

What this tells us is that $D_a$ has updated our prior about $\theta$; we now treat the posterior given $D_a$ as our new prior when we work with $D_b$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.