Is it possible for a bayesian model to "forget" about some points? Say you are doing some online bayesian inference over observations.
You want to infer $\mu, \sigma$ for $X$, with a model $X \sim \mathcal{N}(\mu, \sigma)$.
Now everytime you observe a $X_i$, you update your prior with your new posterior and get a new estimate $\mu_i, \sigma_i$. That becomes the prior for your next observation, etc etc.
Now after $n$ observations, your estimates $\mu_n, \sigma_n$ come from applying "recursively" bayes to your priors $\mu_0, \sigma_0$, where :
$$P(\mu_n, \sigma_n) \propto P(\mu_{n-1}, \sigma_{n-1}) \times P(X_n \sim \mathcal{N}(\mu_{n-1},\sigma_{n-1}) | \mu_{n-1},\sigma_{n-1})$$
Now is it possible to make your bayesian model "forget" about old points without having to recalculate the whole chain ? In other words, if you wanted to apply this model to a stream of data (because you have lots of points), can you make it forget about old points ?
I hope I'm clear enough with my question, I'll edit as comments come if needed.
Thanks a lot
 A: You can remove their contribution from the likelihood or log-likelihood by division or subtraction, exactly reversing their contribution when they were multiplied or added in in the first place.
It's like saying "There's a long multiplication I've done: $3\times 4\times 2\times 5\times 6\times 6\times 2 \times x$. I have the answer, and but I want to compute it as if the 4 weren't there. Is there a way to do that?"
(Yes, just divide by 4. Now think of $x$ as your prior and the numbers as contributions of observations to the likelihood. You need to do the equivalent of 'divide by 4')
This is so facile as to worry me that I'd missed something obvious; I couldn't see a way to make a viable answer out of it beyond 'divide by its contribution to the likelihood'.
But here's something more than the obvious to make it worth an answer: In some cases, you might want to consider accumulated numerical error. If you multiply-and-divde or add-and-subtract floating point terms in and out a lot (say over a million observations, of which you keep only the last five hundred), eventually your computation might accumulate enough numerical 'fluff' to make it instead worth redoing periodically from scratch.
