# Creating a better prior based on past observations

Based on this post,

In plain english, update a prior in bayesian inference means that you start with some guesses about the probability of an event occuring (prior probability), then you observe what happens (likelihood), and depending on what happened you update your initial guess. Once updated, your prior probability is called posterior probability.

Of course, now you can:

• stop with your posterior probability;
• use you posterior probability as a new prior, and update such a probability to obtain a new posterior by observing more evidence (i.e. data).

To the best of my understanding, this describes the following scenario.

1. There existed some distribution $$P[\theta]$$, from which exactly one set of parameters $$\theta_1$$ was randomly generated. The parameter $$\theta_1$$ is unknown to the experimenter.
2. The parameter $$\theta_1$$ was then used to generate some observed data $$D_1, D_2$$ and $$D_3$$ according to some model $$P[D|\theta]$$.
3. One can make a better guess at the value of $$\theta_1$$ than the prior $$P[\theta]$$ by incorporating the data, for example by computing the posterior $$P[\theta_1 | D_1]$$, and then using posterior as a prior two more times to also incorporate $$D_2$$ and $$D_3$$.

I am interested in a slightly more complicated scenario

1. From $$P[\theta]$$ two sets of parameters $$\theta_1$$ and $$\theta_2$$ are generated. These can be thought of as two different experiments, where former is done earlier than the latter.
2. Each set of parameters is used to generate their own data $$D_1$$ and $$D_2$$ correspondingly.
3. Initially experimenter only has access to a not-so-good prior $$\tilde P[\theta]$$ which they seek to improve

Question: Can the data from the first experiment be used to get a better prior distribution than $$\tilde P[\theta]$$ for use in the second experiment? If yes how?

Naively, I had thought that the posterior

$$P[\theta_1 | D_1] = \frac{P[D_1 | \theta_1] \tilde P[\theta_1]}{\sum_{\theta_1} P[D_1 | \theta_1] \tilde P[\theta_1]}$$

from the first experiment can be used as prior to the second experiment, but that is surely wrong. For example, if data $$D_1$$ is so crisp and clear that it tells us exactly what $$\theta_1$$ is, then our posterior would be a delta function $$\delta(\theta = \theta_1)$$. That is a bad prior for future experiments, because future values of $$\theta$$ may be different from $$\theta_1$$.

In general, I am interested in the formal procedure of constructing a new for future use using data from several past experiments.

• Hint: The solution will have to include the correlation between your two $\theta$ values.
– Eoin
Commented Jun 18, 2022 at 21:32
• It sounds like $\theta_i$'s is a time-series but for some reason, you want not to model it as a time-series, why is that?
– Tim
Commented Jun 18, 2022 at 22:04
• @Tim I'm sorry, I don't follow. Could you elaborate on what you mean? $\theta_i$ are random samples from the true distribution $P[\theta]$. Yes, $\theta_i$ are a time series of random variables that are not observed directly and are uncorrelated to each other. What does it mean to model them as time series? Commented Jun 19, 2022 at 13:48
• @Eoin I am currently considering the most simple case, where individual experiments are independent and identically distributed. In other words, $\theta_{i+1}$ is uncorrelated to all previous $\theta$ values. I'm sorry, maybe I'm just being stupid, but I don't get the hint Commented Jun 19, 2022 at 14:16
• Sorry, I misunderstood part of your question.
– Eoin
Commented Jun 19, 2022 at 16:13