Bayesian inference on sequential data

Question

I have a dataset that is potentially contaminated. The variable of interest is continuous (non-Gaussian but for simplicity sake let's just assume that it is). Let's say our prior is that the true mean should be 50% of the in-sample mean but we don't know for sure. As we observe more out-of-sample observations, it either confirms or rejects our prior assumption. For example, 1 out-of-sample observation isn't enough evidence to move away from the prior (in-sample x 50%) but let's say 50 out-of-sample observations are plenty to give a reasonable assessment as to whether the prior is right or wrong. I'm interested in the true out-of-sample mean. What's tricky about this problem (at least to me) is the need for sequential update. What would be a good approach to this problem?

I've already found this question that doesn't really have an answer.

[Edit] One option I've thought about is to assume Gaussian likelihood and prior. The assumed prior is the in-sample mean and precision, but the mean is adjusted by the assumed overstatement. Let prior be $N(\mu_0, \pi_0)$, where $\pi = 1/\sigma^2$ is precision. The posterior is:

$$ \mu_{post}=\frac{\pi_0\mu_0}{\pi_0+\pi}+\frac{\pi x}{\pi_0+\pi} \\ \pi_{post}=\pi_0+\pi $$

I don't know what the precision of each new observation is but I thought a reasonable assumption is to just assume it has the same precision as the in-sample data. This means that the first update my precision doubled, while mean is the average between my prior and the first observation. Then assume my posterior is my new prior for the next update. However, this seems to be adjusting too aggressively to each new observation, particularly at the beginning when I have only a few out-of-sample observations. What I'm expecting to see if at the beginning, my posterior is barely different from my prior, but adjusting closer to the observed out-of-sample data over time.

Answer 1

Here is a framework that might be applicable.

There are two related entities. There are $T_i$ observations for entity: $Y_i = (y_{i1}, \ldots, y_{iT_i})$ for $i \in \{1,2\}$. Each of the two entities depends on an unknown parameter $\mu_i$: $p(Y_i|\mu_i)$. If the observations for entity $i$ were iid (independent and identically distributed) then \begin{equation} p(Y_i|\mu_i) = \prod_{t=1}^{T_i} p(y_{it}|\mu_i) . \end{equation}

Given this setup, the only possible connection between $\mu_2$ and $Y_1$ (for example) is through the joint prior for $(\mu_1,\mu_2)$. In other words, we can learn about $\mu_2$ from $Y_1$ if we believe that we can learn about $\mu_2$ from $\mu_1$. We can complete the connection in two steps. First, \begin{equation} p(\mu_2|Y_1,\mu_1) = \frac{p(Y_2|\mu_2)\,p(\mu_2|\mu_1)}{p(Y_2|\mu_1)} . \end{equation} Second, given $p(\mu_1|Y_1)$ we have \begin{equation} p(\mu_2|Y_1,Y_2) = \int p(\mu_1|Y_2,\mu_1)\,p(\mu_1|Y_1)\,d\mu_1 . \end{equation}

We can arrive at the same place via a different route. First compute \begin{equation} p(\mu_2|Y_1) = \int p(\mu_2|\mu_1)\,p(\mu_1|Y_1)\,d\mu_1 \end{equation} and second \begin{equation} p(\mu_2|Y_1,Y_2) = \frac{p(Y_2|\mu_2)\,p(\mu_2|Y_1)}{p(Y_2|Y_1)} . \end{equation} (Note that the sample information $Y_2$ is contained in the likelihood and the non-sample information $Y_1$ is contained in the prior.)

Either way, you would need to come up with $p(\mu_2|\mu_1)$, the distribution for $\mu_2$ if you knew $\mu_1$, and $p(\mu_1|Y_1)$, which is what you know about $\mu_1$ from its own dataset.

Updating

Suppose we get an additional observation for entity 2 so that $Y_2' = (Y_2, y_2')$. Then \begin{equation} p(\mu_2|Y_1,Y_2') = \frac{p(y_2'|\mu_2)\,p(\mu_2|Y_1,Y_2)}{p(y_2|Y_1,Y_2)} . \end{equation}

Nuisance parameters.

If there are nuisance parameters, they can be integrated out. For example, \begin{equation} p(Y_i|\mu_i) = \int p(Y_i|\mu_i,\sigma_i^2)\,p(\sigma_i^2|\mu_i)\,d\sigma_i^2 . \end{equation} To flesh this example out a bit, suppose \begin{equation} p(Y_i|\mu_i,\sigma_i) = \prod_{t=1}^{T_i} \textsf{N}(y_{it}|\mu_i,\sigma_i^2) \end{equation} and $p(\sigma_i^2|\mu_i) = \textsf{Inv-Gamma}(\sigma_i^2|a,b)$. In this case, \begin{equation} p(Y_i|\mu_i) \propto \textsf{Student}(\mu_i|m_i,s_i^2,\nu_i) , \end{equation} where $(m_i,s_i^2,\nu_i)$ depend on $Y_i$ [and $(a,b)$].

Hyperparameters.

Sometimes it is convenient to express the prior via hyperparameters. In particular, suppose $\mu_1$ and $\mu_2$ are conditionally independent, conditional on the hyperparameter $\phi$: \begin{equation} p(\mu_1,\mu_2|\phi) = p(\mu_1|\phi)\,p(\mu_2|\phi) . \end{equation} The dependence between $\mu_1$ and $\mu_2$ follows from the joint variation due to $\phi$: \begin{equation} p(\mu_1,\mu_2) = \int p(\mu_1|\phi)\,p(\mu_2|\phi)\,p(\phi)\,d\phi . \end{equation} (The symmetric prior implied by the use of the hyperparameter may not be appropriate for your problem.)

Having introduced the hyperparameter, it may be convenient to delay integrating it out until a later stage of the analysis. Note that the joint posterior distribution can be expressed as \begin{equation} p(\mu_1,\mu_2,\phi|Y_1,Y_2) \propto p(Y_1|\mu_1)\,p(Y_2|\mu_2)\,p(\mu_1|\phi)\,p(\mu_2|\phi)\, p(\phi). \end{equation} A Gibbs sampler could be based on the complete set of full conditional posterior distributions: \begin{align} p(\mu_1|Y_1,\phi) &\propto p(Y_1|\mu_1)\,p(\mu_1|\phi) \\ p(\mu_2|Y_2,\phi) &\propto p(Y_2|\mu_2)\,p(\mu_2|\phi) \\ p(\phi|\mu_1,\mu_2) &\propto p(\mu_1|\phi)\,p(\mu_2|\phi)\,p(\phi) . \end{align} There is a sense in which the hyperparameter is the carrier of information between the entities.

Note that $\mu_1$ and $\mu_2$ are treated symmetrically here. In addition to learning about $\mu_2$ from $Y_1$, we also learn about $\mu_1$ from $Y_2$.