Suppose you are measuring $n$ quantities with error. Let $\beta_1,\ldots, \beta_n$ represent the true values and $X_1, \ldots, X_n$ represent the measured values of those quantities. Assume that the errors are centered normal. Let $\sigma_i^2\,, i=1, \ldots, n$ represent the known standard deviation of each measurement. So that the measurements are $$ X_i | \beta_i \sim N(\beta_i, \sigma_i^2)\,.$$

I can recover the above model by writing $$ X_i = \beta_i + \varepsilon_i\,,$$ where $\varepsilon_i \sim N(0, \sigma_{i}^2)$.

Now I make the following extension to the model after which I get confused.

Suppose that $\beta_i \sim N(\mu, \sigma_b^2)$, with known parameters $\mu, \sigma_b^2$. I want to write down form of the posterior distribution $p(\beta_i | X)$.

On the one hand, if the relationship $X_i = \beta_i + \varepsilon_i $ is still in force, then $$ \beta_i = X_i - \varepsilon_i $$ so the posterior is $p(\beta_i | \{X_i\}) \sim N(X_i, \sigma_i^2)$ In particular it does not depend on $\sigma_b, \mu$.

On the other hand, if I just proceed by Bayesian theorem then $$ p(\beta_i | \{X_i\}) = \frac{p(\beta_i, \{X_i\})}{P(\{X_i\})} = \frac{p(\{X_i\}|\beta_i) p(\beta_i)}{P(\{X_i\})} \propto p(\{X_i\}|\beta_i) p(\beta_i) = f_1(\{X_i\}| \beta_i) f_2(\beta)\,, $$

with $f_1(\{X_i\}| \beta_i) = \prod_{i=1}^n f(X_i| \beta_i)$ where $ f(X_i; \beta_i)$ is density of $N(\beta_i, \sigma_i^2)$ and $f_2(\beta_i)$ is the density of $N(\mu, \sigma_b^2)$.

The results of those two approaches differ, what am I confusing here?

Added late: As one of the comments suggested my question is related to this question, but the refereed question asks about the specific form of posterior distribution (why the posterior is normally distributed), this is different from what I was trying to figure out.


I assume the intention is that the $\epsilon$s are independent of the $\beta$s (like in a typical measurement noise model).

Then, $\epsilon_i$ and $X_i=\beta_i+\epsilon_i$ are not independent. The error in the first approach is assuming they are and proceeding as if $\epsilon_i \mid X_i \sim N(0, \sigma_i^2)$ would hold.

  • $\begingroup$ Thanks, so full model specification in the first case should be $ X_i = \beta_i + \varepsilon_i $ plus the fact that $\beta_i $ and $\varepsilon_i$ are independent, in that sense $X_i$ are not symmetric to $\beta_i$, since $X_i$ are not independent of $\varepsilon_i$. $\varepsilon_i |X_i$ is degenerate random variable and I have $\varepsilon_i |X_i = (X_i - \beta_i) $. $\endgroup$ – them Aug 27 '16 at 16:35
  • 1
    $\begingroup$ $\epsilon_i\mid X_i$ is not degenerate (as in constant) unless you treat $\beta_i$ as a constant. $\endgroup$ – Juho Kokkala Aug 27 '16 at 16:48
  • $\begingroup$ Juho Kokkala, thanks for pointing out. Under the model I have described it follows that $\varepsilon | X_i \sim N(X_i - \mu, \sigma_b^2)$. $\endgroup$ – them Aug 27 '16 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.