From https://en.wikipedia.org/wiki/Bayesian_inference

Scenario: A series of N points(${x_{1,..,n}}$) are drawn from distribution p(x|$\theta$),here $\theta$ is a parameter for distribution and itself being a random variable. After getting evidence from N data points, we get a better(less variability) distribution for $\theta$,in short we are guessing value of $\theta$ using Bayes theorem $$ P(\theta \mid {x_{1,..,n}}) = \frac{P(x_{1,..,n} \mid \theta) \, P(\theta)}{P(x_{1,..,n})} $$

I get from other literature that we calculate $P(x_{1,..,n} \mid \theta)$ as $\prod_{k=1}^n P(x_{k} \mid \theta) $ ,this assumes that observations are independent

How well is the assumption justified? Since each observation adds some information to guess $\theta$,next observation totally depends on previous one, in fact exploiting this evidence clues we even try to predict what'd be distribution for next observation by marginalising posterior over prior

$$P(x_{new} \mid x_{1,..,n}) =\int_\theta P(x \mid \theta) P(\theta \mid {x_{1,..,n}}) d\theta$$

  • 2
    $\begingroup$ I edited your title to something more meaningful, feel free to change it if you do not find it appropriate. $\endgroup$
    – Tim
    Oct 29 '17 at 14:32

We only assume that the $x_i$'s are independent conditional on $\theta$, that is, $$P(x_{1,..,n} \mid \theta)=\prod_{k=1}^n P(x_{k} \mid \theta).$$ This means that if we think of the model parameters as some fixed constants, then the $x_i$'s are independent which is often reasonable. The model parameters $\theta$ are fixed constants but we still use a probability distribution to represent our limited state of knowledge about them.

Even though we have conditional independence, this do not imply marginal independence, however, that is, when integrating out $\theta$, we no longer have $$P(x_{1,..,n})=\prod_{k=1}^n P(x_{k})$$ and the distribution of $x_{new}$ conditional on the data (the posterior predictive density) indeed depend $x_{1,..n}$.

  • $\begingroup$ While the answer is correct, it might be a good idea to say what actually the conditional dependence is (it is possible that OP did not hear about it and this has led to confusion). $\endgroup$
    – Tim
    Oct 29 '17 at 14:26
  • $\begingroup$ Yes Tim I didn't understand answer :( $\endgroup$
    – user182728
    Oct 29 '17 at 14:30
  • $\begingroup$ @ZaidSyedMMd I added some more explanation. $\endgroup$ Oct 29 '17 at 16:16
  • $\begingroup$ @ZaidSyedMMd if it does please consider "accepting" the answer (the "v" button under the score on the left). $\endgroup$
    – Tim
    Oct 31 '17 at 9:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy