# confusing notion in Bayesian inference

My confusion is on the observed data $D$, I think for $D$, (1) it could be only one observation, and it could be multiple observation, correct? (2) when there are multiple observations, do we need to assume the multiple observation are independent? I have the confusion here since I referred some text book and wiki, but do not see it is always mentioned the multiple observations are independent. Want to confirm with the expert here, (3) when there are multiple observations and they are independent, then $P(D_1,D_2,...,D_n|w)$ = $P(D_1|w)*P(D_2|w)*...*P(D_n|w)$, correct?

The following material is reproduced from Christopher Bishop's Pattern Recognition and Machine Learning.

• can you give the book name? – hxd1011 Sep 12 '16 at 7:30
• Sure @Xi'an, never did this before. you mean book name or something else? – Lin Ma Sep 13 '16 at 7:31
• @hxd1011, sure pattern recognition and machine learning. – Lin Ma Sep 13 '16 at 7:31

Exchangeability is often interpreted as the Bayesian version of i.i.d assumption . In i.i.d assumption we assume that the parameter we are interested in exists in some meaningful sense and the data at hand (1 or $n$ observations does not matter ) were generated as independent trials conditional on the parameter .

Exchangeability is the condition that the joint density of the data remains the same under re-ordering or re-labeling of the indices of the data , so if we assume for example a binomial model then under exchangeability two sequence of random variables , each with the same length $n$ and the same proportion of ones would be assigned the same probability only the number of ones matters not the location of the ones . So that the assumption of exchangeability in Bayesian inference means we believe that the data are exchangeable then it is as if there is a parameter (say $\theta$) that derives a stochastic model generating the data and a density over $\theta$ that does not depend on the data , this density interpretable as a prior density thus the existence of a prior density over a parameter is a result of the exchangeability rather than an assumption

Edit based on Xi'an's comment :

Some models allow for exchangeability and others do not .In the hierarchical models also we can not consider the entire sequence of data as being exchangeable instead the data within any given group might be considered exchangeable that means we condition on the group and that is so called conditional exchangeability

• Thanks Bahgat for the reply and vote up. After reading your description, I think the answer to my 3 questions are, (1) could be either single or multiple observations, (2) observations are i.i.d. (3) the formula is correct, could you help to review and correct/confirm? Thanks again. – Lin Ma Sep 12 '16 at 2:40
• @LinMa Yes you are right we assume i.i.d data generating process ,I've had also the same problem with Bayesian books notation , e.g using $\theta$ instead of $\Theta$ could refer to multiple parameters so just consider $\theta$ ( and $D$ in your case ) as container contains all parameters (data points ) of interest . – Bahgat Nassour Sep 12 '16 at 2:55
• @Xi'an you are right , see edit – Bahgat Nassour Sep 12 '16 at 18:01
• @LinMa: I am surprised that you accepted this answer it does not answer your question, but describes instead another notion, exchangeability, that is not mentioned or even considered in the excerpt you provided. – Xi'an Sep 13 '16 at 9:25
• The dependence structure in the data should not be dependent (!) on the type of inference one applies to this data, I think, as Bayesian and non-Bayesian methods start from the same statistical model $P(\mathfrak{D}|w)$. It is only when this model is uncertain that the two approaches differ, Bayesian solutions moving to a meta-model. – Xi'an Sep 14 '16 at 6:43

Just on a technicality, the proper decomposition is:

$$P (D_1,...,D_n|w)=P(D_1|w)\times P (D_2|D_1 w)\times\dots P (D_n|D_1 D_2\dots D_{n-1}w)$$

This is always correct - but when assuming conditional independence, it simplifies to the expression you have in the OP. Time series models are examples where you do not have conditional independence (eg auto-regressive or AR models). Standard linear regression models are examples where you do have conditional independence.

• BTW, probabilityislogic, trying to derive your formula from right side to left side, here is my process (using two variables D1 and D2), could you double check if it is correct (I mean my below calculation process does not assume i.i.d. of D1 and D2, correct)? Thanks. $P(D1|w)×P(D2|D1w)$=$[P(D1,w)/P(w)] * [P(D2, D1, w)/P(D1, w)]$ =$P(D2, D1, w)/P(w)$ = $P(D1, D2|w)$ – Lin Ma Sep 13 '16 at 7:53
• Hi probabilityislogic, if you could comment on my above question/comment, it will be great. – Lin Ma Sep 14 '16 at 6:20
• @Lin Ma - that is correct. And you can repeatedly apply this (say with $D2=A2A3$) to get more than two variables – probabilityislogic Sep 15 '16 at 11:09

While the answer of probabilityislogic is making the right point and is thus the answer so far, let me add that any dependence structure within a sample $\mathfrak{D}$ is achievable with the adequate probability model. Statistical inference is possible for such models if there is some degree of repeatability. For instance, assuming a Markov dependence as in the OP means that the conditional distribution of each datapoint given the previous one is the same for all indices. If one does not wish to make any assumption about the dependence structure of the sample $\mathfrak{D}$, this amounts to observing a single realisation of the vector $\mathfrak{D}$, with an unknown distribution for which inference based on $\mathfrak{D}$ only is impossible (or unreliable).