Basic Question on Conditional and Unconditional Independence

Question

I am trying to get my facts straight on the following 3 scenarios:

1) I draw a random sample $y$ from $Bernoulli(p)$. Then let $X_1, X_2, ..., X_n$ be conditionally independent $N(\mu_1,\sigma_1)$ if $y = 0$ or $N(\mu_2,\sigma_2)$ if $y = 1$. Are they conditionally identically distributed, unconditionally identically distributed, or not identically distributed? Theoretically $X_1, X_2, ..., X_n$ are not unconditionally independent. However for all practical purposes this dependence will be irrelevant because I will not observe another sample $y$. Correct?

2) I draw a random sample $y_i$ from $Bernoulli(p)$. Then let $X_i$ distributed as $N(\mu_1,\sigma_1)$ if $y_i = 0$ or as $N(\mu_2,\sigma_2)$ if $y_i = 1$ for $i = 1..n$. $X_1, X_2, ..., X_n$ are unconditionally independent. They are also conditionally independent given any subset of $Y_i$. They are unconditionally identically distributed. However they are conditionally not identically distributed (or maybe they are if we are lucky?). Correct? The difference from first scenario is that $y$ is redrawn.

3) I have two urns. One urn contains $n$ numbers sampled from $N(\mu_1,\sigma_1)$ and the second urn contains $n$ numbers sampled from $N(\mu_2,\sigma_2)$. I would like to build a classifier given a number from any of the urns and determine which urn it came from. Basically I am assigning the urn as a class label to each number in the urns. Is it OK to pour the contents of both urns into a pool and treat them as independent samples to be fed to a machine learning algorithm? Are they going to be samples from unconditionally independent random variables, or conditionally independent random variables? Is this matching any of the above scenarios?

Answer 1

A) Regarding the first question, when we draw the Bernoulli a single time:
1) The variables are conditionally i.i.d by assumption.
2) They are unconditionally identically distributed.
3) They are unconditionally dependent.

Proof.
For clarity, I will concentrate on just two rv's, $X, Z$ and the Bernoulli. Using $f$ to represent either a density or a pmf/probability, we are told that

$$f(x,z\mid y) = f(x\mid y)\cdot f(z\mid y)$$

By the chain rule, the joint density of the three is

$$f(x,z,y) = f(x,z\mid y) \cdot f(y)$$

Combining

$$f(x,z,y)=f(x\mid y)\cdot f(z\mid y)\cdot f(y)$$

To obtain the unconditional joint density of $X,Z$ we integrate out $y$

$$f(x,z)=\int_{S_y}f(x\mid y)\cdot f(z\mid y)\cdot f(y) dy$$

Since $Y$ is a Bernoulli the above transforms into

$$f(x,z)=\sum_{i=0}^1f(x\mid y_i)\cdot f(z\mid y_i)\cdot \text{Prob}(y=i) dy$$

$$ = f(x\mid y=0)\cdot f(z\mid y=0)\cdot(1-p) + f(x\mid y=1)\cdot f(z\mid y=1)\cdot p$$

Using $N_1,N_2$ for the densities of two normals we get

$$f(x,z) = N_1(x)N_1(z)(1-p)+N_2(x)N_2(z)p$$

To obtain the marginal distribution of, say, $X$, we integrate out $Z$:

$$f(x) = \int _{S_z}\Big[N_1(x)N_1(z)(1-p)+N_2(x)N_2(z)p\Big]dz$$

$$\implies f(x) = (1-p)N_1(x)+pN_2(x)$$

and analogously we will get

$$f(z) = (1-p)N_1(z)+pN_2(z)$$

The last two results prove 2), and they also tell us that

$$f(x,z) \neq f(x)\cdot f(z)$$

so they prove unconditional dependence.

Regarding whether the latter "matters for all practical purposes", it depends on whether one wants to make inference or take a decision prior to draw the Bernouli or not. Obviously, if the wheels will turn only after the Bernoulli is drawn, and if it is drawn only a single time for ever and ever, then a priori unconditional dependence does not matter.

---

B) Regarding the second question, when we draw a Bernoulli for each variable:
Here we have two conditioning variables, $Y_x, Y_z$. So we are looking at the joint density

$$f(x,z,y_x,y_z) = f(x,z,y_x\mid y_z) \cdot f(y_z) = f(x,z\mid y_x, y_z)\cdot f(y_x) \cdot f(y_z)$$

We also have

$$f(x,z\mid y_x, y_z) = f(x\mid y_x, y_z)\cdot f(z\mid y_x, y_z)$$

and

$$f(x\mid y_x, y_z) = f(x\mid y_x),\;\;\; f(z\mid y_x, y_z) = f(z\mid y_z)$$

This tells us what we already know, that each variable is conditioned on a different (and independent) sigma algebra. It follows that they are conditionally independent.
Are they conditionally identically distributed? In general no, because now we have the joint support of $\{Y_x, Y_z\}$ to consider, that has four possible outcomes. For two of these outcomes that reflect $y_x=y_z$ they will be identically distributed but for the other two, no. So there is a probability that they will be conditionally identically distributed, $p^2 + (1-p)^2$, if this helps somewhere, while with probability $2p(1-p)$, they won't be.

Returning to the joint density and combining,

$$f(x,z,y_x,y_z) = f(x\mid y_x)\cdot f(z\mid y_z)\cdot f(y_x)\cdot f(y_z)$$

If we follow the same steps as before, and integrate out $y_z$ and $y_x$ we will end up with

$$f(x,z) = [(1-p)N_1(x)+pN_2(x)]\cdot [(1-p)N_1(z)+pN_2(z)]$$

which tells us that, here, $X,Z$ are unconditionally i.i.d.

---

C) Regarding the urn question:
Nothing much to contribute here, except to notice that the situation appears compatible with the following two cases, assuming that we draw from each $X$ only once per run:

1) We run scenario $A$ twice and we put the results separately in each urn.

2) We run scenario $B$ twice, then grouped the observations per resulting conditional distribution coming from both $B$-runs.

But in any case, I don't see the benefit of pooling observations from different distributions for learning purposes, that aims at a classifier that will be able to tell them apart afterwards.

Answer 2

Consider the simpler example of a two-part experiment in which we first select one of two differently-biased coins (say $P(\mathrm{Heads}) = p_0$ for one coin and $P(\mathrm{Heads}) = p_1$ for the other, where $p_0\neq p_1$) and then toss the selected coin $n$ times. The choice of coin to be tossed is random: the two coins are picked with probabilities $\pi_0$ and $\pi_1 = 1-\pi_0$ respectively. We also assume that the tosses of the coin are independent sub-experiments of the main experiment, that is, the tosses are conditionally independent regardless of which coin has been selected for tossing.

If we define $X_i$ as having value $1$ or $0$ according as the $i$-th toss resulted in Heads or Tails respectively, then all the above can be summarized by saying that the $X_i$'s are conditionally independent Bernoulli random variables with parameter $p_0$ or $p_1$ depending on which coin was selected. But the $X_i$ are not unconditionally independent random variables. It is easy to calculate (via the law of total probability) that for each $i$, $$ P\{X_i = 1\} = p_0\cdot \pi_0 + p_1\cdot (1-\pi_0),$$ that is, the $X_i$ are identically distributed random variables. But, $$ P\{X_i = 1, X_j = 1\} = (p_0)^2\cdot \pi_0 + (p_1)^2\cdot (1-\pi_0) \neq P\{X_i = 1\}\cdot P\{X_j = 1\}$$ and so , but the $X_i$ are not unconditionally independent random variables even though they are conditionally independent random variables under both the possible conditions. All the above is the probabilist's take on what has been described thus far.

Statisticians, on the other hand, are skeptical people who have different axes to grind. You are correct in that if the results of the $n$ tosses are the only observations that you will ever have, then whether you think of the $X_i$'s as conditionally independent or unconditionally independent is totally irrelevant. But then, the statistician who knows which coin produced the given results of the $n$ tosses but is skeptical of what he has been told are the values of $p_0$ and $p_1$ might be more interested in estimating the value of $p_0$ (or $p_1$ as the case might be) from the given results of the $n$ tosses. If the $n$ tosses resulted in $k$ Heads, then the maximum-likelihood estimate of $p_i$ would be $\frac kn$. Alternatively, the statistician could devise a confidence interval for the estimate of $p_i$. Or, if he does not know which coin produced the given results, then he might be interested in making a decision as to which coin produced the observed results of $k$ Heads in $n$ tosses. The maximum-likelihood decision rule would choose between the two coins according as whichever of the two likelihoods $p_1^k(1-p_1)^{n-k}$ and $p_0^k(1-p_0)^{n-k}$ was larger, whereas the Neyman-Pearson decision rule would find the decision rule that had the largest power among all decision rules that achieved Type I error probability $\alpha$ or less. In all of these methods, the fact that the experiment is really a two-part experiment which the coin to be tossed has been picked randomly with probability $\pi_i$ of selecting the coin with $P(\mathrm{Heads}) = p_i$ is ignored. All we are doing is making various inferences from the data that $n$ tosses produced $k$ Heads.

To take into account the prior probabilities $\pi_0$ and $\pi_1 = 1-\pi_0$ of the the coin selection, a Bayesian statistician would use a decision rule that minimizes the average error probability, or more generally, results in the smallest possible average cost. It is well known that the minimum average error probability decision rule is the maximum a posteriori probability (MAP) rule: pick the coin that has the larger conditional probability given the data, and that the MAP rule reduces to comparing the likelihood ratio $\frac{f_1(x)}{f_0(x)}$ to the threshold $\frac{\pi_0}{\pi_1}$ and deciding that it is the coin with $P(\mathrm{Heads})=p_1$ if and only if $$\frac{f_1(x)}{f_0(x)} = \frac{p_1^k(1-p_1)^{n-k}}{p_0^k(1-p_0)^{n-k}} > \frac{\pi_0}{\pi_1}.$$ In contrast, the maximum-likelihood decision rule compares the likelihood ratio to $1$. Similarly, the Bayesian statistician can make (Bayesian) estimates of $p_i$ or come up with credible intervals in which the parameter being estimated must lie with high probability and these can be different from the more common frequentist estimates found in most texts.