# Puzzeled with the independence of two R.V.s

I understand that intuitively two R.V.s are independent if knowing one does not change the probability of the other one. This intuitive definition can also be shown mathematically. However, I am a bit confused by the following example.

Let's say we know $X_1$ and $X_2$ are two Gaussian random variables with unknown parameter $\theta$, i.e., $X_1, X_2 \sim \mathcal{N}(\theta,1)$. Now if we know either $X_1$ or $X_2$, then we can infer something about $\theta$ and hence knowing one changes the probability of the other. Consequently, $X_1$ and $X_2$ are dependent by the intuitive definition. I know something is wrong with this logic, otherwise we cannot have independent and identically distributed (i.i.d) random variables.

Could someone help me on this?

Thanks.

• When put together as a mixture distribution $X_1,X_2$ are in superposition. There are not two Gaussian functions here, only one. – Carl Nov 18 '16 at 5:23
• @Carl Sorry if the question sounds otherwise but $X_1~\mathcal{N}(\theta,1)$ and $X_2~\mathcal{N}(\theta,1)$. They are not mixture. – abk Nov 18 '16 at 5:52
• If you keep them separate, then you have no question to ask. – Carl Nov 18 '16 at 6:41

From a frequentist perspective, the parameter $\theta$ is a fixed value. So even though you might be able to infer information about the value of $\theta$, all you're doing is constructing an estimate, $\hat{\theta}$ as a function of the observed outcomes of, say, $X_1$. But if you try to make a statement about the probability of $\theta$ taking a particular value, you're making an error, because it's not a random variable itself.
Trying to talk about, say, $P(X_2 | X_1)$, you'd be tempted to build something out of Bayes' theorem, but then you'd discover that you're using something like $P(\theta = t | X_1 = x_1)$, and that probability is $1$ for the real value of $\theta$, and $0$ otherwise, so the conditional probability $P(X_2 | X_1)$ will just collapse back to $P(X_2)$ as a function of $\theta$. And thus the two variables are still independent, with distributions that both happen to be functions of the same, unknown but estimable, parameter.
• Thanks! Even if you cannot get the exact value of $\theta$ and estimate it by $\theta'$, I think this still changes the probability of $X_2$. I cannot see how $\theta$ being random or not changes anything. – abk Nov 18 '16 at 3:45
• It doesn't, because $X_2$ isn't actually affected by $\theta'$. Like I said, it probably gets a little trickier in Bayesian inference, where it actually does make sense to talk about the probability distribution of the parameter conditional on the data. – ConMan Nov 18 '16 at 6:44
• Excellent answer. Thank you! Now it all makes sense, as you correctly pointed out the value of $\theta'$ does not change anything. $\theta$ is fixed from a frequentist perspective. It would be nice to see an explanation from a Bayesian perspective too. – abk Nov 21 '16 at 3:44