Puzzeled with the independence of two R.V.s

Question

I understand that intuitively two R.V.s are independent if knowing one does not change the probability of the other one. This intuitive definition can also be shown mathematically. However, I am a bit confused by the following example.

Let's say we know $X_1$ and $X_2$ are two Gaussian random variables with unknown parameter $\theta$, i.e., $X_1, X_2 \sim \mathcal{N}(\theta,1)$. Now if we know either $X_1$ or $X_2$, then we can infer something about $\theta$ and hence knowing one changes the probability of the other. Consequently, $X_1$ and $X_2$ are dependent by the intuitive definition. I know something is wrong with this logic, otherwise we cannot have independent and identically distributed (i.i.d) random variables.

Could someone help me on this?

Thanks.

Answer 1

From a frequentist perspective, the parameter $\theta$ is a fixed value. So even though you might be able to infer information about the value of $\theta$, all you're doing is constructing an estimate, $\hat{\theta}$ as a function of the observed outcomes of, say, $X_1$. But if you try to make a statement about the probability of $\theta$ taking a particular value, you're making an error, because it's not a random variable itself.

Trying to talk about, say, $P(X_2 | X_1)$, you'd be tempted to build something out of Bayes' theorem, but then you'd discover that you're using something like $P(\theta = t | X_1 = x_1)$, and that probability is $1$ for the real value of $\theta$, and $0$ otherwise, so the conditional probability $P(X_2 | X_1)$ will just collapse back to $P(X_2)$ as a function of $\theta$. And thus the two variables are still independent, with distributions that both happen to be functions of the same, unknown but estimable, parameter.

In a Bayesian framework, eh, kind of, but I'm insufficiently Bayesian to explain how it works in that perspective.