2
$\begingroup$

I understand that intuitively two R.V.s are independent if knowing one does not change the probability of the other one. This intuitive definition can also be shown mathematically. However, I am a bit confused by the following example.

Let's say we know $X_1$ and $X_2$ are two Gaussian random variables with unknown parameter $\theta$, i.e., $X_1, X_2 \sim \mathcal{N}(\theta,1)$. Now if we know either $X_1$ or $X_2$, then we can infer something about $\theta$ and hence knowing one changes the probability of the other. Consequently, $X_1$ and $X_2$ are dependent by the intuitive definition. I know something is wrong with this logic, otherwise we cannot have independent and identically distributed (i.i.d) random variables.

Could someone help me on this?

Thanks.

Improve this question
$\endgroup$
3
  • $\begingroup$ When put together as a mixture distribution $X_1,X_2$ are in superposition. There are not two Gaussian functions here, only one. $\endgroup$
    – Carl
    Nov 18, 2016 at 5:23
  • $\begingroup$ @Carl Sorry if the question sounds otherwise but $X_1~\mathcal{N}(\theta,1)$ and $X_2~\mathcal{N}(\theta,1)$. They are not mixture. $\endgroup$
    – abk
    Nov 18, 2016 at 5:52
  • $\begingroup$ If you keep them separate, then you have no question to ask. $\endgroup$
    – Carl
    Nov 18, 2016 at 6:41

1 Answer 1

Reset to default
2
$\begingroup$

From a frequentist perspective, the parameter $\theta$ is a fixed value. So even though you might be able to infer information about the value of $\theta$, all you're doing is constructing an estimate, $\hat{\theta}$ as a function of the observed outcomes of, say, $X_1$. But if you try to make a statement about the probability of $\theta$ taking a particular value, you're making an error, because it's not a random variable itself.

Trying to talk about, say, $P(X_2 | X_1)$, you'd be tempted to build something out of Bayes' theorem, but then you'd discover that you're using something like $P(\theta = t | X_1 = x_1)$, and that probability is $1$ for the real value of $\theta$, and $0$ otherwise, so the conditional probability $P(X_2 | X_1)$ will just collapse back to $P(X_2)$ as a function of $\theta$. And thus the two variables are still independent, with distributions that both happen to be functions of the same, unknown but estimable, parameter.

In a Bayesian framework, eh, kind of, but I'm insufficiently Bayesian to explain how it works in that perspective.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks! Even if you cannot get the exact value of $\theta$ and estimate it by $\theta'$, I think this still changes the probability of $X_2$. I cannot see how $\theta$ being random or not changes anything. $\endgroup$
    – abk
    Nov 18, 2016 at 3:45
  • 1
    $\begingroup$ It doesn't, because $X_2$ isn't actually affected by $\theta'$. Like I said, it probably gets a little trickier in Bayesian inference, where it actually does make sense to talk about the probability distribution of the parameter conditional on the data. $\endgroup$
    – ConMan
    Nov 18, 2016 at 6:44
  • $\begingroup$ Excellent answer. Thank you! Now it all makes sense, as you correctly pointed out the value of $\theta'$ does not change anything. $\theta$ is fixed from a frequentist perspective. It would be nice to see an explanation from a Bayesian perspective too. $\endgroup$
    – abk
    Nov 21, 2016 at 3:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.