0
$\begingroup$

To put it in context, I was trying to learn Bayesian parameter estimation (by an example of learning the probability of heads of a coin) and was trying to understand the independence of the samples that we get, specifically, how marginal independence of each sample depends on the assumption of our model. Specifically, how unknown and fixed affects independence.

In the frequentist view, we consider a parameter $\theta$ that is is unknown but fixed. How and why does that allow us to consider that then our samples are independent? For me intuitively, that doesn't completely make sense because each coin toss tells us something about the parameter $\theta$ that we are trying to learn and hence, telling us about the probability of the next toss. Why does assuming that a parameter is fixed, removes the other uncertainty and somehow makes the samples independent?

To put my question even more in context, consider the extract of Probabilisitc Graphical Models (by Koller and Friedman) that lead to my confusion/misconception:

enter image description here

For completeness consider their fig 17.3:

enter image description here

The issue is not the graphical model. I've studied these before. I do understand that observing the model separates (D-separates in fact) the r.v's $X_i$, which are the samples. I do understand conditional independence, and that is not confusing me either. However, what is confusing me, is why we are allowed to assume independence in the frequentist view. How come the argument Koller and Friedman doesn't break down in the frequentist case. Specifically, the following sentence confuses me:

If we do not known $\theta$, then the tosses are not marginally independent: Each toss tells us something about the parameter $\theta$, and thereby about the probability of the next toss.

That makes sense to me in the graphical model. However, it seems that argument could be used in the frequentist case too, since the model is unknown. What is special about the fixed that makes this issue magically disappear?

I think this is more of a intuitive/conceptual issue rather than a mathematical misunderstanding, but I am not sure. Anyway know how "fixed" fixes the independence issue?

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ In the frequentist perspective, $\theta$ is an unknown given value, hence it is impossible to condition upon $\theta$. In other words, the conditional distribution of $X_1,\ldots,X_n$ given $\theta$ is the same as the marginal of $X_1,\ldots,X_n$. $\endgroup$
    – Xi'an
    Commented Jan 27, 2015 at 5:44
  • 2
    $\begingroup$ let me add to @Xi'an's execellent comment: in the frequentist view, there's no meaning to $p(x|\theta)$ since that would mean $p(x,\theta)$ and $p(\theta)$ make sense, which (in frequentist view) is wrong. $\endgroup$
    – Yair Daon
    Commented Jan 27, 2015 at 5:50
  • $\begingroup$ @Xi'an so you are saying, even though we do not know $\theta$, the true distribution does have a "given" (i.e. observed by "god"/Nature) value. i.e. Nature knows this value and then generates the data independently? So the data is inherently "uncorrelated"? How is that different from Bayesian's though? The way I interpreted bayesianism was that, they believe that any parameter in the distribution in their prior might have generated the data, but in reality, the parameter is still fixed, we are just assigning a uncertainty on our belief by assuming it might be a r.v. $\endgroup$
    – Charlie Parker
    Commented Jan 27, 2015 at 5:56
  • $\begingroup$ @Xi'an but bayesians don't necessarily believe that $\theta$ is taking all its possible values, right? They only believe that our uncertainty on non-repeatable events are quantifiable. Right? But the parameter is still one, we just don't know which one, so we weight by a probability which ones deserve more "attention". No? If that is the case, how is it that the $x_i$'s independence doesn't hold as in the frequentist? $\endgroup$
    – Charlie Parker
    Commented Jan 27, 2015 at 5:59
  • $\begingroup$ @Pinocchio though bayesians believe the parameter has a distribution, they can still condition on one specific value. $\endgroup$
    – Yair Daon
    Commented Jan 27, 2015 at 6:01

1 Answer 1

Reset to default
4
$\begingroup$

Two random variables are independent iff $p(x,y) = p(x)p(y)$. This is the definition. Now, if we throw in the parameter then $X,Y$ have some distribution that depends on the parameter $\theta$. According to the Bayesian approach, our model now is $X,Y,\theta \sim p(x,y,\theta)$ (tell me if you need clarification on this notation). To show independence we'd want to show $$ p(x,y) =\int p(x,y,\theta)d\theta = \int p(x) p(y, \theta | x) d\theta =\\ = p(x) p(y|x) =p(x)p(y) $$ which might not necessarily hold (we're Bayesians now)!! This is why marginal independence does not hold. However, if you condition on $\theta$ (pretend you know it exactly) - then the two tosses are just independent tosses of a biased coin (or whatever experiment you perform) under fixed, known, conditions. Hence, they are independent.

Lets summarize: if the parameter $\theta$ is a RV, then $X,Y$ depend on it and so knowing $X$ affects the distribution of $\theta$ and, consequently, the distribution of $Y$. If $\theta$ is not a random variable then its "distribution" is fixed - it is a point mass on its true value. Hence nothing you observe about $X$ will change the distribution of $Y$.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I think I see what you mean, but I am still confused about one thing, so the reason that the following doesn't apply in frequentists "Each toss tells us something about the parameter θ, and thereby about the probability of the next toss" is because of your last paragraph? Or why? It seems like a sentence that holds true regardless of which view you hold, no? every coin toss affects our $\theta$. Also, why does that sentence hold even more in a bayesian setting? $\endgroup$
    – Charlie Parker
    Commented Jan 27, 2015 at 6:19
  • 3
    $\begingroup$ I think Xi'an said it before but: in the frequentist setting, there is no randomness in $\theta$. Each outcome might affect our estimate $\hat{\theta}$ of $\theta$, but $Y$ is drawn not using $\hat{\theta}$, but $\theta$. $\endgroup$
    – Sven
    Commented Jan 27, 2015 at 6:34
  • $\begingroup$ I see @Sven thanks that makes sense now. I think I also see how it things make sense in bayesianism, in bayesianism we are updating our belief about $\theta$ (which is represented as a probability value). Which obviously depends on the data. Our belief/uncertainty about the parameter we think it is has to change as we receive more data, otherwise this whole process it silly. Thanks! $\endgroup$
    – Charlie Parker
    Commented Jan 27, 2015 at 6:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.