4
$\begingroup$

Suppose Alice (A) and Bob (B) each flip the same, potentially-biased coin. Then, P(A=H) < P(A=H | B=H), because Bob's flip increases our suspicion that the coin is biased towards heads.

Now instead suppose that we already know that the coin is biased towards heads. Now is it true that P(A=H) < P(A=H | B=H)? It seems clear to me that the answer is still yes, because we don't know how much it is biased, and B=H still suggests that it is more biased than if B=T.

To be more precise, regardless of the prior $P(\theta)$ we pick (where $\theta$ is the odds of heads), the posterior $P(\theta | H)$ will still be shifted right. It doesn't matter whether the prior is uniform on [0, 1] (example 1) or (0.5, 1] (example 2).

But this post at The Queen Mary University of London (and this answer at Math SE) seems to suggest otherwise:

Now suppose both Martin and Norman toss the same coin. Again let A represent the variable "Norman's toss outcome", and B represent the variable "Martin's toss outcome". Assume also that there is a possibility that the coin in biased towards heads but we do not know this for certain. In this case A and B are not independent. For example, observing that B is Heads causes us to increase our belief in A being Heads (in other words P(a|b)>P(b) in the case when a=Heads and b=Heads).

In Example 2 the variables A and B are both dependent on a separate variable C, "the coin is biased towards Heads" (which has the values True or False). Although A and B are not independent, it turns out that once we know for certain the value of C then any evidence about B cannot change our belief about A. Specifically:

P(A|C) = P(A|B,C)

In such case we say that A and B are conditionally independent given C.

What am I missing?

Edit: This is increasingly striking me as a frequentist vs Bayesian issue. In scenario 1, we're behaving like a Bayesian (incorporating our knowledge about the parameter into our estimate) and scenario 2 like a frequentist (treating it as a fixed value and therefore not updating it based on data).

I think what's confusing me is that there seems to be no principled reason to switch between the two paradigms, other than a subtlety of wording ("there is a possibility that the coin in biased" in the first scenario). A strong frequentist should apply her paradigm to both cases, and likewise a Bayesian. Would you agree?

$\endgroup$
  • $\begingroup$ You can think of it as having a prior with point mass 1 over some value p. Then just apply bayes rule and you will see that no evidence will change your believes about the bias of the coin and therefore also not about the results of the coin tosses. $\endgroup$ – Sebastian Mar 17 at 23:24
  • $\begingroup$ There is also this post which explains why knowing the value of C basically means that you already observed an infinite sequence so one observation does not matter at all. $\endgroup$ – Sebastian Mar 17 at 23:33
  • $\begingroup$ @Sebastian C can take on two values: True (coin is biased) or False (coin is not biased). Knowing the value of $\theta$ is a different matter, of course. $\endgroup$ – monk Mar 17 at 23:45
  • $\begingroup$ If the problem is suggesting that there is only one possible value for $\theta$, then perhaps I see what they're getting at. This makes me wonder whether this is a frequentist (parameter is not a random variable) vs Bayesian issue. $\endgroup$ – monk Mar 17 at 23:48
4
$\begingroup$

The quoted section is implicitly assuming that the event $C = \{ \theta > 0.5 \}$ is sufficient to fully describe the parameter, and so it attains conditional independence of the observable coin flips (e.g., there may be an assumption that there is only one allowable value of $\theta$ in the biased range). Contrarily, your own analysis is saying that even if $C$ is true, there is still uncertainty parameter value, so the coin flips still give information about the underlying parameter $\theta$, and so they remain dependent. Your analysis here is more realistic, and I agree with your assertion that there would still be dependence even once you condition on $C$.

This issue has been discussed in detail in O'Neill (2009), which looks at conditional independence and marginal dependence in exchangeable sequences of random variables. You can also find some associated theorems for statistical dependence in coin-flipping in a series of papers on binomial prediction (O'Neill and Puza 2005; O'Neill 2012; O'Neill 2015). These latter papers discuss the "gambler's fallacy", and show that ---under broad conditions--- one obtains a predictive advantage by betting on whichever outcome of the coin-flip has come up the most in the observed data (to take advantage on information about possible bias).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Ah, I didn't get it from your text, but looking at the first paper you cited, it seems that my edit (regarding frequentism vs Bayesianism) is precisely the issue. Would you agree? (And if so, is it worth updating your answer to make this more clear?) Thanks! $\endgroup$ – monk Mar 17 at 23:56
  • $\begingroup$ Also, please see my newest edit. Does anything about the QMUL problem statement suggest to you a reason to switch from Bayesianism to frequentism, as it appears they did? $\endgroup$ – monk Mar 18 at 16:42
  • $\begingroup$ This issue is complicated, and the first linked paper sets out the situation. The classical freqentist method implicitly conditions on the parameter $\theta$ for all probability statements, so it does not make any explicit use of marginal statements about the relationship between observable values that are not conditional on the parameter. The closest we get to marginal statements are probability statements about pivotal quantities. $\endgroup$ – Ben Mar 18 at 21:11
  • $\begingroup$ Thanks. Don't know if you saw my second comment. Does the QMUL problem wording indicate to you that they wanted us to switch from treating $\theta$ as a random variable to an unknown constant? $\endgroup$ – monk Mar 19 at 16:30
  • $\begingroup$ Probably not --- the very fact that they distinguish between marginal and conditional independence suggests that they are using the Bayesian framework. I suspect that they are implicitly using a model where they posit a single alternative value $\theta_1 > \tfrac{1}{2}$, so that the event $C$ is equivalent to exact specification of that value. If they are assuming this then their analysis becomes correct under the Bayesian framework. (Although again, I find your specification more realistic.) $\endgroup$ – Ben Mar 19 at 21:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.