Suppose Alice (A) and Bob (B) each flip the same, potentially-biased coin. Then, P(A=H) < P(A=H | B=H), because Bob's flip increases our suspicion that the coin is biased towards heads.
Now instead suppose that we already know that the coin is biased towards heads. Now is it true that P(A=H) < P(A=H | B=H)? It seems clear to me that the answer is still yes, because we don't know how much it is biased, and B=H still suggests that it is more biased than if B=T.
To be more precise, regardless of the prior $P(\theta)$ we pick (where $\theta$ is the odds of heads), the posterior $P(\theta | H)$ will still be shifted right. It doesn't matter whether the prior is uniform on [0, 1] (example 1) or (0.5, 1] (example 2).
But this post at The Queen Mary University of London (and this answer at Math SE) seems to suggest otherwise:
Now suppose both Martin and Norman toss the same coin. Again let A represent the variable "Norman's toss outcome", and B represent the variable "Martin's toss outcome". Assume also that there is a possibility that the coin in biased towards heads but we do not know this for certain. In this case A and B are not independent. For example, observing that B is Heads causes us to increase our belief in A being Heads (in other words P(a|b)>P(b) in the case when a=Heads and b=Heads).
In Example 2 the variables A and B are both dependent on a separate variable C, "the coin is biased towards Heads" (which has the values True or False). Although A and B are not independent, it turns out that once we know for certain the value of C then any evidence about B cannot change our belief about A. Specifically:
P(A|C) = P(A|B,C)
In such case we say that A and B are conditionally independent given C.
What am I missing?
Edit: This is increasingly striking me as a frequentist vs Bayesian issue. In scenario 1, we're behaving like a Bayesian (incorporating our knowledge about the parameter into our estimate) and scenario 2 like a frequentist (treating it as a fixed value and therefore not updating it based on data).
I think what's confusing me is that there seems to be no principled reason to switch between the two paradigms, other than a subtlety of wording ("there is a possibility that the coin in biased" in the first scenario). A strong frequentist should apply her paradigm to both cases, and likewise a Bayesian. Would you agree?