11
$\begingroup$

Let's say we are repeatedly tossing a fair coin, and we know number of heads and tails should be roughly equal. When we see a result like 10 heads and 10 tails for a total of 20 tosses, we believe the results and are inclined to believe the coin is fair.

Well when you see a result like 10000 heads and 10000 tails for a total of 20000 tosses, I actually would question the validity of the result (did the experimenter fake the data), as I know this is more unlikely than, say a result of 10093 heads and 9907 tails.

What is the statistical argument behind my intuition?

$\endgroup$
21
$\begingroup$

Assuming a fair coin the outcome of 10000 heads and 10000 tails is actually more likely than an outcome of 10093 heads and 9907 tails.

However, when you say that a real experimenter is unlikely to obtain an equal number of heads and tails, you are implicitly invoking Bayes theorem. Your prior belief about a real experiment is that Prob(No of heads = 10000 in 20000 tosses | Given that experimenter is not faking) is close to 0. Thus, when you see an actual outcome that the 'No of heads = 10000' your posterior about Prob(Experimenter is not faking | observed outcome of 10000 heads) is also close to 0. Thus, you conclude that the experimenter is faking the data.

$\endgroup$
  • $\begingroup$ Very well explained! What a wonderful example to the Bayes theorem approach. $\endgroup$ – Tal Galili Nov 23 '10 at 18:04
  • 1
    $\begingroup$ @Srikant: that prior cannot be formally defined. In any case, Prob(No of heads = X | experimenter is not faking) is always around zero when N=20000, no matter the value of X and no matter your prior. So your posterior for any number is also always close to 0. I don't see what this has to do with Bayes theorem. $\endgroup$ – Joris Meys Nov 25 '10 at 15:29
  • $\begingroup$ All of this from a guy that was holed up trying to prove god existed. Elegant, really. $\endgroup$ – Brandon Bertelsen Nov 25 '10 at 16:29
  • 1
    $\begingroup$ Putting this in a more general perspective, the point, with which I agree, is that Bayes theorem is at work here. Specifically there is are alternative likelihoods (corresponding to different generative processes) for cheating and for honest experimenters. Establishing the cheating is posterior inference with respect to the intuitive and therefore woefully underspecified cheater process. $\endgroup$ – conjugateprior Nov 28 '10 at 16:27
  • 1
    $\begingroup$ @Srikant @whuber: the combinatorials... you're right. I started from an uniform probability, which is off course nonsense in this case. My bad $\endgroup$ – Joris Meys Dec 12 '10 at 21:34
12
$\begingroup$

I like Srikant's explanation, and I think the Bayesian idea is probably the best way to approach a problem like this. But here is another way to see it without Bayes: (in R)

dbinom(10, size = 20, prob = 0.5)/dbinom(10000, 20000, 0.5)

which is about 31.2 on my system. In other words, it is over 30 times more likely to see 10 out of 20 than it is to see 10,000 out of 20,000, even with a fair coin in both cases. This ratio increases without bound as the sample size increases.

This is a sort of likelihood ratio approach, but again, in my gut this feels like a Bayesian judgement call more than anything else.

$\endgroup$
  • $\begingroup$ Why the ratio? Why not just state that the probability of that exact draw is extremely low? $\endgroup$ – Andy W Nov 10 '10 at 17:06
  • 5
    $\begingroup$ The assertion that a particular probability is low out of context isn't convincing. The probability that I am exactly as tall as my height (whatever that may be) is zero. And, yes, it's problematic to even define height with infinite precision, yada, yada, yada... My point is that the maelstrom of existence churns with events of infinitesimal probability happening all the time! 10,000 out of 20,000 - out of context - doesn't surprise me at all. Regardless of what its numerical probability may be. $\endgroup$ – user1108 Nov 11 '10 at 0:12
9
$\begingroup$

A subjectivist Bayesian argument is practically the only way (from a statistical standpoint) you could go about understanding your intuition, which is--properly speaking--the subject of a psychological investigation, not a statistical one. However, it is patently unfair--and therefore invalid--to use a Bayesian approach to argue that an investigator faked the data. The logic of this is perfectly circular: it comes down to saying "based on my prior beliefs about the outcome, I find your result incredible, and therefore you must have cheated." Such an illogical self-serving argument obviously wouldn't stand up in a courtroom or in a peer review process.

Instead, we could take a tip from Ronald Fisher's critique of Mendel's experiments and conduct a formal hypothesis test. Of course it's invalid to test a post hoc hypothesis based on the outcome. But experiments have to be replicated to be believed: that's a tenet of the scientific method. So, having seen one result we think might have been faked, we can formulate an appropriate hypothesis to test future (or additional) results. In this case the critical region would comprise a set of results extremely close to the expectation. For instance, a test at the $\alpha$ = 5% level would view any result between 9,996 and 10,004 as suspect, because (a) this collection is close to our hypothesized "faked" results and (b) under the null hypothesis of no faking (innocent until proven guilty in court!), a result in this range has only a 5% (actually 5.07426%) chance of occurring. Furthermore, we can put this seemingly ad hoc approach in a chi-square context (a la Fisher) simply by squaring the deviation between the observed proportion and the expected proportion, then invoking the Neyman-Pearson lemma in a one-tailed test at the low tail and applying the Normal approximation to the Binomial distribution.

Although such a test cannot prove fakery, it can be applied to future reports from that experimenter to assess the credibility of their claims, without making untoward and unsupportable assumptions based on your intuition alone. This is much more fair and rigorous than invoking a Bayesian argument to implicate someone who might be perfectly innocent and just happened to be so unlucky that they got a beautiful experimental result!

$\endgroup$
5
$\begingroup$

I think your intuition is flawed. It seems you are implicitly comparing a single, "very special" result (exactly 10000 heads) with a set of many results (all "non-special" numbers of heads close to 10000). However, the definition of "special" is an arbitrary choice based on our psychology. How about binary 10000000000000 (decimal 8192) or Hex ABC (decimal 2748) - would that be suspiciously special as well? As Joris Meys commented, the Bayes argument would essentially be the same for any single number of heads, implying that each outcome would be suspicious.

To expand the argument a bit: you want to test a hypothesis ("the experimenter is faking"), and then you choose a test statistic (number of heads). Now, is this test statistic suited to tell you something about your hypothesis? To me, it seems the chosen test statistic is not informative (not a function of a parameter specified as a fixed value in the hypothesis). This goes back to the question what you mean by "cheating". If that means the experimenter controls the coin at will, then this is not reflected in the test statistic. I think you need to be more precise to find a quantifiable indicator, and thus make the question amenable to a statistical test.

$\endgroup$
  • $\begingroup$ +1, But I'm not convinced. What's special about 10,000 is that it exactly equals the expected number of heads under the hypothesis that the coin is fair. This fact is independent of any psychology or system of number representation. The analysis in this response might provide some insight into a situation where, say, 20,005 coins were flipped and 10,000 heads (and therefore 10,005 tails) were noted and somebody's "intuition" suggested fakery took place. $\endgroup$ – whuber Dec 12 '10 at 18:18
  • $\begingroup$ I fully agree that - as you point out in your answer - it all depends on the a-priori definition of the hypothesis: if you define in advance that by "faking the experiment" you mean "achieving a result for number of heads that is close to the expected value", then that's a basis for a statistical test with "number of heads" as a test statistic. However, without such a-priori clarification, the meaning of "faking" and "special value for number of heads" remain cloudy, and it's not clear what they have to do with each other. $\endgroup$ – caracal Dec 13 '10 at 16:06
4
$\begingroup$

The conclusion you draw will be VERY dependent on the prior you choose for the probability of cheating and the prior probability that, given the flipper is lying, x heads are reported.

Putting the most mass on P(10000 heads reported|lying) is a little counter intuitive in my opinion. Unless the reporter is naive, I can't imagine anyone reporting that kind of falsified data (largely for the reasons you mentioned in the original post; it's too suspicious to most people.) If the coin really is unfair and the flipper were to report falsified data, then I think a more reasonable (and very approximate) prior on the reported results might be a discrete uniform prior P(X heads reported|lying) = 1/201 for the integers {9900, ..., 10100} and P(x heads reported|lying) = 0 for all other x. Suppose that you think the prior probability of lying is 0.5. Then some posterior probabilities are:

P(lying|9900 heads reported) = P(lying|10100 heads reported) = 0.70;

P(lying|9950 heads reported) = P(lying|10050 heads reported) = 0.54;

P(lying|10000 heads reported) = 0.47.

Most reasonable numbers of reported heads from a fair coin will result in suspicion. Just to show how sensitive the posterior probabilities are to your priors, if the prior probability of cheating is lowered to 0.10, then the posterior probabilities become:

P(lying|9900 heads reported) = P(lying|10100 heads reported) = 0.21;

P(lying|9950 heads reported) = P(lying|10050 heads reported) = 0.11;

P(lying|10000 heads reported) = 0.09.

So I think the original (and highly rated answer) could be expanded a little bit; in no way should you conclude that the data is falsified without thoroughly considering prior information. Also, just thinking about this intuitively, it seems that the posterior probabilities of lying are likely to be influenced more by the prior probability of lying rather than by the prior distribution of heads reported given that the flipper is lying (except for priors that put all their mass on a small number of heads reported given the flipper is lying, such as in my example.)

$\endgroup$
  • $\begingroup$ I think this is a very good answer, but I disagree with your second paragraph. I don't think Srikant's original conditional probability is counter-intuitive, and simply because it is a hard question to answer is not an argument against it. I also don't think your uniform probability of lying within 9900 to 10100 makes any sense at all, although it is useful for demonstration purposes. $\endgroup$ – Andy W Dec 12 '10 at 23:47
2
$\begingroup$

For the Bayesian explanation, you need a prior probability distribution on the reported results by a lying coin flipper, as well as a prior probability of lying. When you see a value that is much more likely under the lying distribution than the random flips one, that makes your posterior probability of lying much higher.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.