5
$\begingroup$

There is an article in Wikipedia which talks about p-values. In the example section it gives this example:

One roll of a pair of dice

Suppose a researcher rolls a pair of dice once and assumes a null hypothesis that the dice are fair, not loaded or weighted toward any specific number/roll/result; uniform. The test statistic is "the sum of the rolled numbers" and is one-tailed. The researcher rolls the dice and observes that both dice show 6, yielding a test statistic of 12. The p-value of this outcome is 1/36 (because under the assumption of the null hypothesis, the test statistic is uniformly distributed) or about 0.028 (the highest test statistic out of 6×6 = 36 possible outcomes). If the researcher assumed a significance level of 0.05, this result would be deemed significant and the hypothesis that the dice are fair would be rejected.

In this case, a single roll provides a very weak basis (that is, insufficient data) to draw a meaningful conclusion about the dice. This illustrates the danger with blindly applying p-value without considering the experiment design.

You obviously would not publish a paper on the result, yet the p-value is statistically significant. What are some measures to prevent this type of "error"?

PS. It would be great if both a frequentist and a Bayesian methods presented.

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ It should be "significance level of '0.05'" and not '0.005' $\endgroup$
    – Saket Choudhary
    Commented Sep 25, 2015 at 22:55
  • 3
    $\begingroup$ P-values are great, aren't they? $\endgroup$
    – Jeremy Miles
    Commented Sep 26, 2015 at 0:19
  • 2
    $\begingroup$ xkcd.com/1132 $\endgroup$
    – Paul
    Commented Sep 26, 2015 at 11:00
  • $\begingroup$ This is a fairly weak p-value resulting from a one-sided hypothesis test. As far as I know, no one advocated blindly testing arbitrary hypotheses, so the suspicion of the dice rolling, on average, more than 7 must be an important scientific question. Furthermore, the 0.05 significance threshold is a regrettable result of people reading Fisher's earlier works. Fisher would have the researcher make his/her own judgements based on new and possibly old evidence while for Neyman/Pearson this would be the single prespecified hypothesis to test. In any case, reviewers would likely ask for replicates. $\endgroup$
    – Rob Hall
    Commented Sep 30, 2015 at 21:14

2 Answers 2

Reset to default
1
$\begingroup$

This is indeed a weak point of hypothesis testing...it allows you to make "strong" sounding statements even though the data are weak.

Now, if 100 people all performed this same weak experiment, you'd expect about 3 people to get this result. So, if the dice were biased towards 6, then this is evidence for that hypothesis (or at least that they are not fair).

This is a totally valid experiment and conclusion. However, it is incomplete. What we want to know is the range of biases supported by this result (I'll get to this more in a sec.) and the probability that you could reproduce it.

The range of supported probabilities is harder to get than just a Reject/Do Not Reject decision for a hypothesis test. You will need to specify a model for the probabilities (which can be anything from a simple 6-parameter saturated model, to a model where the probability of rolling each number is a function of some underlying parameter $\theta$.) What you would find is that this result, while statistically significant, places very, very loose bounds on the range of actual face probabilities for each die. This is where the weakness would show.

The second way to evaluate this is to use a Bayesian model with a uniform Dirichlet on the prior probabilities for each die face. Then, you can calculate the posterior predictive probability (see (1) on p.4) of rolling a 12 and compare it to the null model of equal face probabilities. You will see that the actual "bump" in the probability of rolling a 12 will increase only slightly, not, say, from $3\%$ to $10\%$. This is another indication of the weakness of the result.

Improve this answer
$\endgroup$
1
$\begingroup$

You have a better chance of rolling more than 3 double-sixes in 45 rolls than you do rolling a double six in only a single roll so the author's single-roll-provides-a-very-weak-basis argument is itself pretty weak.

In fact, it doesn't matter how sophisticated your experimental design is--if your chosen significance level is 5% then 5% of the time you'll end up with statistically significant results by chance alone.

Side note: Bayesians don't use p-values. In fact, some Bayesians will even get a crazed look in their eye if you even mention 'p-values'.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Bayesians don't use p-values; as far as I know they use factors. But those fail just as miserably here if the models being compared are (chance of 6 = 1/6) vs (chance of 6 = 5/6) $\endgroup$
    – Anton
    Commented Sep 26, 2015 at 13:17
  • $\begingroup$ So the question is what are some procedures for making sure that your experimental design will collect enough data or be sophisticated enough? $\endgroup$
    – Anton
    Commented Sep 26, 2015 at 13:24
  • $\begingroup$ Bayesians start with "prior probabilities" and then update these "priors" (to form "posterior probabilities") as new information comes along. $\endgroup$
    – Steve S
    Commented Sep 27, 2015 at 4:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.