Return to Answer

Further, the common claim that the t-test is very robust obviously needs modification. It's not like we have some bizarre edge-case here. It's clearly not level-robust in this example -- and we haven't yet touched on its power behaviour, which is even more easily impacted.

pp=replicate(100000,t.test(rbinom(100,1,1/36),mu=1/36,alternative="less")$p.value)
mean(pp<.01)   #  We need this to be close to 0.01
pp=replicate(100000,t.test(rbinom(100,1,1/36),mu=1/36,alternative="greater")$p.value)
mean(pp<.05)   #  We need this to be close to 0.05
pp=replicate(100000,t.test(rbinom(100,1,1/36),mu=1/36)$p.value)
mean(pp<.005)  #  We need this to be close to 0.005

Further, the common claim that the t-test is very robust obviously needs modification. It's not like we have some bizarre edge-case here. It's clearly not level-robust in this example --- and we haven't yet touched on its power behaviour, which is even more easily impacted.

    pp = replicate(100000, t.test(rbinom(100, 1, 1/36), mu=1/36, 
                    alternative="less")$p.value)
    mean(pp<.01)   #  We need this to be close to 0.01
    pp = replicate(100000, t.test(rbinom(100, 1, 1/36), mu=1/36, 
                    alternative="greater")$p.value)
    mean(pp<.05)   #  We need this to be close to 0.05
    pp = replicate(100000, t.test(rbinom(100, 1, 1/36), 
                    mu=1/36)$p.value)
    mean(pp<.005)  #  We need this to be close to 0.005

pp=replicate(100000,t.test(rbinom(100,1,1/36),mu=1/36,alternative="less")$p.value)
mean(pp<.01)
pp=replicate(100000,t.test(rbinom(100,1,1/36),mu=1/36,alternative="greater")$p.value)
mean(pp<.05)
pp=replicate(100000,t.test(rbinom(100,1,1/36),mu=1/36)$p.value)
mean(pp<.005)

pp=replicate(100000,t.test(rbinom(100,1,1/36),mu=1/36,alternative="less")$p.value)
mean(pp<.01)   #  We need this to be close to 0.01
pp=replicate(100000,t.test(rbinom(100,1,1/36),mu=1/36,alternative="greater")$p.value)
mean(pp<.05)   #  We need this to be close to 0.05
pp=replicate(100000,t.test(rbinom(100,1,1/36),mu=1/36)$p.value)
mean(pp<.005)  #  We need this to be close to 0.005

Before the game, the players want to be confident that the dice are not badly biased against them, so they ask Chaina to roll the pair of dice 100 times to help them check the dice are not going to place either of them at a severe disadvantage. The outcome "double-six" is of particular concern for them both; one benefits from it occurring, and the other from it not occurring.

Consequently Chaina rolls the two six-sided dice and records "1" every time they come up double-six (which is an important roll in the game), for each of 100 attempts. If the dice are fair and rolled properly, we should have to a good approximation a series of 100 independent Bernoulli trials (0/1 values), where the probability of 1 on each trial is 1/36.

Imagine, perhaps, that we're recording (say) wavelength of individual photons where the wavelength is quantized (so we measure an extremely narrow range around a specific value for each of two values) and we're seeing one of the wavelengths about 35 times as frequently as the other. Or imagine any other outcome that is bimodal with very narrow peaks and one much more frequent; other example circumstances that could fit this situation are not difficult to come up with.

Before the game, the players want to be confident that the dice are not badly biased against them, so they ask Chaina to roll the pair of dice 100 times to help them check the dice are not going to place either of them at a severe disadvantage. The outcome "double-six" (which is an important roll in the game) is of particular concern for them both; one benefits from it occurring, and the other from it not occurring.

Consequently Chaina rolls the two six-sided dice and records "1" every time they come up double-six, for each of 100 attempts. If the dice are fair and rolled properly, we should have to a good approximation a series of 100 independent Bernoulli trials (0/1 values), where the probability of 1 on each trial is 1/36.

Imagine, perhaps, that we're recording (say) energy of individual photons where the energy is quantized (so we measure an extremely narrow range around a specific value for each of two values) and we're seeing one of the energies about 35 times as frequently as the other. Or imagine any other outcome that is bimodal with very narrow peaks and one much more frequent; other example circumstances that could fit this situation are not difficult to come up with.