11
$\begingroup$

Let's say that I have a friend (let's call him "George") who says that he can control the roll of dice using his mind (ie, make the dice more likely to fall on a specific number he is thinking of).

How can I design a scientifically rigorous test to determine if he can actually do this? (I don't really think he can, of course, but I want him to agree to the details of a test, Amazing Randi-style, before the test begins.) I want to reduce the (very likely) post test-excuses that he'll come up with.

Here is what I have so far:

  1. Determine the physical dice-rolling technique (which dice, shaker cup, landing surface, etc)

  2. Define a "test session", consisting of X rolls of the dice. This needs to be small enough to do in one sitting, but large enough to determine (after analysis) within 95%-99% confidence whether the dice fell fair, or favored one side

  3. Run Y sessions on the chosen dice (with no influence from George), as a "control" to make sure that the dice show "fair" results on their own

  4. Run Z sessions with George. Before each, roll a separate die to determine what number George will "concentrate on" during that entire session.

  5. Compile and analyze the results.

  6. George makes up some excuses for his dismal performance.

So my questions for you:

  • Any flaws or problems with my overall methodology? Anything George would likely object to?

  • Should I use a D6? Or a D20? Does it matter? Would a die with more faces require more rolls to produce similarly confident results? Or the opposite? I'd prefer fewer rolls than more, due to practical considerations :)

  • What are reasonable values for X, Y, and Z? They are not entirely unrelated; if my chosen value of X only allows 95% confidence for a single session, then 1 out of every 20 sessions could "fail", even without George's influence

  • How do I define "success" or "failure" for an individual session? (I did find this question which goes through the details of a chi-squared test, so I think that's my evaluation methodology, but what are reasonable confidence thresholds?)

  • How do I define "success" or "failure" for the overall test? George might "win" a single session by sheer chance, but how many of the Z sessions would he have to pass to pass the whole test?

I'll probably be analyzing these results in a MS Excel spreadsheet, if that makes any difference.

$\endgroup$
  • $\begingroup$ If he's only manipulating the dice with his mind then someone should be throwing them. I think things like D6 or D20 should be left up to George. What kind of manipulation did George say he could do? Did he say he could make a particular number he wished come up? If so, then successes would be instances of that number while failures would be anything else. $\endgroup$ – John Aug 15 '11 at 20:16
  • $\begingroup$ @John - He's only claimed that he can make that number come up more often than it would otherwise, not that it will always come up that number. $\endgroup$ – BradC Aug 15 '11 at 20:32
  • $\begingroup$ (Originally asked a version of this question at math.stackexchange.com/q/57624/14626 ) $\endgroup$ – BradC Aug 15 '11 at 21:01
  • $\begingroup$ your answers on math by Michael Hardy and TonyK are both quite good. $\endgroup$ – John Aug 15 '11 at 21:16
  • 4
    $\begingroup$ I wouldn't be satisfied with a confidence level as low as 95% or even 99%; extraordinary claims tend to require extraordinary evidence. Or to put it in a more Bayesian context, my prior belief of him having such an ability is so low that I would require a ridiculous amount of evidence to meaningfully alter my posterior belief. $\endgroup$ – Michael McGowan Aug 16 '11 at 15:20
2
$\begingroup$

I would recommend analyzing this in the following way:

Count each role in which George successfully predicts the outcome as a success, and every other one as a failure. Then, you easily calculate a probability of success for George, and a 95% or 99% confidence interval. Does he claim he can predict the outcome "twice as well" as randomly rolling the dice? Then:

H0: p >= 1/3

H1: p < 1/3

(assuming a 6-sided die).

From there, it's pretty straightforward to do the hypothesis test. Also, you can calculate the power a priori pretty easily (even in something like Excel). Pick a number of rolls (like 10), and then make a table with the possible successes as rows (0-10). Then, for each success, calculate the probability he'll have that many successes (if he were to be just guessing, which is what we're assuming he is doing). Also, for each value, determine if it would lead to a rejection or acceptance of the null. Then, to find the power, you can simply add up all the probabilities where the null would be rejected.

$\endgroup$
1
$\begingroup$

A D20 will require more rolls for the same significance level for George succeeding for sure if you need to run the chi-square test. I don't think you need to run the full chis-square test though. You just need to check whether the dice rolls the "chosen" number more often than chance. I would just use the cdf of the binomial to calculate the p-value of rolling the chosen number more often than chance with $\theta=\frac{1}{6}$ as the binomial parameter for D6. I think it is easy to determine the number $X$ based upon the p-value you need for George succeeding. I am not even sure you need Z sessions. Why not just run one session for each side of the die. Does randomizing the chosen side even matter for the hypotheses you are interested in?

$\endgroup$
  • $\begingroup$ Randomizing the chosen side probably doesn't matter, I was just concerned about A) Accounting for any possible true bias in the dice and B) Making sure any one single anomalous session (that 1 in 20 due to true randomness I talked about) isn't taken as a confirmation of his psychic ability. Probably 6 sessions, one for each number, would be sufficient, as long as my X (and confidence level) is high enough. $\endgroup$ – BradC Aug 15 '11 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.