Let's say that I have a friend (let's call him "George") who says that he can control the roll of dice using his mind (ie, make the dice more likely to fall on a specific number he is thinking of).
How can I design a scientifically rigorous test to determine if he can actually do this? (I don't really think he can, of course, but I want him to agree to the details of a test, Amazing Randi-style, before the test begins.) I want to reduce the (very likely) post test-excuses that he'll come up with.
Here is what I have so far:
Determine the physical dice-rolling technique (which dice, shaker cup, landing surface, etc)
Define a "test session", consisting of X rolls of the dice. This needs to be small enough to do in one sitting, but large enough to determine (after analysis) within 95%-99% confidence whether the dice fell fair, or favored one side
Run Y sessions on the chosen dice (with no influence from George), as a "control" to make sure that the dice show "fair" results on their own
Run Z sessions with George. Before each, roll a separate die to determine what number George will "concentrate on" during that entire session.
Compile and analyze the results.
George makes up some excuses for his dismal performance.
So my questions for you:
Any flaws or problems with my overall methodology? Anything George would likely object to?
Should I use a D6? Or a D20? Does it matter? Would a die with more faces require more rolls to produce similarly confident results? Or the opposite? I'd prefer fewer rolls than more, due to practical considerations :)
What are reasonable values for X, Y, and Z? They are not entirely unrelated; if my chosen value of X only allows 95% confidence for a single session, then 1 out of every 20 sessions could "fail", even without George's influence
How do I define "success" or "failure" for an individual session? (I did find this question which goes through the details of a chi-squared test, so I think that's my evaluation methodology, but what are reasonable confidence thresholds?)
How do I define "success" or "failure" for the overall test? George might "win" a single session by sheer chance, but how many of the Z sessions would he have to pass to pass the whole test?
I'll probably be analyzing these results in a MS Excel spreadsheet, if that makes any difference.