Statistical significance with insufficient data There is an article in Wikipedia which talks about p-values. In the example section it gives this example:

One roll of a pair of dice
Suppose a researcher rolls a pair of dice once and assumes a null hypothesis that the dice are fair, not loaded or weighted toward any specific number/roll/result; uniform. The test statistic is "the sum of the rolled numbers" and is one-tailed. The researcher rolls the dice and observes that both dice show 6, yielding a test statistic of 12. The p-value of this outcome is 1/36 (because under the assumption of the null hypothesis, the test statistic is uniformly distributed) or about 0.028 (the highest test statistic out of 6×6 = 36 possible outcomes). If the researcher assumed a significance level of 0.05, this result would be deemed significant and the hypothesis that the dice are fair would be rejected.
In this case, a single roll provides a very weak basis (that is, insufficient data) to draw a meaningful conclusion about the dice. This illustrates the danger with blindly applying p-value without considering the experiment design.

You obviously would not publish a paper on the result, yet the p-value is statistically significant. What are some measures to prevent this type of "error"?
PS. It would be great if both a frequentist and a Bayesian methods presented.
 A: This is indeed a weak point of hypothesis testing...it allows you to make "strong" sounding statements even though the data are weak. 
Now, if 100 people all performed this same weak experiment, you'd expect about 3 people to get this result. So, if the dice were biased towards 6, then this is evidence for that hypothesis (or at least that they are not fair).
This is a totally valid experiment and conclusion. However, it is incomplete. What we want to know is the range of biases supported by this result (I'll get to this more in a sec.) and the probability that you could reproduce it.
The range of supported probabilities is harder to get than just a Reject/Do Not Reject decision for a hypothesis test. You will need to specify a model for the probabilities (which can be anything from a simple 6-parameter saturated model, to a model where the probability of rolling each number is a function of some underlying parameter $\theta$.) What you would find is that this result, while statistically significant, places very, very loose bounds on the range of actual face probabilities for each die. This is where the weakness would show.
The second way to evaluate this is to use a Bayesian model with a uniform Dirichlet on the prior probabilities for each die face. Then, you can calculate the posterior predictive probability (see (1) on p.4) of rolling a 12 and compare it to the null model of equal face probabilities. You will see that the actual "bump" in the probability of rolling a 12 will increase only slightly, not, say, from $3\%$ to $10\%$. This is another indication of the weakness of the result.
A: You have a better chance of rolling more than 3 double-sixes in 45 rolls than you do rolling a double six in only a single roll so the author's single-roll-provides-a-very-weak-basis argument is itself pretty weak. 
In fact, it doesn't matter how sophisticated your experimental design is--if your chosen significance level is 5% then 5% of the time you'll end up with statistically significant results by chance alone.

Side note: Bayesians don't use p-values. In fact, some Bayesians will even get a crazed look in their eye if you even mention 'p-values'.
