This is indeed a weak point of hypothesis testing...it allows you to make "strong" sounding statements even though the data are weak.

Now, if 100 people all performed this same weak experiment, you'd expect about 3 people to get this result. So, if the dice were biased towards 6, then this is evidence for that hypothesis (or at least that they are not fair).

This is a totally valid experiment and conclusion. However, it is incomplete. What we want to know is the range of biases supported by this result (I'll get to this more in a sec.) and the probability that you could reproduce it.

The range of supported probabilities is harder to get than just a Reject/Do Not Reject decision for a hypothesis test. You will need to specify a model for the probabilities (which can be anything from a simple 6-parameter saturated model, to a model where the probability of rolling each number of a function of some underlying parameter $\theta$.) What you would find is that this result, while statistically significant, places very, very loose bounds on the range of actual face probabilities for each die. This is where the weakness would show.

The second way to evaluate this is to use a Bayesian model with a uniform Dirichlet on the prior probabilities for each die face. Then, you can perform posterior predictive inference

This is indeed a weak point of hypothesis testing...it allows you to make "strong" sounding statements even though the data are weak.

Now, if 100 people all performed this same weak experiment, you'd expect about 3 people to get this result. So, if the dice were biased towards 6, then this is evidence for that hypothesis (or at least that they are not fair).

This is a totally valid experiment and conclusion. However, it is incomplete. What we want to know is the range of biases supported by this result (I'll get to this more in a sec.) and the probability that you could reproduce it.

The range of supported probabilities is harder to get than just a Reject/Do Not Reject decision for a hypothesis test. You will need to specify a model for the probabilities (which can be anything from a simple 6-parameter saturated model, to a model where the probability of rolling each number is a function of some underlying parameter $\theta$.) What you would find is that this result, while statistically significant, places very, very loose bounds on the range of actual face probabilities for each die. This is where the weakness would show.

The second way to evaluate this is to use a Bayesian model with a uniform Dirichlet on the prior probabilities for each die face. Then, you can calculate the posterior predictive probability (see (1) on p.4) of rolling a 12 and compare it to the null model of equal face probabilities. You will see that the actual "bump" in the probability of rolling a 12 will increase only slightly, not, say, from $3\%$ to $10\%$. This is another indication of the weakness of the result.

This is indeed a weak point of hypothesis testing...you can make "strong" sounding statements even though the data are weak.

Now, if 100 people all performed this same weak experiment, you'd expect about 3 people to get this result. So, if the dice were biased towards 6, then this is evidence for that hypothesis (or at least that they are not fair).

This is a totally valid experiment and conclusion. However, it is incomplete.

This is indeed a weak point of hypothesis testing...it allows you to make "strong" sounding statements even though the data are weak.

Now, if 100 people all performed this same weak experiment, you'd expect about 3 people to get this result. So, if the dice were biased towards 6, then this is evidence for that hypothesis (or at least that they are not fair).

This is a totally valid experiment and conclusion. However, it is incomplete. What we want to know is the range of biases supported by this result (I'll get to this more in a sec.) and the probability that you could reproduce it.

The range of supported probabilities is harder to get than just a Reject/Do Not Reject decision for a hypothesis test. You will need to specify a model for the probabilities (which can be anything from a simple 6-parameter saturated model, to a model where the probability of rolling each number of a function of some underlying parameter $\theta$.) What you would find is that this result, while statistically significant, places very, very loose bounds on the range of actual face probabilities for each die. This is where the weakness would show.

The second way to evaluate this is to use a Bayesian model with a uniform Dirichlet on the prior probabilities for each die face. Then, you can perform posterior predictive inference