0
$\begingroup$

tl;dr: I have N weighted coins, each of which has been flipped some number of times (n_i), and for which I've measured the probability of a heads (p_i). How do I determine whether my largest p_i is significant?

Example: I have 3 coins as follows:

Coin number Number of flips p(heads)
1 100 0.72
2 200 0.51
3 50 0.50

Current thoughts:

  • I think this is somewhat related to the multiple comparisons problem and family-wise error rate? But these aren't really statistical tests with a notion of significance.
  • Maybe just plotting a bunch of beta functions is the best way to visualize this?

Background: I'm looking at the winrates of ~200 cards in Dominion, specifically those bought from the Black Market. Because the Black Market is randomly seeded each game, and some cards haven't been around as long, each one has a different number of occurrences. Generally, winrates for these cards vary around the 60% mark, with a few outliers in the 40s or 70s (ignore the "corrected" column for now): see image

$\endgroup$
1
  • $\begingroup$ Frequentist statistical significance is largely a question of how big $n$ is. Eyeballing your table, it is clear that the coin 1's flip outcomes would very unlikely result from a fair coin (small $p$-value). $\endgroup$ Jul 13, 2023 at 19:26

1 Answer 1

0
$\begingroup$

If I am understanding your question correctly, you are asking how the sample size affects the underlying probability of the coin (or card).

In this case, I refer you to Laplace's Rule of Succession. The Wikipedia article provides good intuition, but essentially, we want to consider the data along with observing a successful event and an unsuccessful event. In the case of the coins, if we had no data, after two flips we would expect to have seen one head and one tail. Now, including these two flips with the observed data, we can update our prior belief about the probability of the coin.

For the first coin, where after 100 flips we observed 72 heads, the probability that our next flip is a head is $\frac{72 + 1}{100 + 2} \approx 0.716$.

In the Dominion case, assuming the cards have either a successful or unsuccessful outcome, the same rule applies.

$\endgroup$
2
  • $\begingroup$ I don't think this applies? I'm less concerned with the underlying probability of the coins, and more interested in how you determine whether your most heavily weighted coin actually IS the most heavily weighted. Maybe phrased another way, my question is closer to this: In the XKCD comic 882 (explainxkcd.com/wiki/index.php/882:_Significant), we see 1/20 erroneously identified outliers, based simply on the p-value chosen. In situations where we didn't choose a p-value, and instead just have raw data, how frequently do we expect erroneous outliers? $\endgroup$ Jul 7, 2023 at 21:16
  • $\begingroup$ My apologies, I misunderstood. In the case of the coins, I think you could just apply Bayesian inference and choose the coin with highest mean in the posterior distribution. $\endgroup$
    – djr
    Jul 8, 2023 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.