compare weighted coins I have n (much larger than 2) coins that can give Head or Tail in a toss. For each coin, I have one sequence of trials which may have different lengths. For example, a conceptualized dataset could look like:
Coin 1: HHT
Coin 2: HHTTHHTHTTHHHTTHTHHHHH
Coin 3: THHHTTHHTTTTTTT
I would like to test whether all coins are identical, i.e., whether they have the same probability of giving Head or Tail. How would you approach this problem? Thank you.
 A: You could perform a $C$ likelihood ratio tests, where $C$ is the number of coins.
Let $L(\theta|y) = \prod_{c\neq c_j} \operatorname{Binomial}(y_c; n_c,\theta_c)\cdot\operatorname{Binomial}(y_{c_j}; n_{c_j},\theta_{c_j})$. We can test $H_0: \mbox{all $c_i$ are equal}$ vs. $H_j:c_i=c_{i'},\; \forall i,i' \neq j$, by finding the tail probability of the test statistic
$$T_j(y) = -2(\log L(\hat{\theta}_0|y) - \log L(\hat{\theta}_j|y)) $$
assuming that it follows a $\chi^2_1$ distribution. Here $\hat{\theta}_0$ maximizes $L(\theta|y)$ with the constraint of $H_0$ and $\hat{\theta}_j$ maximizes over the constraint of $H_j$. Since there are possibly a lot of p-values, you'll want to adjust them for multiple comparisons, e.g. using Bonferroni.
Intuitively, this is a way to see if one coin is behaving differently than the rest.
A: I think the Tukey–Kramer method is appropriate here. Implemented in R as TukeyHSD. Although it does make a normality assumption, it should be fine as long as your samples are larger than about 20 or so for each coin. If the means are reasonably close together, the homogeneity of variance assumption will roughly hold as well.
Quoting wikipedia:

When doing all pairwise comparisons, this method is considered the best available when confidence intervals are needed or sample sizes are not equal

More details here
A: Especially if some of the coin sequences are relatively short (so it is important to model the number of heads from a binomial rather than normal distribution), one option is to fit a random-effects binomial model to the data (i.e. with a random effect of coin), and to compare it to a model without the random effect using a likelihood ratio test.
Edited to add:  If the likelihood ratio test supports the random effects model, then it is appropriate to conclude that the coins are not identical.
