4
$\begingroup$

I would like to check if a coin flips randomly, based on observational data. The catch is, the coin can have two sides, but also three, four, up to nine. The number of sides differs in each observation (each toss).

I can generate observations but it's relatively costly, so generally the sample size is small, like 20, 30, maybe 50, although I could probably go for more if needed.

What's the statistical test to apply here, and how, if there are any tricky aspects?

How many observations do I need? Am I correct in thinking that the more pronounced the bias is, the fewer observations I need to detect it?

EDIT to answer the comments: I know how many sides there are each time and I can control it to some degree. The coin is supposed to flip randomly but I suspect it does not. If it indeed does not flip randomly, I would like to be able to say: "here is the sound statistical evidence that this coin does not flip randomly". Probably something about p-value being less than 0.05, as usual.

By "randomly", I meant each side should have the same probability of "coming up" (1/d, d being the number of sides). The alternative hypothesis is that one of the sides (heads, or the "first" one, it's the same the whole time, it's always present) has a higher probability. I don't care about different tails.

TLDR: I think the coin is biased towards heads. The coin can have more than one tails and the number of tails can change from observation to observation. In other words, the expected probability of heads differs in each observation.

UPDATE: Here are the calculations of p-values for the coin being biased towards heads, based on the answer about Poisson-binomial distribution. Two sets of observations, real data.

> x = c(2,4,6,7,8,2,3,3,4,4,3,3)
> 1-poisbinom::ppoisbinom(9-1,1/x)
[1] 0.0009309207
> b = rep(9, 34)
> b
 [1] 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
> 1-poisbinom::ppoisbinom(7-1,1/x)
[1] 0.02906178

It would seem that the coin is indeed biased, even though the sample sizes are small.

For the interested, the described situation is from a computer game where you can have up to nine units on board. A certain unit gives a bonus to one of your units (including itself) each turn. It is supposed to do it randomly (specifically, the implicit expectation is uniform random), but in practice it seems to select itself more often than it should.

The bonus itself doesn't matter for the purposes of this question. The units are the sides of the coin, and the unit that gives the bonus is heads. Each turn there's a coin flip and the bonus lands on one of the units. @Sextus Empiricus

Improve this question
$\endgroup$
9
  • 1
    $\begingroup$ Obviously a "coin" is a metaphor, which means we can't assume much. In particular, could you elaborate on the mechanism determining the number of sides? Is that random, too, or is it determined? If it's random, what do you know about the distribution? Do you even know the number of sides each time? To get your final question ("how many observations") answered, you will need to tell us your decision making needs. In particular, what are the costs of making false positive or false negative decisions? $\endgroup$
    – whuber ♦
    Commented Jan 9 at 23:06
  • 1
    $\begingroup$ Could you also explain more specifically what you mean by "randomly"? Would that mean (in addition to the usual independence assumptions) that each outcome of a $d$-sided coin has a chance of $1/d$ of appearing? If so, what is the alternative hypothesis? The narrower you can make it, the simpler and more powerful the test will be. Otherwise, it appears you're really testing eight hypotheses simultaneously. The textbook solution is a chi-squared test but your expected counts are so low you would have to resort to its permutation test equivalent. $\endgroup$
    – whuber ♦
    Commented Jan 9 at 23:34
  • $\begingroup$ The change of the number of sides each flip makes this situation difficult. If there are changes each flip, then what about the bias? Is the bias/unfairness, conditional on the number of sides, at least the same every flip? And is there a relationship in the bias for cases with different number of sides, or should the flips with different numbers of sides be considered independent cases? $\endgroup$
    – Sextus Empiricus
    Commented Jan 9 at 23:50
  • $\begingroup$ "this coin does not flip randomly" Do you mean the coin does not flip with a uniform random distribution? Or the coin has a bias for particular outcomes? If not, what do you mean by 'not flip randomly'? $\endgroup$
    – Sextus Empiricus
    Commented Jan 9 at 23:53
  • 1
    $\begingroup$ You are correct that the more biased the "coin" is, the easier it will be to detect. The trouble is, given the situation, it seems that the bias would be able to vary from toss to toss (if the number of sides varies and the number of sides that are tails varies, then how could the bias not vary?) That would, I think make this impossible. It would probably help if you told us what was actually going on, rather than use the coin metaphor (see @whuber 's first comment). $\endgroup$
    – Peter Flom
    Commented Jan 10 at 2:44

1 Answer 1

Reset to default
2
$\begingroup$

If you have a coin with variable probability for heads as function of some other variable, as in your example $p_{heads}(x) = 1/x$, then you can consider the probability for the total number of heads as a Poisson binomial distributed variable, which considers the number of heads in coin flips when the probability of heads can be different for different flips.

A test for the hypothesis $H_0:p_{heads}(x) = 1/x$ versus the alternative $H_a: p_{heads}(x) > 1/x$ can be performed in the usual way by computing a p-value in a comparison of the observed number of heads with the distribution for the number of heads assuming that the null hypothesis is true.

For example say we have 20 coin flips with

x = c(2, 4, 3, 3, 2, 5, 2, 3, 3, 3, 2, 2, 3, 5, 4, 6, 3, 2, 2)

then the distribution according to the null hypothesis is like

example of distribution

In software there are function that compute this for you. For example in r you can use

### observed value of number of faces
x = c(2, 4, 3, 3, 2, 5, 2, 3, 3, 3, 2, 2, 3, 5, 4, 6, 3, 2, 2)

### potential outcome for number of heads
k = 0:length(x)

### compute and plot cumulative distribution     
p = poisbinom::ppoisbinom(k,1/x)    
plot(k,p,
     main = "example of Poisson binomial cumulative distribution",
     ylab = "P(observed heads <= k)", type = "s")
points(k,p, pch = 20)

### compute and plot pmf    
m = poisbinom::dpoisbinom(k,1/x)    
plot(k,m,
     main = "example of Poisson probability mass distribution",
     ylab = "P(observed heads = k)", type ="h")
points(k,m, pch = 20)

### example computation of one-sided hypothesis p-value
### when observed heads is 11
1-poisbinom::ppoisbinom(11-1,1/x)

A p-value in your case would be the one sided tail probability. Say you observe 11 heads and with the example values for $x$ then $\text{p-value} = P(\text{observed} >= 11) \approx 0.04$

  • The question about the number of samples to take can be answered with a power analysis. For a given discrepancy from the null hypothesis you state a desired power (probability to reject the null hypothesis at a given desired significance threshold level), and compute how many observations will reach that power. In your case it might be tricky to compute this since your null distribution that is required for these computations is not fixed as it depends on $x$. If you know the distribution of $x$ then you can potentially compute the power with simulations or approximations.

  • In the above we only look at the total sum of heads. There can be discrepancies from the hypothesis $p_{heads}(x) = 1/x$ that are not noticed by a hypothesis test that considers only the total number of heads. This is the case when $p_{heads}(x) < 1/x$ for some $x$ and $p_{heads}(x) > 1/x$ for other $x$. In that case $p_{heads}(x) \neq 1/x$, but won't be noticed. If you like to consider a more broader range of alternative hypotheses then you could use a chi-squared test by considering the number of heads and tails conditional on the value $x$. For example you could have a table describing the observations like $$\begin{array}{r|cccc} X & 2 & 3 & 4 & 5 \\ \hline tails & 14 & 10 & 18& 4 \\ heads & 15 & 4 & 4 & 2 \end{array}$$

    and compare with expectation values using a chi-squared test (or use individual binomial tests and some correction for multiple testing)

    $$\begin{array}{r|cccc} X & 2 & 3 & 4 & 5 \\ \hline tails & 14.5 & 9.33 & 16 5 & 4.8 \\ heads & 14.5 & 4.66 & 5.5 & 1.2 \end{array}$$

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you. I have digested the first part of the answer so far (about the Poisson-binomial distribution). Why do you subtract 1 from the number of observed heads, is this the degrees of freedom thing? $\endgroup$
    – Nucular
    Commented Jan 10 at 15:47
  • $\begingroup$ @Nucular for discrete variables, the case that $x \geq k$ is equivalent to the the case that we do not have $x \leq k-1$. That's what creates the 'subtract 1'. $\endgroup$
    – Sextus Empiricus
    Commented Jan 10 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.