9
$\begingroup$

For example, say we have three coins. Due to several tests, we have 95% confidence intervals for X (the heads percentage) of each coin. Coin A is 0-5%, coin B is 1-9%, and coin C is 40-70%.

[For the curious, these CI's were created using inverse binomial CDF, based a small sample of ~30 flips of the coin. Not surprisingly, we have no clue about the composition of the coin-bag these coins were drawn from.]

Can we say anything about the number of heads we would expect to see if we flipped all three? Or if we chose one coin randomly and flipped it? Based on the computed CI's alone? What if we can also use the info that led us to those CI's?

Edit: let me tell you much more about my problem.

We have 50 coins, all of which we suspect are very low probability heads (well under 5%). We do 30 trials, where each trial consists of flipping all the coins and recording all the results. The value we seek is a CI on the average number of heads in a trial.

One approach would be to record the 30 "numbers of heads", and do a standard normal approximation. The problem is that many of these 30 numbers are 0 and 1, so while we get the correct mean, I fear the resulting CI is way tighter than it has any right to be.

What I would like to do is calculate a CI for each coin, using it's 30 binary outcomes. I have calculated that for 0 heads out of 30, this leads to a 95% CI of 0-11.5% (with a mean of 0, of course). I did this by substituting values for P in "cumulative distribution function(1) of binomial distribution n=30 p=P" in Wolfram Alpha until the result was around 0.025. For 1 heads out of 30, I get a CI of 0-17.2%

Once I have done this, can I add up the CI's in any way?

$\endgroup$

2 Answers 2

3
$\begingroup$

There are multiple methods to calculate the binomial confidence intervals, and in case of your small (~30) sample size this may mean notable differences.

Can we say anything about the number of heads we would expect to see if we flipped all three? Assuming that the normal approximation was used and was appropriate to get the three confidence intervals we can go on with the normal approximation: expected number of heads $E=\frac{0+5}{2} + \frac{1+9}{2} + \frac{40+70}{2}$, $CI = E ± ½ \sqrt{(5-0)^2 + (9-1)^2 + (70-40)^2}$, see sum of normally distributed random variables

Or if we chose one coin randomly and flipped it? If randomly meant choosing one of these 3 coins with equal probabilities than we expect ⅓ times the expected value and a CI of $\frac{1}{\sqrt{3}}$ times the width of the result of the previous question.

What if we can also use the info that led us to those CI's? I that info could mean all or some of your assumptions before setting up the experiment, the experimental conditions and the observations. In that case you could formulate the full conditional probability, tell a bit more about the coins, tackle the small sample size better, particularly giving more accurate CIs for the previous questions.

$\endgroup$
2
  • $\begingroup$ Thanks for the input! Unfortunately my data cannot fairly be approximated as normal. I may have to post another question to discuss my shady binomial confidence intervals, but for now, I just want to combine oracle-given CI's. $\endgroup$
    – Brad
    Apr 16, 2011 at 5:42
  • $\begingroup$ normal approx would be particularly bad for the 0-5% coin $\endgroup$ Apr 16, 2011 at 6:38
1
$\begingroup$

This is a problem which can be relatively easily solved using a Bayesian approach. This is one of the "re-using" the original data answers, rather than one which uses the CIs.

Each coin has some long run frequency of heads $\theta_i$ $(i=1,\dots,50)$. Now you have some partial information about these frequencies, that you suspect them to be below 5%. Not exactly clear what is meant by this: do you want to test this? or do you have information which says that it should be less than 5%? And do you think its 5% because of the data, or did you think this before seeing the data? Given that you have a reasonably large amount of data, this kind of information probably won't make much difference - so I will simply ignore it.

Now there has been no logical connection stated between each coin - so I will not pre-suppose that one exists. This means in words that knowing the results of one coin doesn't tell you anything about any other coin.

So we start by supposing that the only thing that was known about each coin is that it is possible for each to give heads, and possible for it to give tails (i.e. we assume that if we flipped the coin until we saw both a head and a tail we would not sample forever). This results in a uniform prior for each:

$$p(\theta_{i}|I)=1$$

(where $I$ simply denotes the "prior information" or assumptions contained in the problem). Now you will throw each coin $n_{i}=30$ times, and conditional on $\theta_{i}$ the number of heads observed $x_{i}$ will have a binomial sampling distribution

$$(x_{i}|n_{i},\theta_{i},I)\sim Binomial(n_{i},\theta_{i})$$

The posterior distribution for each $\theta_{i}$ will have a beta distribution:

$$(\theta_{i}|x_{i},n_{i},I)\sim Beta(x_{i}+1,n_{i}-x_{i}+1)\implies p(\theta_{i}|x_{i},n_{i},I)= \frac{\theta_{i}^{x_{i}}(1-\theta_{i})^{n_{i}-x_{i}}}{B(x_{i}+1,n_{i}-x_{i}+1)}$$

Where $B(a,b)$ is the beta function. Note that each posterior has mode (most likely value) equal to the observed proportion $\frac{x_{i}}{n_{i}}$. Now this is the posterior for each of the $50$ coins is always proper, well behaved, and exact even when the observed fraction is zero (no approximations have been made). Denote the number of future trials as $m_{i}$ and the unknown number of heads in these trials by $y_{i}$

So if you knew which coin you picked, then you have a posterior predictive for the next result:

$$p(y_{i}|m_{i},x_{i},n_{i},\text{ith coin},I)=\int_{0}^{1}p(y_{i}|m_{i},\theta_{i},I)p(\theta_{i}|x_{i},n_{i},I)d\theta_{i}$$

$$={m_{i} \choose y_{i}}\int_{0}^{1}\theta_{i}^{y_{i}}(1-\theta_{i})^{m_{i}-y_{i}}\frac{\theta_{i}^{x_{i}}(1-\theta_{i})^{n_{i}-x_{i}}}{B(x_{i}+1,n_{i}-x_{i}+1)}d\theta_{i}$$

$$={m_{i} \choose y_{i}}\frac{B(x_{i}+y_{i}+1,m_{i}+n_{i}-x_{i}-y_{i}+1)}{B(x_{i}+1,n_{i}-x_{i}+1)}=\frac{{x_{i}+y_{i} \choose y_{i}}{m_{i}+n_{i}-x_{i}-y_{i} \choose m_{i}-y_{i}}}{{m_{i}+n_{i}+1 \choose m_{i}}}$$

Which looks like a hypergeometric like distribution, but not quite, as the "random variable" $y_{i}$ appears in different places to what you would see in a hypergeometric. This is called a beta-binomial compound distribution. Note that it is "parameter free" - it only depends on what is unknown but of interest $y_{i}$ and what is known $m_{i},x_{i},n_{i}$ - no "pluggin in" of any estimate is required - hence this is an exact inference.

What if we chose one coin randomly and flipped it $m$ times?

But if you now don't know which coin is going to be picked, then the answer cannot depend on $i$. To do this we simply average out, or marginalise out the $i$. Assuming that you have no reason to suspect any one coin will be preferred in the choice (what is really meant by "randomly" I think), then each is equally likely, and we get:

$$p(y|m,D,I)=\frac{1}{50}\sum_{i=1}^{50}\frac{{x_{i}+y \choose y}{m+n_{i}-x_{i}-y \choose m-y}}{{m+n_{i}+1 \choose m}}$$

Where $D\equiv x_{1},\dots,x_{50},n_{1},\dots,n_{50}$ ("the data") and the $i$ subscript has been dropped from $m,y$ to indicate that the result doesn't depend on $i$ (i.e. doesn't depend on which coin is flipped). Now this is an exact answer to your problem. You can calculate a 95% interval for $y$ using this probability distribution, setting $m=30$ (since this is the sample size you used). This should give you what you are after - in that the frequentist CI is usually phrased in a predictive fashion "if we sampled again what would we likely the true value be covered by the CI 95% of the time" - a CI talks about the future.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.