# Bootstrapping and comparing multiple proportions

EDIT 1: Thanks to @gung for pointing out that if the bag initially had 324 sweets, then as Mother hands them out, there are consecutively fewer amounts she can hand out. To simplify things (because I'm interested in understanding the basics instead of something too advanced), he offered a rewording.

EDIT 2: This is not homework. I'm just trying to learn some statistics in a fun way

COMMENT: From all of the kind comments/answers thus far, it occurs to me that I really need to work on defining my questions in the future much better because I didn't realise how much the answer could vary depending on how it's posed (very useful lesson!). I initially thought this was well posed but it's good to learn these things and hopefully improve, thank you.

Let's say a a mother has a bag containing an infinite number of sweets.

We keep a track of how many sweets she gives to each of her children named "A", "B", ..., "G" from this bag, and how many she gives out to an unknown number of adults. When the mother comes up to each person, she opens a new, identical bag of sweets, reaches in & grabs some for that individual. At the end of the day, she has given out a total of 324 sweets!

> (DF <- data.frame(A=15, B=4, C=1, D=4, E=44, F=4, G=1, Adults=251, Total=324))

A B C D  E F G Adults Total
15 4 1 4 44 4 1    251   324


Question 1: Her children were given a total of 73 sweets out of 324. The adults were give 251 sweets out of 324. They children want to know if the adults were given a statistically significant proportion of the sweets even though their mother claims that everyone had an equal chance of getting the same number of sweets and it was just luck that some got more than others. So basically we want to compare the entire group of children against the entire group of adults, not individuals.

Question 2: Did one or more of the children get a statistically significant number of more sweets when compared to their siblings (i.e. we ignore the Adults for this question and only consider the children "A", ..., "G")? And if so, which one(s) got a statistically significant proportion? Mother again claims everyone had an equal chance of getting the same number of sweets, and it was just luck that some got more than others.

Additional Information: I want to use a bootstrapping approach because I seem to get a better feel for probability when using simulation (NB: I am not a statistician, just doing this for fun). Below I will give my approach to the first question which I think is OK but would appreciate some advice on if I'm doing it correctly. I have no idea how to approach the second question because there are multiple proportions involved.

My approach to Question One:

Hypotheses: My null hypothesis is that children and adults had an equal chance of getting the same number of sweets. My alternative hypothesis is that Mother is more generous with giving out sweets to adults than to her children.

Research Question: How often might we observe a proportion of sweets as small as the children got, if the mother really was handing out sweets fairly?

Methodology: We can test this directly by creating a hypothetical population that embodies the null hypothesis, with 50% "children" and 50% "adults" and repeatedly drawing random samples of 324 from it, with replacement, recording the results. We then look at how many of these samples have 73 or fewer "children" nodes in them (this is our p-value).

replicates <- 999
size <- 324
runs <- list(replicates)
reference <- 73/size

for(i in 1:replicates){
runs[[i]] <- table(sample(x=c("children", "adult"), size=size, replace=TRUE, prob=c(0.5, 0.5)))
}

bootstraps <- as.data.frame(do.call(rbind, runs))
# 1   166      158
# 2   171      153
# 3   158      166
# 4   151      173
# 5   156      168
# etc.

sum(bootstraps$children <= reference) / replicates #[1] 0  Conclusion: My p-value is effectively zero, which means there is strong evidence that Mother has been handing out sweets unfairly because getting 73 or fewer sweets is very unlikely to have happened by luck alone. As for question 2, I'm drawing a complete blank :( • Note that, as the mother moves through the room, the number of sweets in the bag goes down. Thus, each number is a proportion of a different total. This makes your situation different from the prototypical multinomial logistic regression situation. Jul 18, 2012 at 13:14 • Ahh, good point! OK, let me try and rewrite this so that isn't an issue (I don't want to make it too complicated so that I can follow up on any answers!) Jul 18, 2012 at 13:19 • What makes you think that you can assume the number of adults? It seems that you can at best assume that under your null the distribution of candies to adults is the same as for the kids. This leads to a distribution of possible numbers of adults given the total. Furthermore, it's not really necessary to know anything about OLS or other distributions for the bootstrap solution, but it might help in guesstimating the outcome. Finally, if this is homework it should be marked as such. – John Jul 18, 2012 at 13:50 • OK, then a question about the wording of #1. Generosity implies generosity towards an individual, but perhaps you meant group. Do you mean that there was more generosity toward the group of adults compared to the group of children or toward individual adults compared to individual children? In the former case, even if there was 1000 adults she's more generous toward adults. – John Jul 18, 2012 at 15:42 • Clair, I want to acknowledge that it's obvious you did put a lot of thought & effort into formulating the question as best as possible. If only every CV user did the same! However, statistical questions can hinge on details that are not obvious unless you are very thoroughly versed in the underlying ideas. When you're trying to learn these things, you are not thoroughly versed in this manner, by definition. The resulting conversation & proffered answers are intended to help bring these obscure underlying details out, not as criticism. Jul 18, 2012 at 20:03 ## 3 Answers The questions, especially the second one, are meaningless as they stand right now. The problem is that the concept of "taking a random number of candies from the bag" is not defined. Even with a finite number of candies in the bag there could be multiple definitions. For example, the following two both sound reasonable, but give different results: 1. If there are$n$candies in the bag, the probabilities of taking exactly$0, 1, \ldots , n$candies are all the same:$1/(n+1)$. 2. We go through each candy, and decide whether to take it out with probability$p$. This means that the probability of exactly$x$candies is${n\choose x} p^x (1-p)^{n-x}$. Once you go to an infinite bag, neither of those options apply. So all we can say that there is some unknown distribution that gives the probability of$x$candies. Using the bootstrap idea you would estimate this as the observed distribution:$P(1 \text{ candy})=2/7$,$P(4 \text{ candies})=3/7$,$P(15 \text{ candies})=1/7$, and$P(44 \text{ candies})=1/7\$. Note that this implies that the question of comparing the number of candies received by different children is almost meaningless. You could potentially calculate the probability of receiving the actual number or fewer candies for each child, and some would be less lucky than others, but somebody would have to be less lucky by definition.

As for the first question, you would need to bootstrap from the observed distribution after making some assumption about the adults, or the process that sends children/adults to Mom. I can think of several options, but nothing is totally satisfactory, as you would want to keep the number of children fixed at 7 and the total number of candies fixed at 324 while keeping the observed distribution of candies per handful and varying the number of adults appropriately. Perhaps letting go of some of these conditions (eg total number of candies) is reasonable.

• @John, that's where I am going as well. One would perhaps simulate from the observed distribution, but keep going only until the candy total hits 324. Any simulations where this number was not reached exactly should be discarded. Then assign the first 7 numbers to children, the rest to adults. Now we can look at arbitrary "fairness" statistics. Jul 18, 2012 at 16:18

The way you've constructed your problem you can't really get the answers you want, even with bootstrapping. I'll give you responses from the bootstrap perspective.

#1 All you can conclude is that she was equally generous. It's because you only have one sample of the random process, the one for kids. Extrapolating that to the adults means it's still the same distribution and any proper bootstrap of the problem from the kids data has to show the adults are the same. (if you meant generosity to be each individual adult and each individual child, which is what makes sense)

#2 This isn't answerable because you don't have any other knowledge than the distribution you have. Therefore, that is your definition of random and you can't call any of the samples unfairly higher than any other. Perhaps if you had more than one sample to each child you could, but you don't.

The bootstrap is not needed here because the null distribution is well defined. Bootstrap should not be used just because you are comfortable with simulation. The bootstrap works in this situation and will give essentally the same results as the straightforward method. So your motivation is not a factor here. Sounds like you handled the first question properly.

For question 2 do a pairwise comparison of the proportions (two sample binomial or corresponding bootstrap). Adjust the p-values for multiplicity. The children with signifiicant p-values in the adjusted comparison can be considered to be treated differently with respect to the sweets received.

This answer was to the initial question before it was changed. Random selection of a number to give a child muddies the waters (adds to the uncertainty). Statistically significant differences between children can be due to preferentially handing to some children more than others (the effect of preference which you would be interested in) or just because by chance certain children happened to get the luck of the draw and get more sweets when selected (a random event not of interest). As Aniko points out the problem is only well defined if you specify a probability distribution for the number drawn.

• How is the null well defined when the number of adults is unknown?
– John
Jul 18, 2012 at 14:00
• The first question is not handled properly because it completely ignores the fact that the candies are handed out in batches, not one at a time. Jul 18, 2012 at 14:23
• @John The problem was edited and changed after I gave my answer. But the number of adults doesn't enter the problem because it is treated as one group. The number of children is important and is specified. Jul 18, 2012 at 14:49
• @Aniko I don't think the OP said that they were handed out in batches. The OP is assuming one at a time. Jul 18, 2012 at 14:53
• @Aniko Sorry I read it over and you are right. I don't think that was the way it was when I initially answered. Now the distribution of the number drawn comes into play. It didn't in my answer. Jul 18, 2012 at 15:00