# Error in the design of an experiment

Suppose that we have a population with a subpopulation that exhibits a certain characteristic. We are interested in the performance of a group of people from the population on a group test. The question is do groups of people composed of individuals entirely from the subpopulation perform better than groups of people from the population. In order to apply statistical techniques to make this determination I propose the following. First randomly select a large number of groups from the population and measure each groups performance statistics. Next randomly select the same number of groups from the subpopulation and measure each groups performance statistics. Compare the statistics(primarily moments of the empirical distributions) from the two categories to draw conclusions about performance. My question is what if any are the flaws with this experiment and how to fix them?

For example let's say we have 10,000 people in our population and we want to see how well a group of 10 people performs(in terms of time to completion) on a scavenger hunt. We want to know if groups of people composed of high IQ individuals do better. We start by taking a random sample of 1000 groups of 10 from the population and measure their performance on a scavenger hunt. Next we consider people from the group of 10,000 that have an IQ over 140. We take a sample of 1000 groups of 10 from this subpopulation and measure their performance on a scavenger hunt. We compare statistics associated to the two groups to determine if indeed groups of high IQ individuals perform better.

• Are you talking about sampling from an observed dataset - i.e. you have some sort of test scores for individual people and select N1 groups from the whole dataset and then N2 groups from a subset of the whole dataset (your subpopulation) where N1=N2. You then compute maybe the mean score for each group so you want to compare the distribution of the N1 whole population means to the N2 subpopulation means? Furthermore I imagine your whole dataset can be viewed as a sample from some larger population? – dandar Jan 11 '16 at 17:13
• @dandar Exactly right. – Wintermute Jan 11 '16 at 18:12
• It sounds as though what you are proposing to do is some sort of resampling scheme which sometimes people call a bootstrap. Can you give an example of your groups - for example do you want to sample different groups, or different "units" (people perhaps) within the same groups? If within-group observations are expected to be correlated then I think this can be exploited in particular bootstrap methods - like the cluster bootstrap. Perhaps you could give a detailed example? – dandar Jan 12 '16 at 8:57
• A) It is not clear if you are measuring something related to a single person or to a group of people: for example how fast a person solve a problem versus how fast a group if 5 people working together solve a problem. B) it is not clear why you use the term sub-population. Are there 2 populations (people with engineering degree versus people with art degree) or a small part of a singe population (people with engineering degree versus the rest of humankind)? – Jacques Wainer Jan 12 '16 at 21:14
• @JacquesWainer I have added an example. Hopefully this will make the problem clear. – Wintermute Jan 13 '16 at 15:03

First let us say tha there are two populations of people - IQ>140 and IQ<=140 (pop A and B).

The procedure you propose is correct. Only two discussion points:

1) there is no need for the same number of groups sampled from population A and B. Your experiment is not paired - in the sense that there is no correspondence between a group selected from pop A and another selected from pop B. You will be performing a non-paired statistical test to compare the set of groups.

2) The only issue that requires some further thinking is whether the same person X can be part of two different groups (either from pop A or B). That is, when you sample 10 people from population A do you have to be sure that these people have not been part of any other group tested? I dont think so. The claim being tested is that groups with members only from population A perform differently than groups with member only from population B. So to perform some statistics you need a set of groups (all of them different). But it is the group that is different among themselves. If person X belong to group A1 and A2, the groups can still be different - provided that at least one person in A1 is not in A2! So I think there is no problem if the same person gets selected to two different groups.