# Error in the design of an experiment

Suppose that we have a population with a subpopulation that exhibits a certain characteristic. We are interested in the performance of a group of people from the population on a group test. The question is do groups of people composed of individuals entirely from the subpopulation perform better than groups of people from the population. In order to apply statistical techniques to make this determination I propose the following. First randomly select a large number of groups from the population and measure each groups performance statistics. Next randomly select the same number of groups from the subpopulation and measure each groups performance statistics. Compare the statistics(primarily moments of the empirical distributions) from the two categories to draw conclusions about performance. My question is what if any are the flaws with this experiment and how to fix them?

For example let's say we have 10,000 people in our population and we want to see how well a group of 10 people performs(in terms of time to completion) on a scavenger hunt. We want to know if groups of people composed of high IQ individuals do better. We start by taking a random sample of 1000 groups of 10 from the population and measure their performance on a scavenger hunt. Next we consider people from the group of 10,000 that have an IQ over 140. We take a sample of 1000 groups of 10 from this subpopulation and measure their performance on a scavenger hunt. We compare statistics associated to the two groups to determine if indeed groups of high IQ individuals perform better.

• Are you talking about sampling from an observed dataset - i.e. you have some sort of test scores for individual people and select N1 groups from the whole dataset and then N2 groups from a subset of the whole dataset (your subpopulation) where N1=N2. You then compute maybe the mean score for each group so you want to compare the distribution of the N1 whole population means to the N2 subpopulation means? Furthermore I imagine your whole dataset can be viewed as a sample from some larger population? Commented Jan 11, 2016 at 17:13
• @dandar Exactly right. Commented Jan 11, 2016 at 18:12
• It sounds as though what you are proposing to do is some sort of resampling scheme which sometimes people call a bootstrap. Can you give an example of your groups - for example do you want to sample different groups, or different "units" (people perhaps) within the same groups? If within-group observations are expected to be correlated then I think this can be exploited in particular bootstrap methods - like the cluster bootstrap. Perhaps you could give a detailed example? Commented Jan 12, 2016 at 8:57
• A) It is not clear if you are measuring something related to a single person or to a group of people: for example how fast a person solve a problem versus how fast a group if 5 people working together solve a problem. B) it is not clear why you use the term sub-population. Are there 2 populations (people with engineering degree versus people with art degree) or a small part of a singe population (people with engineering degree versus the rest of humankind)? Commented Jan 12, 2016 at 21:14
• @JacquesWainer I have added an example. Hopefully this will make the problem clear. Commented Jan 13, 2016 at 15:03