I have a simple question. If I have a small number of observations of a large number of variables, is it the distribution of the variables that matters for a statistical test or the distribution of observations for each variable.

For example if I ask 6 people in two groups (3 people per group) 2,000 questions and each question can be answered with a score from 1-10 Is it the distribution of the scores from those 2,000 questions that determines what test I should use or the distributions of the 6 scores for each of the 2,000 questions. Or does it matter what I am trying to test? If I want to know which of those 2,000 questions is significantly different between groups, what distribution is more important?


What is most important is the question being answered and the science leading up to the data.

One test that requires very few assumptions about the distribution is the permutation test. But ${6\choose3} =20$ so with the traditional $\alpha=0.05$ the only way that you will see a significant difference is if there is complete separation between the 2 groups. To have any better chance you need to make some assumptions about the underlying distribution. These assumptions need to come from what you know about the process/science that generates the data.

Also note that with 2,000 comparisons you would expect to see about 100 questions with a "significant" difference due only to chance if there is no real difference between the groups. S, you really need to consider adjusting for multiple comparisons, but with only 3 observations per group you will have very little power (and even less if you adjust for multiple comparisons).

You may need to rethink your strategy and either limit the questions that you try to answer or collect more data on more subjects.

  • $\begingroup$ What does "complete separation" mean for answers on a scale of 1 to 10? Your analysis appears to be predicated on unstated assumptions about an unstated null hypothesis. In particular, with a permutation test applied to one question the results are conditional on the set of six answers and its power is conditional on those answers. Thus some evaluation of the distribution of answers would still seem useful. Overall it is unclear what relevance this answer might have for the original issue of formulating a distributional model to identify individual questions on which the groups differ. $\endgroup$
    – whuber
    Feb 25 '14 at 19:48
  • $\begingroup$ @whuber, by complete separation I mean that the largest value in the one group is less than the smallest in the other group. This is the only way to get a p-value=0.05 with $6\choose3$ possible permutations. This was meant as a starting point where a specific distribution would not be needed (but still assumes exchangeability and the null is that both groups come from the same population/distribution). $\endgroup$
    – Greg Snow
    Feb 25 '14 at 21:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.