# What distribution to use for a statistical test

I have a simple question. If I have a small number of observations of a large number of variables, is it the distribution of the variables that matters for a statistical test or the distribution of observations for each variable.

For example if I ask 6 people in two groups (3 people per group) 2,000 questions and each question can be answered with a score from 1-10 Is it the distribution of the scores from those 2,000 questions that determines what test I should use or the distributions of the 6 scores for each of the 2,000 questions. Or does it matter what I am trying to test? If I want to know which of those 2,000 questions is significantly different between groups, what distribution is more important?

One test that requires very few assumptions about the distribution is the permutation test. But ${6\choose3} =20$ so with the traditional $\alpha=0.05$ the only way that you will see a significant difference is if there is complete separation between the 2 groups. To have any better chance you need to make some assumptions about the underlying distribution. These assumptions need to come from what you know about the process/science that generates the data.
• @whuber, by complete separation I mean that the largest value in the one group is less than the smallest in the other group. This is the only way to get a p-value=0.05 with $6\choose3$ possible permutations. This was meant as a starting point where a specific distribution would not be needed (but still assumes exchangeability and the null is that both groups come from the same population/distribution). Feb 25 '14 at 21:15