What distribution to use for a statistical test I have a simple question. If I have a small number of observations of a large number of variables, is it the distribution of the variables that matters for a statistical test or the distribution of observations for each variable. 
For example if I ask 6 people in two groups (3 people per group) 2,000 questions and each question can be answered with a score from 1-10 Is it the distribution of the scores from those 2,000 questions that determines what test I should use or the distributions of the 6 scores for each of the 2,000 questions. Or does it matter what I am trying to test? If I want to know which of those 2,000 questions is significantly different between groups, what distribution is more important?
 A: What is most important is the question being answered and the science leading up to the data.  
One test that requires very few assumptions about the distribution is the permutation test.  But ${6\choose3} =20$ so with the traditional $\alpha=0.05$ the only way that you will see a significant difference is if there is complete separation between the 2 groups.  To have any better chance you need to make some assumptions about the underlying distribution.  These assumptions need to come from what you know about the process/science that generates the data.
Also note that with 2,000 comparisons you would expect to see about 100 questions with a "significant" difference due only to chance if there is no real difference between the groups.  S, you really need to consider adjusting for multiple comparisons, but with only 3 observations per group you will have very little power (and even less if you adjust for multiple comparisons).
You may need to rethink your strategy and either limit the questions that you try to answer or collect more data on more subjects.
