Distribution of probabilities from multiple statistical tests I’ve been trying to analyze a data set comprised of a small number of human skeletons from 4 archaeological sites in the 800 B.C. to A.D. 1200 time period. Since little is known of the peoples from this era, I would like to ‘wring’ as much information as possible out of the bones. The main analyses that interest me are differences/similarities between the 4 sites, and between the sexes, for a variety of parameters (e.g., presence/absence of certain pathological conditions, dental problems, etc.). Because of the small sample sizes, regular null hypothesis statistical tests (NHST) do not have enough power to detect anthropologically meaningful differences. Thus, the ‘regular’ methods of analysis are not appropriate, so I have a question about the validity of a different approach.
Before I became aware of the importance of power in NHST, I computed almost 100 tests using the chi-square statistic (simulated 10,000 times), with the result that the distribution of the probabilities generated from these tests approximates the right half of the normal distribution. My reaction to this was that the tests represented random sampling from the chi-square (now normal because of the sample size) distribution, thereby negating any belief that there is statistical reason to believe that the people in the four sites, as well as the males and females, are different for the multiple frequencies compared. If there were a large number of significant tests, I feel that the distribution would not take this form. Is this a reasonable conclusion? Does the lack of power apply to this reasoning?
 A: I would use descriptive statistics and exploratory data graphs and abstain from comments about what is "statistically meaningful". You didn't tell us about the sample sizes, but chances are that they are so low that with very many variables to look at you can't nail down anything as clearly significant in an appropriate manner. But then do you have to?
Regarding your question, I don't know how exactly the 100 tests were done and surely not what exactly you simulated 10000 times, however regardless of the overall distribution, chances are that a single extreme p-value of $10^{-24}$ or something (if occurring; or more precisely something that is indeed significant with proper Bonferroni correction over the full number of tests) would be meaningful, and due to the lack of power non-significance as always doesn't mean that the null hypothesis is true, it actually pretty much doesn't mean anything with this number of tests. Not sure where you got the idea from the approximating a half normal means something, I don't think so. You just can't tell much with any statistical "guarantee" in that setting.
