I’ve been trying to analyze a data set comprised of a small number of human skeletons from 4 archaeological sites in the 800 B.C. to A.D. 1200 time period. Since little is known of the peoples from this era, I would like to ‘wring’ as much information as possible out of the bones. The main analyses that interest me are differences/similarities between the 4 sites, and between the sexes, for a variety of parameters (e.g., presence/absence of certain pathological conditions, dental problems, etc.). Because of the small sample sizes, regular null hypothesis statistical tests (NHST) do not have enough power to detect anthropologically meaningful differences. Thus, the ‘regular’ methods of analysis are not appropriate, so I have a question about the validity of a different approach.

Before I became aware of the importance of power in NHST, I computed almost 100 tests using the chi-square statistic (simulated 10,000 times), with the result that the distribution of the probabilities generated from these tests approximating the right half of the normal distribution. My reaction to this was that the tests represented random sampling from the chi-square (now normal because of the sample size) distribution, thereby negating any belief that there is statistical reason to believe that the people in the four sites, as well as the males and females, are different for the multiple frequencies compared. If there were a large number of significant tests, I feel that the distribution would not take this form. Is this a reasonable conclusion? Does the lack of power apply to this reasoning?

I’ve been trying to analyze a data set comprised of a small number of human skeletons from 4 archaeological sites in the 800 B.C. to A.D. 1200 time period. Since little is known of the peoples from this era, I would like to ‘wring’ as much information as possible out of the bones. The main analyses that interest me are differences/similarities between the 4 sites, and between the sexes, for a variety of parameters (e.g., presence/absence of certain pathological conditions, dental problems, etc.). Because of the small sample sizes, regular null hypothesis statistical tests (NHST) do not have enough power to detect anthropologically meaningful differences. Thus, the ‘regular’ methods of analysis are not appropriate, so I have a question about the validity of a different approach.

Before I became aware of the importance of power in NHST, I computed almost 100 tests using the chi-square statistic (simulated 10,000 times), with the result that the distribution of the probabilities generated from these tests approximates the right half of the normal distribution. My reaction to this was that the tests represented random sampling from the chi-square (now normal because of the sample size) distribution, thereby negating any belief that there is statistical reason to believe that the people in the four sites, as well as the males and females, are different for the multiple frequencies compared. If there were a large number of significant tests, I feel that the distribution would not take this form. Is this a reasonable conclusion? Does the lack of power apply to this reasoning?