# Is significance of the p-value reliable with extremely small sample sizes? [duplicate]

As I'm completely new to statistics and clinical trials, I would ask you for help and clarification.

Let's assume we have 2 really small samples (each of n = 2, or so) derived from two normally distributed, infinite populations (with sigma of each being unknown). We would like to check whether the means of those populations are significantly different (alpha = 0.05). To do so, we perform a t-test for difference of the means and achieve statistical significance (p-value = 0.04).

I was told, that this approach is wrong and one cannot infer from the p-value derived from the test of such a small sample size. I do not understand why.

I know, that if the sample size is so small:

• we can not test for population normality (and t-test is based on this assumption)
• the mean of our sample is more likely to be distant from the population mean
• the variance of our sample is more likely to be underestimated

But I thought, that if:

• I already know, that my population is distributed normally
• SD of the mean depends on the sample size
• t-test takes into account the issue with the underestimated population variance (by the fact, that t-distribution depends on the sample size and the smaller the sample size, the more the variance is underestimated; am I right?)

this solves all the issues. I know that this way I'll receive really huge confidence intervals, but I'm still able to achieve significance when the differences are big enough. As I understand, wider confidence intervals are "a penalty" for inaccurate estimates of population mean and variance. But why p-value should be considered wrong as well? The distribution is normal, so I'm not violating any assumptions.

If the p-value would be non-significant (p>0.05) I definitely won't infer that the null hypothesis is true, because I know this test is likely to be underpowered. I only want to know how can I interpret the obtained significant p-value and why it's not as valuable, as a significant p-value derived from the sample of n=30?

To extend this problem, what would happen if the population distribution were not known and I used some sort of nonparametric test, instead of the aforementioned t-test?

Thanks!

• You should add to your list of assumptions that the variances are equal. On another note, if your alpha is set at .05, & p=.04, then you would reject the null, & the question of the test being "underpowered" is a non-issue. Lastly, you may find this question of interest: Is there a minimum sample size required for the t-test to be valid?. – gung - Reinstate Monica Jan 9 '13 at 1:00
• About the p-value part: yes, I'm sorry, my brain was trying to think about p being significant and non-significant at the same time, so this happened. I meant the situation, when p is not significant and test is likely to be underpowered, so I won't infer about the truthfulness of the null hypothesis. – grajlord Jan 9 '13 at 10:38
• It's quite unlikely you should be inferring the truthfulness of the null hypothesis even if your power is decent. – gung - Reinstate Monica Jan 9 '13 at 13:12
• I've seen responses that you need to include the assumption that the variances of the two samples need to be equal. I'm not sure if this is true and would like clarification. You can use a Welch-Satterthwaite t-test for a two-sample t-test assuming unequal variance. See here for basic information: en.wikipedia.org/wiki/Welch's_t_test – Armadillo Sep 3 '13 at 15:14