As I'm completely new to statistics and clinical trials, I would ask you for help and clarification.
Let's assume we have 2 really small samples (each of n = 2, or so) derived from two normally distributed, infinite populations (with sigma of each being unknown). We would like to check whether the means of those populations are significantly different (alpha = 0.05). To do so, we perform a t-test for difference of the means and achieve statistical significance (p-value = 0.04).
I was told, that this approach is wrong and one cannot infer from the p-value derived from the test of such a small sample size. I do not understand why.
I know, that if the sample size is so small:
- we can not test for population normality (and t-test is based on this assumption)
- the mean of our sample is more likely to be distant from the population mean
- the variance of our sample is more likely to be underestimated
But I thought, that if:
- I already know, that my population is distributed normally
- SD of the mean depends on the sample size
- t-test takes into account the issue with the underestimated population variance (by the fact, that t-distribution depends on the sample size and the smaller the sample size, the more the variance is underestimated; am I right?)
this solves all the issues. I know that this way I'll receive really huge confidence intervals, but I'm still able to achieve significance when the differences are big enough. As I understand, wider confidence intervals are "a penalty" for inaccurate estimates of population mean and variance. But why p-value should be considered wrong as well? The distribution is normal, so I'm not violating any assumptions.
If the p-value would be non-significant (p>0.05) I definitely won't infer that the null hypothesis is true, because I know this test is likely to be underpowered. I only want to know how can I interpret the obtained significant p-value and why it's not as valuable, as a significant p-value derived from the sample of n=30?
To extend this problem, what would happen if the population distribution were not known and I used some sort of nonparametric test, instead of the aforementioned t-test?