8
$\begingroup$

As I'm completely new to statistics and clinical trials, I would ask you for help and clarification.

Let's assume we have 2 really small samples (each of n = 2, or so) derived from two normally distributed, infinite populations (with sigma of each being unknown). We would like to check whether the means of those populations are significantly different (alpha = 0.05). To do so, we perform a t-test for difference of the means and achieve statistical significance (p-value = 0.04).

I was told, that this approach is wrong and one cannot infer from the p-value derived from the test of such a small sample size. I do not understand why.

I know, that if the sample size is so small:

  • we can not test for population normality (and t-test is based on this assumption)
  • the mean of our sample is more likely to be distant from the population mean
  • the variance of our sample is more likely to be underestimated

But I thought, that if:

  • I already know, that my population is distributed normally
  • SD of the mean depends on the sample size
  • t-test takes into account the issue with the underestimated population variance (by the fact, that t-distribution depends on the sample size and the smaller the sample size, the more the variance is underestimated; am I right?)

this solves all the issues. I know that this way I'll receive really huge confidence intervals, but I'm still able to achieve significance when the differences are big enough. As I understand, wider confidence intervals are "a penalty" for inaccurate estimates of population mean and variance. But why p-value should be considered wrong as well? The distribution is normal, so I'm not violating any assumptions.

If the p-value would be non-significant (p>0.05) I definitely won't infer that the null hypothesis is true, because I know this test is likely to be underpowered. I only want to know how can I interpret the obtained significant p-value and why it's not as valuable, as a significant p-value derived from the sample of n=30?

To extend this problem, what would happen if the population distribution were not known and I used some sort of nonparametric test, instead of the aforementioned t-test?

Thanks!

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ You should add to your list of assumptions that the variances are equal. On another note, if your alpha is set at .05, & p=.04, then you would reject the null, & the question of the test being "underpowered" is a non-issue. Lastly, you may find this question of interest: Is there a minimum sample size required for the t-test to be valid?. $\endgroup$
    – gung - Reinstate Monica
    Commented Jan 9, 2013 at 1:00
  • $\begingroup$ About the p-value part: yes, I'm sorry, my brain was trying to think about p being significant and non-significant at the same time, so this happened. I meant the situation, when p is not significant and test is likely to be underpowered, so I won't infer about the truthfulness of the null hypothesis. $\endgroup$
    – grajlord
    Commented Jan 9, 2013 at 10:38
  • 1
    $\begingroup$ It's quite unlikely you should be inferring the truthfulness of the null hypothesis even if your power is decent. $\endgroup$
    – gung - Reinstate Monica
    Commented Jan 9, 2013 at 13:12
  • $\begingroup$ I've seen responses that you need to include the assumption that the variances of the two samples need to be equal. I'm not sure if this is true and would like clarification. You can use a Welch-Satterthwaite t-test for a two-sample t-test assuming unequal variance. See here for basic information: en.wikipedia.org/wiki/Welch's_t_test $\endgroup$
    – Armadillo
    Commented Sep 3, 2013 at 15:14

1 Answer 1

Reset to default
4
$\begingroup$

If you know that they are distributed normally you are right (with the addition, as @gung points out, that the variances need to be qual). The use of the t statistic with the appropriate degrees of freedom means that this approach is alright despite the tiny sample size.

On your secondary question about what to do if the population distribution is not known, it would depend on which nonparametric test you used.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So am I right by saying, that the "only" direct issue we're left when using small sample sizes is that the tests we perform are very vulnerable to the assumptions we make and assuming wrong can influence the p-value greatly? And although this can work pretty well in theory, in practice there are seldom situations we are able to assume correctly all the conditions needed to perform the correct tests? $\endgroup$
    – grajlord
    Commented Jan 9, 2013 at 14:54
  • 1
    $\begingroup$ @grajlord in this particular case, yes, but you can't generalise beyond this to "tests" in general. A lot of statistical tests depend on asymptotic properties and will be highly misleading with small samples. t-tests happen to have been developed specifically for small samples. $\endgroup$
    – Peter Ellis
    Commented Jan 9, 2013 at 18:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.