I would like to know if a small test group (~5 samples) is significantly different from baseline (300-400 samples). I initially considered Student's T-Test, but it seems that I wouldn't be able to reliably determine normality with the small test group (and the baseline group seems not to be normally distributed). Is the Mann-Whitney U Test a suitable choice, or should I consider something else? I know these tests don't require equal sample sizes, but is there a point when a large difference in sample sizes becomes a concern, particularly when the smaller group is less than 5 or 10?
2
-
$\begingroup$ gung's response is very helpful in terms of hypothesis testing; I would be interested to hear opinions on whether hypothesis testing is appropriate at all in such a situation and what possible alternatives would be. $\endgroup$– kdkCommented Apr 21, 2017 at 21:01
-
2$\begingroup$ For a different approach you might look at Vargha and Delaney A. It expresses the probability that an observation in one group is greater than an observation in the other group. This might be a very valuable piece of information. It is linearly related to Cliff's delta. $\endgroup$– Sal MangiaficoCommented May 24, 2018 at 21:14
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
The t-test does not require equal sample sizes (cf., How should one interpret the comparison of means from different sample sizes?), although violations of the assumptions could have more impact as this becomes more extreme. In general, it isn't considered a good idea to test for normality and then select a test based on the results (cf., How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples). You could you the Mann-Whitney U-test, but it's worth noting that it actually tests a slightly different hypothesis (cf., Why would parametric statistics ever be preferred over nonparametric?). In your case, I would probably bootstrap. For more information, see this CV thread: How to perform a bootstrap test to compare the means of two samples? (Edit: I thought that the bootstrap would be OK because of the large sample size in the baseline group, but this is apparently not true; thus, bootstrapping is not a good choice here.)
I suppose the Mann-Whitney U-test would be your best choice.
answered Apr 17, 2017 at 17:31
-
1$\begingroup$ Thanks for your response and the helpful links. I've heard bootstrapping suggested for small samples, but I've also read that it doesn't actually solve the issue. I agree that Mann-Whitney U tests a slightly different hypothesis, but I don't think it's a less valid one. However, does the small sample make any hypothesis testing a poor choice? Would it ever be appropriate to construct a prediction interval for my large baseline and simply check to see if my few test samples fall within that? $\endgroup$– kdkCommented Apr 20, 2017 at 17:30
-
1$\begingroup$ Post indicating issues with using bootstrapping to combat the small sample problem: stats.stackexchange.com/questions/112147/… $\endgroup$– kdkCommented Apr 20, 2017 at 17:31
-
$\begingroup$ @kdk, you're right about that. I was thinking that the large size of your baseline group would save you, but I tried a quick simulation (akin to the 1 in the thread you link) & it doesn't. I'll edit this answer. $\endgroup$ Commented Apr 20, 2017 at 23:20