# Statistical tests for uneven groups

We performed an experiment (usability testing), which had two independent samples, one group with 5 participants and the other with 24. Based on the analysis of the distribution of the data (Shapiro-Wilk test), we concluded that we operate with data where some measurements distributed normally and others do not confirm to normal distribution.

I was wondering which test would be the most appropriate for data that is normally distributed and which for the data that is not normally distributed in our case, since we compare a group with 5 participants with a group that has 24?

We found T-Test (for normally distributed data) and Mann Whitney test (for data that is not normally distributed), however, we have two groups that are uneven and based on the conflicting information we are not sure if we can perform these tests.

• You can start with this answer: stats.stackexchange.com/questions/121852/… as it discusses sample-size issues and robustness of t-tests. – Tim Mar 11 '16 at 12:19
• The issue is not the fact that the sample sizes are uneven, but that the sample sizes are small. See the link in Tim's comment for a good discussion of the small sample size testing situation. – user44764 Mar 12 '16 at 4:19
• The conclusion that one of your samples is actually drawn from a normal distribution is not a correct conclusion from failure to reject the null in a Shapiro-Willk. Failure to detect non-normality doesn't tell you it's normal, it tells you you failed to detect the non-normality you have. What your testing shows is that at least some of your data aren't drawn from a normal distribution (or that you made a type I error, perhaps). – Glen_b -Reinstate Monica Mar 15 '16 at 1:57
• stats.stackexchange.com/questions/47498/… – Shijia Bian Mar 18 '16 at 7:01
• The link above might be helpful – Shijia Bian Mar 18 '16 at 7:01

A simple yet robust approach could be to compute the difference between medians and then compute 95% confidence interval of such statistic with bootstrap (percentile). Such confidence interval would be adequate for inference.

You can find here some useful references:

http://r.789695.n4.nabble.com/CI-for-the-median-difference-td4399508.html

http://r.789695.n4.nabble.com/CI-for-the-median-difference-td4399508.html

http://www.statalist.org/forums/forum/general-stata-discussion/general/564770-hypothesis-testing-for-bootstrapped-differences-in-medians-in-a-randomized-clinical-trial

Yet, the small samples (5 cases in one group), limits substantially external validity, even if your inferential estimates were quite precise (anyway very unlikely).

normal data, uneven sample sizes: Welche's t-test expects normal distribution, but allows for uneven samples sizes (and unequal variance between groups). For your samples with normal distribution, but uneven sample sizes, this test will likely give the most power. https://en.wikipedia.org/wiki/Welch%27s_t_test

non-normal data, uneven sample sizes: Welche's t-test is also suitable to use with ranked data, which converts the test to a non-parametric test. If you are heavily concerned about uneven sample sizes, this may be a good candidate for your samples with non-normal distribution. Here, you rank order transform the data before running Welche's t-test. That being said, the Mann Whitney test (ranksum) you've been using already, is non-parametric, and should not be biased by uneven sample sizes. And so you will likely find that Welche's ranked t-test closely approximates the results given by a ranksum.

However (as others have mentioned), you may have a larger issue to solve, in that your one sample size is just 5 participants. A group this small will require a substantial effect size (mean difference) in-order to be found to be significant. Before proceeding further, it may be worth conducting a power analysis, in order to determine how many participants would be required to detect your hypothesized effect.

I suggest Approximate Randomization (AR). It applies to test statistics following any distribution and I read somewhere that it is evaluated to be either equally accurate or more accurate than other methods that assume certain distributions even when their assumptions are met. Another nice thing about AR is that it immediately spits out the $p$-value (no need to lookup distribution tables as in $t$-test).

• Approximate Randomization (I have never seen this name for it) are known as permutation tests en.wikipedia.org/wiki/… – Jacques Wainer Mar 13 '16 at 4:43
• @JacquesWainer close (but not exactly true). Approximate randomization tests can be called "approximate permutation tests", which is different than "permutation tests". It's correct alternative names are mentioned in the same wiki page that you have linked, but in a different section: en.wikipedia.org/wiki/… -- the problem with permutation tests is that they can be very expensive, thus we can use approximate permutation tests (I call approximate randomization) which are asymptotically equal to it, but computationally more doable. – caveman Mar 13 '16 at 4:53
• I doubt there would be any computational problems with fewer than 30 data points. – dsaxton Mar 15 '16 at 22:33
• @dsaxton, can you say why please? I forgot the reasonable limit of data points before the Fisher exact test becomes too demanding. I think it's combinatoric, but I forgot exactly the factors that play in the speed of the combinatoric growth of the test. – caveman Mar 17 '16 at 3:00