# What test should I use for two groups of different number of measurements and low number of samples?

I have data that include 20 samples divided into 2 groups (category A and category B). The groups are independent, none of the value in one group repeat in other. N(A) =14, N(B) = 6.

here is the data:

category A                  category B
0.0119888167559  0.023185483871
0.00101354303189     0.312090168227
8.95231103909e-06    0.503371693147
2.9580256165e-05    0.522824974411
0.0596266691309  0.114932864532
4.02612958098e-05   3.32126606662e-05
0.337753287524
0.0115114590662
0.19273480545
0.232453117898
3.69713102632e-05
3.00480769231e-05
0.192851577717
1.58790650407e-05


I would like to show that mean values of 2 groups differ significantly. But I am very confused which test statistic I should use.

Here are the tests that I've performed so far:

1. Wilcoxon rank sum test (Mann-Whitney test) (two-tailed) W=20, p=0.07575

2. Student t-test (two-tailed) t = -2,24259 p = 0,03775

3. Welch t-test (two-tailed, unpaired, correction=False) t=-1.7109, p = 0.1376

So as you see 3 tests present 3 different probabilities...to be more complicated ...

1. Normality test (Shapiro-Wilk) I've checked also the normality of my data, and the first group category A is normally distributed (Test Shapiro-Wilka = 0,704713, p 0,000413591, p<0.05) but second is not-category B (Test Shapiro-Wilka = 0,868539, p 0,220442, p>0.05) probably because of low number of samples.

A list of questions:

Q1: Can I assume that my data in 2 groups are normally distributed and use Student t test or Welch t-test? Q2: OR Should I use non-parametric Mann Whitney test? (I've written that it has low power for low number of samples...) Q3: Another think is the equality of variation between groups, when I assume that there are equal I can use Student t- test, if not I can use Welch t-test...should I first perform test for variant equality?

To summarize post - I need help to find a test that will be OK to publicate results in publication for: - small number of samples in one group (less than 10) - unequal number of samples in groups - data not normally distributed in one group - showing the difference of means (optional)

• You might wish to read up on two basic ways: the Student's t-test or the Wilcoxon rank sum test. (You might want to add the self-study tag if this question relates to a course of study or similar.) Good luck to you. – user140401 Dec 10 '16 at 10:51
• Wilcoxon rank sum test is for equal number of samples in groups so it won't work in this situation. Student test is parametric test and I don't think that my data is normally distributed because of small number of sample size ... any other ideas? Thanks. – Agata Dec 10 '16 at 11:19
• I do. The sample sizes are small enough for you to edit the question and post the complete data set here, together with what you've tried. This might attract others to respond. – user140401 Dec 10 '16 at 11:21
• Mayby a Mann-Whitney U Test? – Agata Dec 10 '16 at 11:23
• Mann-Whitney U test = Wilcoxon rank sum test. Neither requires equal sample sizes, as you suggest three comments ago. However, the Wilcoxon signed ranks test does. – user140401 Dec 10 '16 at 11:25

Your interpretation focuses mainly on the p-values (in particular, whether they fall above or below a critical threshold for "significance")*, so the most important thing is to make sure the p-value for your test is reflecting what you think it is. In other words, you want the p-value to tell you the probability of seeing a difference between groups that is this dramatic, assuming that in fact there was no real difference between groups (the null hypothesis). The calculation of a p-value from data via a test relies on assumptions, so if those assumptions aren't met the p-value might be bogus. Here's where you are at with the tests you suggest:

1. Wilcoxon rank sum test (a.k.a. Mann-Whitney U): Assumes your data are independent observations (as do all of these tests). Makes no assumptions about the size of the groups, variances of the groups, or the shapes of the distributions (e.g. normality). You're fine here --- the p-value you get for this test is probably an accurate reflection of the probability of getting two distributions this different assuming the null hypothesis is true.
2. Student t-test (also some times called just an "independent samples t-test"): Assumes your data are independent observations. Also assumes the variances in the two groups are equal, and that each group is normally distributed. Although it's actually pretty robust to violations in the assumption of normality (so the p-values usually end up being pretty accurate even when your data aren't normally distributed), the fact that you have very unequal sample sizes compounds the potential issue of unequal variances, making that a much larger problem. Your p-value here may not really reflect what you think it does! It is NOT a good idea to test for equality of variances and then pick whether to use a t-test or Welch approximation t-test based on that --- rather, you should avoid Student's t-test completely when your sample sizes are so different.
3. Welch t-test: Assumes your data are independent observations. Does NOT assume variances are equal, so it's fine if they aren't (and it's fine with different sample sizes). It does still assume the data are normally distributed, but as long as your sample size is large enough a violation of the assumption of normality doesn't affect p-values very much.

So using the Student's t-test would be a bad idea, but either the nonparametric Mann-Whitney U test or the Welch approximation t-test would be fine. You're right that the Mann-Whitney U is generally less powerful than a parametric test --- that's because it discards a lot of information in your data by switching from using the scores themselves to using the ranks of the scores (i.e. it only cares about which scores are higher than which other scores, not how much higher they are).

A fourth option would be to use a resampling procedure called a permutation test (see this question and this question for a description and instructions). It makes no assumptions about normal distributions or equal variances, and works well for small sample sizes.

In sum, you could use the non-parametric test, the Welch test, or a permutation test, but not the Student's t-test.

I recommend the Welch t-test in most situations like this because it's simple to run and interpret (the interpretetion of the nonparametric test can be a little tricky since it's testing the difference in ranks, as I mentioned), and it's appropriate for unequal sample sizes. The nonparametric test and the permutation test are safer, though, because they make no assumptions about normally distributed data, so if you feel comfortable running and interpreting those you could do that instead.

• Note that p-values (which can be tricky to interpret even under the best circumstances) don't matter for some other approaches to statistical analysis. Bayesian tests provide the probability of a hypothesis (or a probability distribution over a range of possible values), rather than providing the probability of observing your data given the null hypothesis. If you're relying on frequentist significance tests, though, it's vital that you understand what needs to be true about your data and your testing procedure in order for your p-values to be accurate.

I think your testing provided all the information you need. You are in an ambiguous situation whereby one sample is explicitly Normally distributed. Meanwhile, the second one apparently is not but is probably way too small to statistically test whether it is truly Normally distributed or not.

You have done the right thing anyway. And, that is to test whether the average of the two samples are different using both parametric and nonparametric tests. Your situation is very common (ambivalence whether the samples are truly Normally distributed or not). Therefore, testing such hypothesis both ways is most often the best way to go.

The three tests you have used cover the whole range of underlying statistical assumptions from the unpaired Student t test that assumes that your data is Normally distributed and that your samples have reasonably similar variances to the Welch's test that relaxes the assumption of equal variance (can handle unequal variance) and the Mann-Whitney test that does not assume the underlying data is Normally distributed.

The three tests directionally tell you the same thing. Category B's average is greater than Category A's average (at different Alpha threshold levels). Granted two of the tests are associated with p-values meeting standard statistically significant thresholds. And, one of the tests (Welch) has a p-value a bit greater than that. I think the divergence, of the Welch test in particular, is due in good part to the small size of the respective samples more than the lack of a true difference between the two groups averages. Heck, Category B average is more than 3 x greater than Category A. So, this difference is very material.

Keep in mind that in various statistical testing it is very common for tests to have different sensitivities and generate diverging outcome in terms of statistical significance.

• "from an empirical standpoint it won't make a material difference" Do you have evidence for this? The simulations I linked to (and other references in that post) show that empirically there IS an inflation in type 1 error rate for using independent samples t-test when sample sizes are unequal. – Rose Hartman Dec 24 '16 at 20:32
• This is a fine distinction, but to be clear the nonparametric test doesn't test whether "Category B average is statistically significant greater than Category A" as you suggest --- it tests whether an observation selected from Category B is more likely to be higher than a score selected from Category A than vice versa. Especially if one or both of the distributions is very non-normal, the averages may actually be the same or even show the reverse pattern. – Rose Hartman Dec 24 '16 at 20:57
• Regarding the "material difference" issue. It seems at times as you indicate it can make a difference. This question raised by Agata is a case in point. I have encountered quite a few situations where it does not make much difference if at all. I have shared some of my findings in this slide share.net: presentation.slideshare.net/gaetanlion/unpaired-t-test-family. – Sympa Dec 25 '16 at 0:30
• Rose, I accept and understand your comments and I have thoroughly edited my answer. Thanks for your constructive feedback. – Sympa Dec 25 '16 at 0:33
• Thanks for your response, Sympa. There's one other thing you may want to update --- you followed OP's interpretation of the tests of normality, but s/he actually got the effect reversed (a mistake I didn't catch on my first read either). The Shapiro-Wilk test has normality as the null hypothesis, so a significant result means the distribution tested deviates from normal. OP found that the larger group is significantly NOT normal, and the smaller group didn't deviate significantly from normal (but that could just be because the n's so small, so the test is under powered, as you point out). – Rose Hartman Dec 25 '16 at 5:58