Testing for significance between means, having one normal distributed sample and one non normal distributed

Question

I have following problem:

Within an independent groups 1-factor design I have two independent groups, with a sample size of 20 each. The data of the treatment group is not normally distributed, whereas the data for the control group is (checked with Shapiro-Wilk Normality Test). Now I want to check if the differences of the means of both groups are significant. What is the appropriate test for this? I think it should be the Wilcoxon Rank Sum and Signed Rank Test, but I am not sure...

Could please anybody help me?

Answer 1

If you are 100% sure that the two samples are drawn from populations with different distributions (one Gaussian, one not), are you sure you need any statistical test? You are already sure that the two populations are different. Isn't that enough? Does it really help to test for differences in means or medians? (The answer, of course, depends on your scientific goals, which were not part of the original question.)

Answer 2

You probably want the The Wilcoxon Rank Sum, also called Mann-Whitney U here. The Signed Rank Test is for paired samples, so not appropriate in your case.

Your choice is basically between a t-test or Wilcoxon, and come down to a balance between

• On the one hand, t-test is more powerful particularly in small samples (which you have), but does assume normality
• On the other hand, Wilcoxon is more powerful when far from normality, and is asymptotically close to the t-test in power (but you are far from the asymptote!)

So the first question you need to ask is, how non-normal are we talking? If we are talking seriously bad*, e.g. clearly bimodal (or multimodal!), or massive skewed, then the t-test can probably be ruled out.

On the other hand, as a general rule of thumb, if it looks bell shaped, then t-test is probably ok.

Arguably, the proper way to proceed, is to do a study of power for your particular set up before you handle the real data too much. Presuming you have some vague idea of the shape of the non-normal distribution* you should generate pretend samples from your weird distribution and your normal distribution and see how well the t-test and Wilcoxon work on your Monte Carlo data.

Don't forget you need to check both False Positives and False Negatives. So run once where you make the null hypothesis true (i.e. the samples have the same mean) and another where they differ by the approximate effect size you are looking for (or the effect size that would be "materially interesting" to you).

*I am assuming you must either have some a priori reason to believe that one sample is non-normal, or it is absolutely hideous - otherwise it would be almost impossible to spot much deviation from normal with only 20 data points.

Answer 3

If data in the treatment group is not normal while the control group is it sounds like the treatment may only be affecting a subset of the sample or having variable levels of effect. Comparing means under such circumstances would be losing out on this information. You should attempt to offer explanations for why this change of distribution occurred rather than only comparing means. The rank tests assume that both groups come from the same shape distribution. If you believe the distributions are different the tests are not useful for your purposes.

Let us take an example of what can happen with the U-test. We will make our control group come from a normal distribution with mean=0. Meanwhile the treatment will have negative effects on half the subjects and positive effects on the other half. So the treatment group will come from two normal distributions. The first with mean=-5, the second with mean=5. All distributions have sd=1 and both groups have sample size=100. Red shows the treatment group while blue shows the control group:

Results of doing a U-test (which is also called the Wilcoxon test):

        Wilcoxon rank sum test with continuity correction

data:  a and b 
W = 4999, p-value = 0.999
alternative hypothesis: true location shift is not equal to 0

We can see it returns "not significant". Would you really want to conclude the treatment had no effect?

R code for generating the above:

##Generate Data
control<-rnorm(100,0,1) # create control data
treatment<-c(rnorm(50,-5,1),rnorm(50,5,1)) # create treatment data


##Plot data
# Get min/max values (for plotting)
min.val<-min(control,treatment)
max.val<-max(control,treatment)

# make plots
hist(treatment, breaks=seq(min.val-.1,max.val+.1,.5), col="Red",
xlab="Value", ylim=c(0,20),
main="Results"
)
hist(control, add=T, breaks=seq(min.val-.1,max.val+.1,.5),col="Blue")

##perform U-test
wilcox.test(treatment,control)

Answer 4

I think in such case when we have two samples one normally distributed and another was not it is more accurate to make transformation for data to get normal distribution for two samples and compare the means by t-test. As t-test is more powerful for small samples. Otherwise we can used non parametric test depend on the type of samples independent or dependent.