How to perform Hypothesis Testing (2 samples) when one sample is normal and the other is non-normal? [duplicate]

Question

Recently, I was helping a colleague to perform hypothesis testing for 2 independent samples, but the data one sample followed a normal distribution and the other followed a non-normal distribution. What would be the best approach when having a situation like this one? Is it better to perform a non-parametric test (for example, mann-whithey U-test) than a parametric (say t-test)?

Answer 1

This previous post and this one may be helpful. Non-parametric tests might be an option. However, when two samples have very different distributions, I think a good first step would be to ask why you think that is and look at how different the distributions are. For instance, if the groups have different distributions on outcome Y and the groups are very different in terms of context or demographics (e.g., one is a 50/50 mix of men and women, one is 95% men), it might alter your research question and choice of analysis.

It also may be helpful to know the size of each group. If your sample size is large enough perhaps it could be possible to match people/subgroups in some way.

Answer 2

What test to use depends on the shapes of the distributions of the two samples. There must be some similarities or it doesn't make much sense to do tests to find their differences.

For example, one sample may come from a process in which results are waiting times from a three step procedure and and the other sample from a process in which there may be four steps. That might lead to gamma distributions with shape parameters $\alpha = 3$ and $4$ respectively. If you knew more about the underlying processes you might use the precise population distribution families to get a 'tailor made' most powerful test.

But if both of the two samples are mildly right skewed, you might be interested in whether the two populations from which they arise have different centers, so that one distribution gives generally larger values than the other.

Perhaps data summaries from the two samples x1 and x2 are as shown below.

summary(x1); length(x1); sd(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   3.16   16.55   26.68   30.80   41.95   88.93 
[1] 100       # sample size
[1] 16.75382  # sample SD
length(unique(x1))
[1] 99        # only one tie in 100 observations

summary(x2); length(x2); sd(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   4.16   21.82   34.52   38.52   48.70  105.74 
[1] 120
[1] 20.24919
length(unique(x2))
[1] 118

The median of the second sample is larger than the median of the first. The question is whether this shift in location in large enough to be called 'statistically significant' at the 5% level.

Notched boxplots. Boxplots show different medians and mild right skewness in both samples with few outliers. Furthermore, 'notches' in the sides of the boxes are nonparametric confidence intervals, roughly scaled so that nonoverlapping intervals--as here--may indicate a difference in locations.

boxplot(x1, x2, col="skyblue2", pch=20, names=T, notch=T)

Two-sample Wilcoxon test. With such large sample sizes, moderate skewness, and no far outliers, some statisticians might be comfortable using a Welch two-sample t test, which does not assume equal population variances, but does assume both populations are normal. I would prefer to use a two-sample Wilcoxon rank sum test (unless there were a large number of tied observations). With a P-value $0.005 < 0.05 = 5\%$ the Wilcoxon test finds a significant difference between the two samples.

wilcox.test(x1,x2)

        Wilcoxon rank sum test with continuity correction

data:  x1 and x2
W = 4678, p-value = 0.004938
alternative hypothesis: true location shift is not equal to 0

ECDF plots and stochastic domination. Because the two samples are of slightly different shapes one could quibble whether the Wilcoxon test has found a difference in sample means. However, empirical CDFs (ECDFs) of the two samples show that values of Sample 2 are consistently greater than values of Sample 1. (The ECDF of Sample 2 plots consistently to the right of the ECDF of Sample 1, thus below.) One says that Sample 2 stochastically dominates Sample 1.

hdr = "ECDFs of Sample 1 (blue) and Sample 2"
plot(ecdf(x2), col="brown", main=hdr)
 lines(ecdf(x1), col="blue")

Kolmogorov-Smirnov Test. A two-sample Kolmogorov-Smirnov test looks to see if two ECDFs differ significantly. For my fictitious data, its test statistic $D$ is the maximum vertical distance between the two ECDFs shown above. For small samples, this test often has very poor power to distinguish between two distributions. For my data the KS-test has P-value barely below 5%.

        Two-sample Kolmogorov-Smirnov test

data:  x1 and x2
D = 0.18833, p-value = 0.04174
alternative hypothesis: two-sided

Warning message:
In ks.test(x1, x2) : p-value will be approximate 
  in the presence of ties

Permutation test. A permutation test also shows that the two samples are significantly different in location. Although I wonder whether the pooled t statistic has exactly a t distribution, this statistic seems a reasonable 'metric' for measuring the difference in location. The observed pooled t statistic is $-3.04.$

x = c(x1, x2)
g = c(rep(1, 100), rep(2, 120))
t.obs = t.test(x~g, var.eq=T)$stat;  t.obs
        t 
-3.040704 

In a permutation test, we repeatedly scramble the $n_1+n_2 = 220$ observations between one sample of size 100 and and another of size 120, and find the pooled t statistic for each scrambled version of the data. [In the R code below sample(g) does the random permuting.] That gives us a good idea of the permutation distribution of the chosen metric, with which we can compare t.obs to find the P-value $0.0033$ of the permutation test.

set.seed(511)
t.prm = replicate(10^4, t.test(x~sample(g), var.eq=T)$stat)
mean(abs(t.prm) >= abs(t.obs))
[1] 0.0033      # P-value of permutation test
length(unique(t.prm))
[1] 9186        # Nr of unique values of metric

The histogram below shows the simulated permutation distribution. The P-value of the two-sided test is the area outside the vertical dotted lines.

hist(t.prm, prob=T, col="skyblue2", main="Permutation Dist'n")
 abline(v = c(t.obs,-t.obs), col="red", lwd=2, lty="dotted")

For my fictitious data, there was really no need to do a permutation test because the two-sample Wilcoxon rank sum test gave useful results. However, your actual data may have too many ties for a Wilcoxon test or two sample distributions may not be of approximately the same shape. In such cases, the permutation test may be useful.

Note: The R code used to sample the fictitious data used above is as follows:

set.seed(2010)
x1 = round(rgamma(100, 3, .1), 2)
x2 = round(rgamma(120, 4, .1), 2)

Answer 3

Yes, it would be better to use a non-parametric test. You should not use a t-test if you know that one of your samples in not Normal.

In my opinion, the best way to test if the mean (or median or some other statistic) of two samples are different is to use a permutation test. Basically, you shuffle the labels of whether the samples came from one or the other group and take your stat on the shuffled data. And you do that a few thousand times and compare the actual difference of the means to the shuffled difference of the means.

There is a matlab implementation of this here: https://github.com/erlichlab/elutils/blob/master/%2Bstats/bootmean.m