How to perform Hypothesis Testing (2 samples) when one sample is normal and the other is non-normal? Recently, I was helping a colleague to perform hypothesis testing for 2 independent samples, but the data one sample followed a normal distribution and the other followed a non-normal distribution.
What would be the best approach when having a situation like this one?
Is it better to perform a non-parametric test (for example, mann-whithey U-test) than a parametric (say t-test)?
 A: This previous post and this one may be helpful. Non-parametric tests might be an option. However, when two samples have very different distributions, I think a good first step would be to ask why you think that is and look at how different the distributions are. For instance, if the groups have different distributions on outcome Y and the groups are very different in terms of context or demographics (e.g., one is a 50/50 mix of men and women, one is 95% men), it might alter your research question and choice of analysis.
It also may be helpful to know the size of each group. If your sample size is large enough perhaps it could be possible to match people/subgroups in some way.
A: Yes, it would be better to use a non-parametric test. You should not use a t-test if you know that one of your samples in not Normal.
In my opinion, the best way to test if the mean (or median or some other statistic) of two samples are different is to use a permutation test. Basically, you shuffle the labels of whether the samples came from one or the other group and take your stat on the shuffled data. And you do that a few thousand times and compare the actual difference of the means to the shuffled difference of the means.
There is a matlab implementation of this here: https://github.com/erlichlab/elutils/blob/master/%2Bstats/bootmean.m
