1
$\begingroup$

I am working on an online learning problem, in which I need to detect whether two independent samples (possibly of different sizes) come from the same underlying distribution. Let us call the samples $S_{1}$ and $S_{2}$, such that $100 \leq |S_{1}|, |S_{2}| \leq 1000$ and both can come from any kind of distributions (i.e., I can make no assumptions on their shape or form).

After going over multiple online resources, I found out that there exist two main tests that can serve my goal:

  • paired T test: with the assumption that $|S_{1}| = |S_{2}|$ and that both samples come from normal distribution.

  • Mann-Whitney U test: with no assumptions on the samples' sizes and that both need to come from normal distributions.

Which of the two tests should I use, and what is the accuracy of Mann-Whitney U test if both samples come from normal distributions**(question 1)? Is there a rule of thumb on which test one should pick based on sample size **(question 2)?

Improve this question
$\endgroup$
7
  • $\begingroup$ Could you please explain, what you mean by $100 \leq |S_{1}|, |S_{2}| \leq 1000$ ? $\endgroup$
    – Bernhard
    Commented Mar 12, 2018 at 13:17
  • $\begingroup$ @Bernhard I mean that each sample size will contain a number of elements between $100$ and a $1000$. For instance, $S_{1}$ might contain $111$ elements and $S_{2}$ $527$ elements. $\endgroup$
    – nick.katsip
    Commented Mar 12, 2018 at 13:37
  • 1
    $\begingroup$ 1. What you have not stated is your hypothesis that is to be tested. What are you trying to find out? 2. Why would you use a paired t-test on independent samples?? What is the justification for this? 3. Also note that there is no requirement whatever in the Mann-Whitney in relation to the normal distribution. $\endgroup$
    – Glen_b
    Commented Mar 13, 2018 at 4:44
  • $\begingroup$ @Glen_b the hypothesis I want to test is whether the distributions, from which $S_{1}$ and $S_{2}$ are the same (or at least similar). That can be translated as similarity in medians (or even deviation from median values). Bernhard explained why I don't need to use a paired T-test, so this is a mistake on my end. Turning to comment 3, I am aware of that. Thank you for your comment. $\endgroup$
    – nick.katsip
    Commented Mar 13, 2018 at 7:04
  • 1
    $\begingroup$ 1. The distributions can differ in all sorts of ways even with identical medians; if you want to test for differences in distribution, then test that. 2. Neither the Mann Whitney nor the two sample t-test is specifically a test of medians. 3. In relation to my earlier comment 3. then I don't understand what you mean when you wrote "Mann-Whitney U test: with no assumptions on the samples' sizes and that both need to come from normal distributions." in your question $\endgroup$
    – Glen_b
    Commented Mar 13, 2018 at 8:14

1 Answer 1

Reset to default
2
$\begingroup$

Please note, that the non-paired t-test as well as the U-test both tell you, whether $S_1$ or $S_2$ leads to systematically larger or smaller values. They will not detect different distributions, as long if not one produces larger or smaller values.

t.test(rnorm(100), runif(100, -1, +1))

In R this will perform a t-test comparing values from a standard normal distribution with values from a uniform distribution between -1 and +1 and there is no significant result. The same is true for the U test.

As there are no assumptions about the underlying distribution, I would tend towards a U-test, but the t-test is quite robust against deviation from normality when you compare large numbers (and yours are large in this respect). It will however eventually loose it's advantages with respect to statistical power in non-normally distributed data.

So you are basically free to choose between both tests with only a slight and subjective tendency towards the nonparametric test.

As you have independent data, you do not want a paired t-test.

Improve this answer
$\endgroup$
11
  • $\begingroup$ thank you very much for your answer. Just a clarification: Is the reason that you mention t-test's robustness when comparing large samples due to the Central Limit Theorem? $\endgroup$
    – nick.katsip
    Commented Mar 12, 2018 at 19:41
  • 1
    $\begingroup$ @Bernhard the two sample t-test has a degree of level-robustness for the reasons you mention (depending on your tolerance for impact on type I error rate). However, it doesn't necessarily have very good power in all such situations, and in fact the power may sometimes be fairly poor. Since people care about more than type I error, this may often be an important consideration, particularly if it's large sample sizes and small effect sizes $\endgroup$
    – Glen_b
    Commented Mar 13, 2018 at 4:47
  • $\begingroup$ @Glen_b I had the power included in my text but probably understated it. Thanks for making this more prominent. $\endgroup$
    – Bernhard
    Commented Mar 13, 2018 at 6:55
  • 1
    $\begingroup$ @nick.katsip Yes, the Central Limit Theorem allows us to regard the means as normally distributed where the distribution of the data does not matter. In my personal taste, if I may add that, the t-test is preferable if you need a nice effect size measure and thus if you need to do power analysis. Also, it seems to be easier to understand a confidence interval of the difference of means then that of a quasi-median. If you just need a p-value, have high enough numbers and no strong distribution assumptions, go with the rank-sum test/U-test. $\endgroup$
    – Bernhard
    Commented Mar 13, 2018 at 7:05
  • 1
    $\begingroup$ @nick.katsip $S_1$ ist over 100, which is a large number in this respect und $S_2$ will probably also be "large", if you describe it as "less then 1000". If you feel uncomfortable with the whole "large, whatever that is" -thing, then go for nonparametric. Your whole discussion with Glen_b here is of no importance for the U-test. If you want to stick with t-tests but are unshure about unequal sample sizes and unequal variances you might consider a modification of the t-test known as the Welch-test (or Welch correction of the t-test). $\endgroup$
    – Bernhard
    Commented Mar 13, 2018 at 8:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.