5
$\begingroup$

If you analyze the same data with a t test and the nonparametric Mann-Whitney test, which do you expect to have the lower P value?

$\endgroup$
1
  • $\begingroup$ Someone asked me this, so I am posting both the question and my answer. $\endgroup$ Aug 31, 2012 at 15:19

2 Answers 2

7
$\begingroup$

It depends.

If you assume that the data are sampled from Gaussian distributions, then the t test has a bit more power (depending on sample size) so will -- on average -- have a lower P value. But only on average. For any particular set of data, the t test may give a higher or a lower P value.

If you don't assume the data are sampled from Gaussian distributions, then the Mann-Whitney test may have more power (depending on how far the distribution is from Gaussian). If so, you'd expect the Mann-Whitney test to have the lower P value on average, but the results are not predictable for any particular set of data.

What does "on average" mean? Perform both tests on many sets of (simulated) data. Compute the average P value from the t test, and also the average P value from the Mann-Whitney test. Now compare the two averages.

$\endgroup$
2
  • $\begingroup$ Is there Monte Carlo research or an analytic solution on which you based this answer? $\endgroup$
    – Joel W.
    Aug 31, 2012 at 15:46
  • 3
    $\begingroup$ @JoelW: an analytic approach to the relative power of the Wilcoxon-Mann-Whitney and t-test is discussed in this question. As Harvey says, for data from a normal distribution the t-test has more power in large samples - it's relative power versus the U test is $\pi/3$ - but for non-normal data, the U test can have rather more power (3 times on exponential, 1.5 on double exponential, infinitely so on Cauchy). For uniformly distributed data, they're equally powerful. $\endgroup$
    – Silverfish
    Jan 1, 2015 at 22:44
1
$\begingroup$

Here's an example showing exactly the behavior @Harvey describes above. We simulate varying degrees of normality by appending an outlier of varying degrees to a random normal draw of sample size 100 and calculate t-test p-values and U-test p-values for each, then plot as a function of how much of an outlier we appended.

boxplots, sorry

For simulations where the outlier is still pretty normal, we get more power from the t-test. For simulations where the outlier is incredibly non-normal, we get more power from the U-test. This is because the t-test's power drops enormously (and appears to asymptote around 0.33) while the U-test's power is fixed without regard to normality (as it should be).

R code for the figure shown below:

library(tidyverse)
map_dfr(2^(1:10), function(non_norm){
  bind_rows(replicate(20, {
    a <- rnorm(100)
    b <- c(rnorm(99, mean = 2), non_norm)
    c(stud_t_test=t.test(a, b)$p.value, u_test=wilcox.test(a, b)$p.value)
  }, simplify = FALSE)) %>%
    mutate(non_norm)
}) %>%
  pivot_longer(ends_with("test"), names_to = "test_type", values_to = "p_value") %>%
  ggplot(aes(x=non_norm, y=p_value, color=test_type, group=interaction(non_norm, test_type))) +
  geom_boxplot() +
  scale_x_continuous(trans = "log2", breaks = 2^(1:10)) +
  scale_y_log10(breaks = 10^(-(0:3)*10)) +
  theme_bw()
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.