# Different p value from Wilcoxon and ANOVA

I got a different $$p$$-value from a Wilcoxon test and an ANOVA test. When I use the Wilcoxon test, the $$p$$-value is less than $$0.05$$:

treatmentA <- drug$$weight[treatment == "A"] treatmentB <- drug$$weight[treatment == "B"]

wilcox.test(treatmentA, treatmentB, paired = F)


If run ANOVA, the $$p$$-value is more than $$0.05$$:

aov(weight~treatment, data = drug)


There are $$2$$ independent samples (comparing weights from Treatment A and B)), more than $$100$$ people in the sample, missing data, and the weight is a skewed distribution.

What should I use? Is the result because of missing values?

• Are the p values close (0.04 vs 0.06)? You are providing very little information for a meaningful answer. One of the assumptions for performing ANOVA is: "Each group sample is drawn from a normally distributed population." if your data is non normal than Wilcoxon is probably the better test. Commented Jun 12 at 1:18
• Thanks, p from Wilcoxan in 0.041, while p from anova is 0.14. the data is non-normal distribution.
– bibi
Commented Jun 12 at 1:27
• You should decide which test you are going to do before you look at the results and decide. You wouldn't expect them to give the same result, or there wouldn't be two different tests. Commented Jun 12 at 3:27
• I'm curious why you used aov with two samples, rather than a two sample t-test. Commented Jun 12 at 3:48
• In addition to great comments and a great answer so far, I would recommend working with weight on a log scale. If anyone wondering about the statistical propriety of choosing a transformation after looking at the data, I counter that this would be my advice without looking at your data. That needn't be a log transformation; a generalized linear model with log link would be a better idea. If you want to show us your data, that would allow comparison of procedures and results. Commented Jun 12 at 12:54

The two tests "see" the data differently and they correspond to inference about different population parameters. They will yield different p-values and they tell you different things.

You did an anova but with two samples that should be equivalent to the t-test. In this specific case aov would give you a equal-variance anova (not the Welch-Satterthwaite; you can get that with oneway.test) and would correspond to the equal-variance t-test (not the default Welch t-test that R gives if you don't specify).

While the t-test measures difference in means and estimates the population difference in means ($$\mu_A-\mu_B$$), for a continuous response variable the Wilcoxon-Mann-Whitney test measures the proportion of times $$A_i>B_j$$ and correspondingly estimates $$P(A>B)$$ (in particular, $$\frac{U}{mn}$$ is the estimate, where $$m$$ and $$n$$ are the sample sizes and $$U$$ is the usual Mann-Whitney statistic). These are very different things to be hypothesizing about.

You shouldn't choose between them post hoc, particularly not looking at p-values like this. It's a recipe for special pleading (You end up with things like "well, now I've looked at my data, I can see a good reason I should have chosen the one with the p-value I like" ... even if not consciously chosen in that way), multiple bites at the cherry. This is, in short, p-hacking.

I strongly recommend looking at the points I made here, but since the context is slightly different I'll hazard a brief recapitulation (adapted to present context):

A hypothesis will be a statement about one or more population parameters. If you want to perform a hypothesis test, you must start with a hypothesis to test, then choose a suitable test for that hypothesis.

Treating tests that relate to quite distinct hypotheses as if they were readily-substituted options is poor practice. For example, the effects that you're estimating under the two tests you mention (the estimates of the quantities I described earlier) might not even be in the same direction.

Presumably you care enough about which parameter you're discussing and which direction you claim the difference is in or how large it might be to take some care over which effect you're hypothesizing about and making claims about in a conclusion.

You can of course make an additional assumption that the effect of interest is a pure location-shift in which case the parameter being targeted should coincide (if the population means are finite), as indeed would that of a number of other possible tests. However, since each test can respond to other classes of alternative, your additional assumption may mislead you when those other kinds of alternatives actually hold.

There's no particular need for either test to make such pure-shift assumption, both can maintain their usual type I error rate (which doesn't involve the effect under the alternative) and also maintain good power against a wide variety of alternatives without that specific requirement.

Choose your parameter first (what population quantity are we hypothesizing about), then choose how to measure it in practice, then choose how to test that hypothesis (considering the reasonableness of the assumptions under $$H_0$$, and whether the power might be adequate under likely classes of alternative). Then when you actually know what it is you're trying to find out about the population, you're in a position to go ahead and design experiments/studies for that particular question and consequently, to collect data to answer it.

Then you do a test of your hypothesis. Just one. Do not entertain multiple distinct tests and then look at the data before you choose which one you like; if you do, your actual significance levels, and hence actual p-values, will tend to be higher than you claim them to be.

Is the result because of missing values?

No. Both tests use the same non-missing values - unless you did something unintended in the code by default I believe they would both just omit the missing values.

Skewness is not itself necessarily dispositive; a two sample t-test may be perfectly fine under a moderate amount of skewness; the significance level will be unlikely to exceed the desired significance level by more than a tiny amount, though it may be lower. Power (relative to that from a more suitable model) may be more affected. If "weights" are necessarily positive and I'm interested in means (as typically I would be) then I might wish to choose a more suitable model, perhaps a gamma GLM would suffice.

However a t-test(/anova) should usually do okay on two samples unless the populations distributions are quite far from normal (say if they were very strongly skew; the population of weights might not be). An alternative of you were concerned about maintaining type I error rate (and didn't want to choose a model) might be a permutation test of means. You could use the t-statistic in such a test. All you'd need is that you would believe that the population distributions should be very similar if $$H_0$$ were true. (They don't have to be similar when it's false; if your sequence of alternatives progresses gradually from the similarity under $$H_0$$ toward whatever sort of difference you end up at, things may work perfectly well; the data are not necessarily relevant to the assumptions under the null and may not be that crucial for the alternative if you're not relying on power calculations; even then you have other options.)