Which test to choose when the results from t-test and Wilcoxon test are different?

Question

I have a sample of 48. According to the central limit theorem, I may consider means of every continuous variable in my sample to have a normal distribution. However, one variable has a mean of 14 +/- 8 and when I draw a normal quantile plot, it seems to not be a normal distribution. If I use a t-test to compare this variable between 2 groups then the p - value is < 0.05 and the diffenrence is significant. But when I use a Wilcoxon test I have the result, p = 0.07 and I cannot conclude a significant difference between 2 groups.

Which test must I have to use in this situation, t-test or Wilcoxon test?

Answer 1

The difference between p =.05 and p = .07 at that sample size is not significant by itself (Gelman and Stern, 2006; see also here) - so both tests give you basically the same result.

If a parametric and a non-parametric test disagree I would take a close look at the raw data (which I would do anyway ...); my first guess would be that outliers or other violations of the parametric assumptions distort the parametric test.

Answer 2

If a parametric model is "correct" (meaning that it reasonably approximates reality) then it will give more efficient estimates and yield a more powerful test. So in such a case you could expect the parametric test to be able to detect smaller effect with the parametric test. But if the assumption of normal error is violated in a significant way then the parametric test could give incorrect results. In that case the nonparametric method is at least still reliable. So you would go with the nonparametric test. The real difficulty is determining how large a departure from normality is enough to invalidate the parametric test.

But Felix got it right. You don't have that problem in this case. The difference between 0.05 and 0.07 is not much and can be expected to happen when two different tests are used. The tests agree. If you think they are radically different you are putting too much emphasis on 0.05 as a cutoff for significance.

Answer 3

I hesitate a little to post this, since it probably not a full answer. But I feel than an important aspect is missing in the others and since comments are limited in size and not-editable I will post it like this.

It is true that the t-Test will generally give you good results if the sample size is large enough. What is large enough depends on the underlying distribution. Even so, it is possible to have a sample large enough for the CLT to kick in for the mean so that the nonparametric Wilcoxon test and the t-Test still give very different answers and are both right. That is because the t-Test tests the means and Wilcoxon test the medians. And in non-symmetric distributions these can differ.

In this case you don't just have to check whether it is valid to use these tests. As said, it is very possible that both will give reliable results. You also need to think about what you want to know. A typical example is income which can have a very high mean and a much larger median.

I say this because I do not know what you have a sample of, what interests you and what the deviations from normality look like. In many cases the distributions are symmetrical enough so that both tests answer more or less the same question. Sometimes they answer very different questions and this is not linked to the validity of the answers.

In your case I suspect that this is probably not the case, but it might help anyhow. As said, a difference between a p-value of 0.05 and 0.07 is not significant.

EDIT: I decided to expand on this even more, due to comments. It is true that we compare two samples. It is still the case that the Wilcoxon (even the Rank-Sum) test looks for a median shift > 0 and t.test looks for a mean shift > 0. Note that the median shift is not the shift of medians. Generate data in R like this:

x1 <- 100 + 0.01*rnorm(1000) 
 # Effectively constant, with some jitter to avoid ties
shift.down <- seq(-10, 0, by = 10/499)
shift.up <- seq(0, 100, by = 100/499)
x2 <- 100 + c(shift.down, shift.up)
t.test(x1, x2)
mean(x1-x2)        # will be significant
wilcox.test(x1,x2) # will be insignificant
median(x1-x2)
median(x1)-median(x2)

This works because both the median shift and the shift of medians is zero. However,

x1 <- rnorm(1000)
x2 <- rnorm(1000)
x2[x2>0] <- x2[x2>0]^4
median(x1) - median(x2)
median(x1 - x2)
mean(x1) - mean(x2)
mean(x1 - x2)
t.test(x1, x2)
wilcox.test(x1, x2)

will give a significant wilcoxon test since we have a median shift even though we do not have a shift of medians.

Answer 4

The assumption of normality in the t-test is about the specific sample that you have, not a theoretical distribution. Just because they should be theoretically normal doesn't mean they are, as you suggest from your own examination of the distribution. Therefore, you've violated the t-test assumptions and the Wilcoxon is a better choice. This is pretty common with a small N. It's unclear from the way you've worded things what the N in each group is. Is it a paired test?

It sounds like you're saying you have a few t-tests to do. If that's the case then you should probably use the non parametric test for all of them. It doesn't sound like you want to really estimate a parameter, but just check significance. Therefore, you wouldn't be getting anything extra out of the parametric test and it's not doing what you want when you violate it's assumptions.

Furthermore, if you really do have a number of tests to do within the same experiment then what your'e calling significant might not be once alpha is adjusted for the multiple comparisons. But theory would drive what needs to be done there and you haven't given us any.

Further to Felix S's answer, he implies that you should examine the data to see if the parametric effect was driven by particular data points. There might be something meaningful you can say about that.