4
$\begingroup$

I have a sample of 48. According to the central limit theorem, I may consider means of every continuous variable in my sample to have a normal distribution. However, one variable has a mean of 14 +/- 8 and when I draw a normal quantile plot, it seems to not be a normal distribution. If I use a t-test to compare this variable between 2 groups then the p - value is < 0.05 and the diffenrence is significant. But when I use a Wilcoxon test I have the result, p = 0.07 and I cannot conclude a significant difference between 2 groups.

Which test must I have to use in this situation, t-test or Wilcoxon test?

$\endgroup$
  • $\begingroup$ This is a small N. You might want to post the actual numbers to get better advice on your problem. $\endgroup$ – John Jul 27 '12 at 9:10
6
$\begingroup$

The difference between p =.05 and p = .07 at that sample size is not significant by itself (Gelman and Stern, 2006; see also here) - so both tests give you basically the same result.

If a parametric and a non-parametric test disagree I would take a close look at the raw data (which I would do anyway ...); my first guess would be that outliers or other violations of the parametric assumptions distort the parametric test.

$\endgroup$
5
$\begingroup$

If a parametric model is "correct" (meaning that it reasonably approximates reality) then it will give more efficient estimates and yield a more powerful test. So in such a case you could expect the parametric test to be able to detect smaller effect with the parametric test. But if the assumption of normal error is violated in a significant way then the parametric test could give incorrect results. In that case the nonparametric method is at least still reliable. So you would go with the nonparametric test. The real difficulty is determining how large a departure from normality is enough to invalidate the parametric test.

But Felix got it right. You don't have that problem in this case. The difference between 0.05 and 0.07 is not much and can be expected to happen when two different tests are used. The tests agree. If you think they are radically different you are putting too much emphasis on 0.05 as a cutoff for significance.

$\endgroup$
4
$\begingroup$

I hesitate a little to post this, since it probably not a full answer. But I feel than an important aspect is missing in the others and since comments are limited in size and not-editable I will post it like this.

It is true that the t-Test will generally give you good results if the sample size is large enough. What is large enough depends on the underlying distribution. Even so, it is possible to have a sample large enough for the CLT to kick in for the mean so that the nonparametric Wilcoxon test and the t-Test still give very different answers and are both right. That is because the t-Test tests the means and Wilcoxon test the medians. And in non-symmetric distributions these can differ.

In this case you don't just have to check whether it is valid to use these tests. As said, it is very possible that both will give reliable results. You also need to think about what you want to know. A typical example is income which can have a very high mean and a much larger median.

I say this because I do not know what you have a sample of, what interests you and what the deviations from normality look like. In many cases the distributions are symmetrical enough so that both tests answer more or less the same question. Sometimes they answer very different questions and this is not linked to the validity of the answers.

In your case I suspect that this is probably not the case, but it might help anyhow. As said, a difference between a p-value of 0.05 and 0.07 is not significant.

EDIT: I decided to expand on this even more, due to comments. It is true that we compare two samples. It is still the case that the Wilcoxon (even the Rank-Sum) test looks for a median shift > 0 and t.test looks for a mean shift > 0. Note that the median shift is not the shift of medians. Generate data in R like this:

x1 <- 100 + 0.01*rnorm(1000) #Effectively constant, with some jitter to avoid ties
shift.down <- seq(-10,0, by = 10/499)
shift.up <- seq(0,100, by = 100/499)
x2 <- 100 + c(shift.down, shift.up)
t.test(x1,x2)
mean(x1-x2) # will be significant
wilcox.test(x1,x2) #will be insignificant
median(x1-x2)
median(x1)-median(x2)

This works because both the median shift and the shift of medians is zero. However,

x1 <- rnorm(1000)
x2 <- rnorm(1000)
x2[x2>0] <- x2[x2>0]^4
median(x1)-median(x2)
median(x1-x2)
mean(x1)-mean(x2)
mean(x1-x2)
t.test(x1,x2)
wilcox.test(x1,x2)

will give a significant wilcoxon test since we have a median shift even though we do not have a shift of medians.

$\endgroup$
  • $\begingroup$ I agree with most of what you say but have minor issues on a couple of points. (1) The CLT cannot always be relied on. Some distributions particularly heavytailed ones that don't have second moments will not converge to a normal (probably not the case here). $\endgroup$ – Michael R. Chernick Jul 27 '12 at 12:36
  • $\begingroup$ 2) While sum nonparametric tests could be said to test medians I do not think it is quite right to say that is what the Wilcoxon rank sum test does. It is really a test of location shift. If the populations have different "centers" both the mean and the median will shift although depending on the shape of the distributions maybe not in the same way. So i think you can view the test as a test of both means and medians since it should have power against shifts or you could say it tests neither because it doesn't directly test either. $\endgroup$ – Michael R. Chernick Jul 27 '12 at 12:46
  • $\begingroup$ To add to Michael's first comment, sometimes the CLT still applies asymptotically but for any reasonable finite sample size, it hasn't "kicked in" yet. These examples are usually right on the boundary of the assumptions of the CLT being met (i.e. the variance "almost" doesn't exist). See here for an example. $\endgroup$ – Macro Jul 27 '12 at 12:51
  • $\begingroup$ I am aware of 1) and thought of mentioned it. For practical purposes though you can usually expect that any real world quantity is bounded and this does not apply in real life. 2) Yes, it would have been more correct and precise to say that we test the mean shift vs. the median shift. I don't see your point about "center" since the problem is exactly that depending on your definition "center" might mean median or mean. $\endgroup$ – Erik Jul 27 '12 at 12:53
  • $\begingroup$ @Macro: Sure, but I mention all that. I even use the words "large enough for the CLT to kick in" and "what is large enough depends on the underlying distribution". I don't see what your comment adds to that, with the exception of the nice example. $\endgroup$ – Erik Jul 27 '12 at 12:54
2
$\begingroup$

The assumption of normality in the t-test is about the specific sample that you have, not a theoretical distribution. Just because they should be theoretically normal doesn't mean they are, as you suggest from your own examination of the distribution. Therefore, you've violated the t-test assumptions and the Wilcoxon is a better choice. This is pretty common with a small N. It's unclear from the way you've worded things what the N in each group is. Is it a paired test?

It sounds like you're saying you have a few t-tests to do. If that's the case then you should probably use the non parametric test for all of them. It doesn't sound like you want to really estimate a parameter, but just check significance. Therefore, you wouldn't be getting anything extra out of the parametric test and it's not doing what you want when you violate it's assumptions.

Furthermore, if you really do have a number of tests to do within the same experiment then what your'e calling significant might not be once alpha is adjusted for the multiple comparisons. But theory would drive what needs to be done there and you haven't given us any.

Further to Felix S's answer, he implies that you should examine the data to see if the parametric effect was driven by particular data points. There might be something meaningful you can say about that.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.