5
$\begingroup$

I have two independent samples and I need to compare their values, to say if values in the first group are higher than values in the second group. I performed the Levene's test and the variances are very unequal.

Is it better to use the SPSS version of the t-test (equal variances not assumed) or the U Mann-Whitney test?

Do you have any formal reference (a paper I can cite) for the SPSS t-test version when equal variances are not assumed?

Improve this question
$\endgroup$
2
  • $\begingroup$ How large are the two sample sizes and the two variances? $\endgroup$
    – Michael M
    Commented Apr 11, 2014 at 16:45
  • $\begingroup$ About 15-20 observations in each group. SD is higher than the mean in most cases. For instance, Group1: mean 10, sd 11, Group2: mean .20, sd .40. $\endgroup$
    – Forinstance
    Commented Apr 11, 2014 at 16:49

4 Answers 4

Reset to default
7
$\begingroup$

The Mann-Whitney doesn't require equal variances unless you're specifically looking for location-shift alternatives.

In particular, it is able to test whether the probability of values in the first group are larger than the values in the second group, which is quite a general alternative that sounds like it's related to your original question.

Not only can the Mann-Whitney deal with transformed-location shifts very well (e.g. a scale-shift is a location-shift in the logs), it has power against any alternative that makes $P(X>Y)$ differ from $\frac{1}{2}$.

The Mann-Whitney U-statistic counts the number of times a value in one sample exceeds a value in the other. That's a scaled estimate of the probability that a random value from one population exceeds the other.

shift in P(X<Y) from 1/2

There's more detail here.

Also see the discussion here.

As for which is better, well, that really depends on a number of things. If the data are even a little more heavy-tailed than normal, you may be better with the Mann-Whitney, but it depends on the situation - discreteness and skewness can both complicate that situation, and it also depends on the precise alternatives of interest.

Improve this answer
$\endgroup$
3
$\begingroup$

If the sample sizes are unequal, we should use the unpooled variances t-test. If they're equal, use the pooled. Here's an excerpt from "Understanding and Using Statistics in Psychology" (which I co-authored, with Phil Banyard).

"There are a number of different ways to decide if your variances (or standard deviations) are the same. One of the most common is Levene’s test. If Levene’s test is statistically significant, this means that your variances are different from one another and you should use the unpooled variances t-test, which does not assume homogeneity of variance. If Levene’s test is not statistically significant, you have no evidence that your variances are different. In which case you may be able to use the pooled variances t-test (that’s the normal t-test).

OK, now we’ve got that clear, we tell you the second reason that this is a bit tricky. (Were you still waiting for the second reason that it was tricky?) The problem is that a non-significant result does not mean that the variances are the same. It just means that we haven’t found that the variances are different. And the problem with the tests, such as the Levene’s test, is that they are dependent on the sample size, so when the sample size is small the Levene’s test is not very good at detecting differences in the variances. When the sample size is large, the Levene’s test is very good at detecting differences in the variances.

When do you think that it matters the most that the variances are the same? Yep, that’s right, when the sample size is small. So, when the Levene’s test is good at telling us when the variances are different is precisely when we don’t really care. And when the Levene’s test is not very good is precisely when we do care. It’s a bit like having a solar powered torch – it only works when the sun is shining. (We’ve nearly finished, and if you didn’t quite understand the last part, you are really going to like the next part.) It turns out that homogeneity of variance doesn’t really matter, when the sample sizes are about equal. So if we have equal (or approximately equal) sample sizes we can ignore the assumption of homogeneity of variance, and use the pooled variances t-test.

When the sample sizes are unequal, homogeneity of variance matters a lot more. Given that we only have tests of homogeneity of variance that can tell us if we definitely have it, not if we definitely don’t have it, we should not rely on these, and if the sample sizes are unequal, we should use the unpooled variances t-test. (If you are really gripped by this, there is an excellent article by Zimmerman (2004), that you should read)."

Here's the Zimmerman ref: http://onlinelibrary.wiley.com/doi/10.1348/000711004849222/full . It might be behind a paywall, but the abstract tells you everything you need to know.

Here's the publisher info on the book this is from: http://www.sagepub.com/books/Book226292?siteId=sage-us&prodTypes=any&q=jeremy+miles&fs=1

Also, the equal variances not assumed is not the "SPSS version of the test", it's Welch's t-test. The reference is Welch, B. L. (1938). The significance of the difference between two means when the population variances are unequal. Biometrika 34: 29-35., which is quite a while before SPSS was produced.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks. Since my groups have different sizes, is it better to use the Welch's t-test (which I wish it was called that way directly in SPSS to make our life easier) or the U Mann-Whitney test according to you? $\endgroup$
    – Forinstance
    Commented Apr 11, 2014 at 16:58
  • $\begingroup$ Welch. If you want to compare means, don't use the M-W, because it doesn't compare means. $\endgroup$
    – Jeremy Miles
    Commented Apr 11, 2014 at 18:08
  • 2
    $\begingroup$ I'm suprised no one has brought this up: Mann-Whitney U test is non-parametric test. You would usually utilize this test (instead of the t-test for example) when you are concerned about the assumption of normality holding for your data. Look at a histogram of your data. Does it look normal? If it is clearly skewed you should go with Mann-Whitney. If its pretty close to normal you should be fine with t-test. $\endgroup$
    – bdeonovic
    Commented Apr 11, 2014 at 21:46
  • 1
    $\begingroup$ Jeremy - What is the basis of the claim that the t-test is usually more powerful when there's no indication that we're dealing with normality, nor even that the interest lies in testing means? It seems to be quite a strong assertion unless you limit the situations you're making the claim about. $\endgroup$
    – Glen_b
    Commented Apr 12, 2014 at 9:42
  • 2
    $\begingroup$ It is not correct to say that the $t$-test is generally more powerful. The Wilcoxon test has $\frac{3}{\pi}$ efficiency under normality and is usually much more efficient under non-normality. And in line with another earlier comment, it is not good practice to call statistical tests by their SPSS names. $\endgroup$
    – Frank Harrell
    Commented Apr 12, 2014 at 11:36
1
$\begingroup$

You haven't said exactly what you are measuring. If it is count data then this paper is very helpful. I am curious about the nature of your measurements, since if you have a mean of 10 and a SD of 11, a negative number for a value one standard deviation below the mean may or may not be meaningful.

Review: analysis of parasite and other skewed counts Neal Alexander

DOI: 10.1111/j.1365-3156.2012.02987.x

It is free at:

http://onlinelibrary.wiley.com/doi/10.1111/j.1365-3156.2012.02987.x/abstract;jsessionid=75360AF6882780119FBE6D905DFECFF7.f01t04

Improve this answer
$\endgroup$
1
$\begingroup$

Before using any test see if a log transform will make the variances more similar and if so then apply the test to the transformed values. Any conclusion that you might draw form log-transformed data will be equally applicable to the raw untransformed data. See my answeer to this question and the comments elicited for more thoughts: Comparing smoke and hormones level in two groups of people. Which test?

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ That multi-step procedure will not preserve the type I error of the test. $\endgroup$
    – Frank Harrell
    Commented Apr 12, 2014 at 11:36
  • $\begingroup$ @FrankHarrell Two questions: 1. What multistep procedure? 2. Is it essential to preserve the type I error rate at the expense of increased type II error and an inability to see the evidence in the data? $\endgroup$
    – Michael Lew
    Commented Apr 12, 2014 at 21:55
  • 1
    $\begingroup$ Multi-step = pre-testing (equal variances, normality, etc). You're right that failure to satisfy model assumptions will result in worse type II error, but if you entertain non-normality or non-equal variances it is better to just allow for these up front and use a nonparametric test. $\endgroup$
    – Frank Harrell
    Commented Apr 13, 2014 at 2:19
  • $\begingroup$ Are you sure that non-parameteric tests don't perform better when the data are transformed? It seems to me that there are types of data that routinely should be transformed and that transformation has no effect on type I errors. See this: stats.stackexchange.com/questions/27951/… $\endgroup$
    – Michael Lew
    Commented Apr 13, 2014 at 4:09
  • 2
    $\begingroup$ Nonparametric independent group comparisons (unpaired data) are invariant to transformation of $Y$. For methods that require transformations to be "right" (usually, parametric methods) the act of finding the transformations creates tremendous uncertainty that can greatly effect the true coverage of the "final" confidence interval and can affect type I error. This is described in an excellent paper by J Faraway: The Cost of Data Analysis, J Comp Graph Stat 1:213, 1992. $\endgroup$
    – Frank Harrell
    Commented Apr 13, 2014 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.