Choosing the right test when comparing two groups: Mann-Whitney, Kolmogorov-Smirnov or other?

I need help, since my skills and knowledge in statistics are really limited. Using Pearson correlation I found that there is some relationship between a binary one and a second variable taking values between 1 and 4. So now I want to understand if there are significant differences between the two groups defined by the binary variable.

The 2 variables are the following:

X = (0 0 0 0 0 0 0 0 0 0 1 1 1 1)
Y = (4 3 4 3 3 3 3 3 3 3 4 4 4 4)


To determine if there's a significant difference between $Y_{X=0}$ and $Y_{X=1}$, I used a Mann-Whitney test. Was this the correct choice? Would it have been preferable to use a K-S test? And why not a t-test?

I am actively looking for an answer to these questions.

-
– user900978 Mar 21 '12 at 12:30
I've made significant edits to your post. Please check carefully that I didn't alter its original meaning. – chl Mar 21 '12 at 19:23
It is noteworthy that the answers reflect two very different interpretations of these data, so a clarification is in order. The reply by Matt Parker (and also the edit by @chl) views $X$ as indicating two groups (0 and 1) and takes the question as a request to compare the $Y$ values between the groups. The reply by Glen_b takes the question as a request to compare the 14 values of $X$ to the 14 values of $Y$. Which interpretation is the intended one? – whuber Mar 21 '12 at 19:42
Thanks to Chi for his edits, now my question seems really clear. The right interpretations is the first, Matt and Chi caught the idea in my mind. – user900978 Mar 22 '12 at 13:55

I think the Mann-Whitney/Wilcoxon ranked-sum test is the appropriate test. The KS test is specifically for comparing continuous distributions - your ratings are ordinal, so it doesn't seem appropriate here.

The t-test and the Wilcoxon ranked-sum differ in that the t-test is comparing the means of the two distributions, while the Wilcoxon is comparing the 'locations' by looking at how the values of the two distributions compare when ranked. When your entire ratings distribution has only two values, one group has only ratings of 4 and your sample size is 14, the t-test seems less appropriate to me. It works, but I just have a harder time with it conceptually. This data is more binomial than it is continuous!

Here's how I'd do all of that in R, which is a freely available software for statistical computing (a step up from using websites to compute tests, I think...)

# A vector of data for people with smartphone experience
smartphone <- c(4, 4, 4, 4)

# A vector of data for people without smartphone experience
dumbphone <- c(4, 3, 4, 3, 3, 3, 3, 3, 3, 3)

# The Mann-Whitney/Wilcoxon ranked-sum test
wilcox.test(x = smartphone, y = dumbphone)

# t-test for comparison
t.test(x = smartphone, y = dumbphone)

# And, why not, a test of proportions
# Consider 4 as the event, comparing 4/4 to 2/10
prop.test(x = c(4, 2), n = c(4, 10))

-
 Hi Matt, really tnaks for your help. There's less fog on my mind now! – user900978 Mar 22 '12 at 13:21

Plainly there's a difference between the groups - the locations are completely different, with no overlap in the groups. Any sensible test will reject even at this small sample size.

I assume you want to ask a different question of the data than that.

-
At he following link: elegans.som.vcu.edu/~leon/stats/utest.cgi i found that with Mann-Whitney thre's a marginally significant difference between the first 10 elements of Y and the last 4. However, my question is: is Mann-Whitney the right test to try for this sampale? Is K-S better? and the T-test? I have some difficult with statistical tests.. – user900978 Mar 21 '12 at 12:33
@user900978 - Please be clearer about exactly which difference you want to test for significance. Apparently you don't want to test the difference between X and Y, as I think Glen_b (and I) initially thought. If you are testing the first 10 values of Y vs. the last 4, then it is important whether you are doing so based on some prior hypothesis or merely because, after collecting the data, that is where you have noticed the greatest difference. If the latter, any statistical test will be clouded by the "data snooping" that you have done. – rolando2 Mar 21 '12 at 12:55
Tnaks for your answers. Excuse me if I explain bad my questions, I'm not so used to speak in English. However, I need to find if there is the first 10 values of Y are significantly different from the last 4 values of Y. The first 10 values are ratings provided by people who hasn't experience with smartphones, the last 4 from people having experience with smartphones. – user900978 Mar 21 '12 at 13:50
If your decision to test this particular division is based on looking at the data, your test p-values are meaningless (at the least, way too small). – Glen_b Feb 17 at 3:22

You should t-test the mean difference between smart phone users and non-smart phone users. Pearson's R is not appropriate for binary variables because it assumes that both variables are normally distributed (and X, in your case, cannot be because it is dichotomous).

A t-test will ask if the mean score on the outcome (Y in this case) is significantly different across the two categories of X. This assumes that your Y variable is something that is appropriate for averaging. In other words it should at least be ordinal (meaning rank ordered categories) with 5 or so possible categories, an interval level of measurement is technically required but, in practice, ordinal is fine.

There are several statistical tests and correlation coefficients that can be calculated and what you choose will probably be a function of whatever the convention is in your field for the sort of data you have. The t-test is pretty common across all fields, that's why I suggest it.

Note also that the t-test doesn't give you an effect size - you are comparing the means of the two groups under the null hypothesis that the difference between them is 0 in the population. A statistically significant finding means only that the means are not the same in the population. You might also consider some effect size statistic - that's what Pearson's R is, but again that's not appropriate for binary variables.

-
 I wanted to add that the Mann-Whitney test is essentially the same as the t-test but assumes that the Y variable is ordinal, rather than interval. In practice the results will be the same - the difference between the groups will be significant at the .05 level. – Will Mar 21 '12 at 16:19 Will, thanks for your answer. You have been really clear. And what about Kolmogorov-Smirnov? If I right understood, it gives similiar results, but is preferred to Mann-Whitney for small samples. Is this right? – user900978 Mar 21 '12 at 16:44 I'm not familiar with the K-S test but after a bit of googling, it looks to me to be a test for comparing the distribution of a continuous variable across two groups or samples. This would not be appropriate for your data. For starters, one group consists of only 4 cases and there is no 'distribution' because they all have the same value (4). More generally, if your data is as you describe (i.e. 10 cases) then there are going to be minimal conclusions you can draw from it, regardless of how thoughtfully you design your analysis. – Will Mar 22 '12 at 15:17

Another option is Pearson's Chi-square test, which is appropriate for categorical variables. You can use it to test whether the variable Y is independent of X, in other words, whether Y has any influence on X. However, you would need a sample larger than 10 values, and also I'm not sure if this test can be used with a 2x2 contingency table... I seem to recall there was a minimum number of cells.

-
Hmm...We're always glad of contributions, but you seem so uncertain, perhaps you should do a little research before committing to a reply? – whuber Mar 21 '12 at 20:27
I'm uncertain about the calculation details, but I'm pretty certain that the test could be useful in this case. – Ernest A Mar 21 '12 at 20:57