Choosing the right test when comparing two groups: Mann-Whitney, Kolmogorov-Smirnov or other?

Question

I need help, since my skills and knowledge in statistics are really limited. Using Pearson correlation I found that there is some relationship between a binary one and a second variable taking values between 1 and 4. So now I want to understand if there are significant differences between the two groups defined by the binary variable.

The 2 variables are the following:

X = (0 0 0 0 0 0 0 0 0 0 1 1 1 1)
Y = (4 3 4 3 3 3 3 3 3 3 4 4 4 4)

To determine if there's a significant difference between $Y_{X=0}$ and $Y_{X=1}$, I used a Mann-Whitney test. Was this the correct choice? Would it have been preferable to use a K-S test? And why not a t-test?

I am actively looking for an answer to these questions.

Answer 1

I think the Mann-Whitney/Wilcoxon ranked-sum test is the appropriate test. The KS test is specifically for comparing continuous distributions - your ratings are ordinal, so it doesn't seem appropriate here.

The t-test and the Wilcoxon ranked-sum differ in that the t-test is comparing the means of the two distributions, while the Wilcoxon is comparing the 'locations' by looking at how the values of the two distributions compare when ranked. When your entire ratings distribution has only two values, one group has only ratings of 4 and your sample size is 14, the t-test seems less appropriate to me. It works, but I just have a harder time with it conceptually. This data is more binomial than it is continuous!

Here's how I'd do all of that in R, which is a freely available software for statistical computing (a step up from using websites to compute tests, I think...)

# A vector of data for people with smartphone experience
smartphone <- c(4, 4, 4, 4)

# A vector of data for people without smartphone experience
dumbphone <- c(4, 3, 4, 3, 3, 3, 3, 3, 3, 3)

# The Mann-Whitney/Wilcoxon ranked-sum test
wilcox.test(x = smartphone, y = dumbphone)

# t-test for comparison
t.test(x = smartphone, y = dumbphone)

# And, why not, a test of proportions
# Consider 4 as the event, comparing 4/4 to 2/10
prop.test(x = c(4, 2), n = c(4, 10))

Answer 2

Plainly there's a difference between the groups - the locations are completely different, with no overlap in the groups. Any sensible test will reject even at this small sample size.

I assume you want to ask a different question of the data than that.

Answer 3

You should t-test the mean difference between smart phone users and non-smart phone users. Pearson's R is not appropriate for binary variables because it assumes that both variables are normally distributed (and X, in your case, cannot be because it is dichotomous).

A t-test will ask if the mean score on the outcome (Y in this case) is significantly different across the two categories of X. This assumes that your Y variable is something that is appropriate for averaging. In other words it should at least be ordinal (meaning rank ordered categories) with 5 or so possible categories, an interval level of measurement is technically required but, in practice, ordinal is fine.

There are several statistical tests and correlation coefficients that can be calculated and what you choose will probably be a function of whatever the convention is in your field for the sort of data you have. The t-test is pretty common across all fields, that's why I suggest it.

Note also that the t-test doesn't give you an effect size - you are comparing the means of the two groups under the null hypothesis that the difference between them is 0 in the population. A statistically significant finding means only that the means are not the same in the population. You might also consider some effect size statistic - that's what Pearson's R is, but again that's not appropriate for binary variables.

Answer 4

Another option is Pearson's Chi-square test, which is appropriate for categorical variables. You can use it to test whether the variable Y is independent of X, in other words, whether Y has any influence on X. However, you would need a sample larger than 10 values, and also I'm not sure if this test can be used with a 2x2 contingency table... I seem to recall there was a minimum number of cells.