How to test group differences on a five point variable?

Question

I have a series of observations that fall into bins (or "scores"); that is, the data can be 0, 1, 2, 3 or 4. There are two groups of such data, control and treated. I know the number of individuals with each score for each group.

What is the best way to determine whether these groups are different or not?

A colleague suggested just arranging the data as individual data points with the given score, and doing the analysis on those two columns of data. Since there are ten individuals per group, this is not difficult, but I do not believe that I am getting a valid answer.

Answer 1

What you are looking for seems to be a test for comparing two groups where observations are kind of ordinal data. In this case, I would suggest to apply a trend test to see if there are any differences between the CTL and TRT group.

Using a t-test would not acknowledge the fact your data are discrete, and the Gaussian assumption may be seriously violated if scores distribution isn't symmetric as is often the case with Likert scores (such as the ones you seem to report). Don't know if these data come from a case-control study or not, but you might also apply rank-based method as suggested by @propfol: If it is not a matched design, the Wilcoxon-Mann-Whitney test (wilcox.test() in R) is fine, and ask for an exact p-value although you may encounter problem with tied observations. The efficiency of the WMW test is $3/\pi$ with respect to the t-test if normality holds but it may even be better otherwise, I seem to remember.

Given your sample size, you may also consider applying a permutation test (see the perm or coin R packages).

Check also those related questions:

• Group differences on a five point Likert item
• Under what conditions should Likert scales be used as ordinal or interval data?

Answer 2

This question is a little unusual because the nature of "different" is unspecified. This response is formulated in the spirit of trying to detect as many kinds of differences as possible, not just changes of location ("trend").

One approach that might have more power than most, while remaining agnostic about the relative magnitudes represented by the five groups (e.g., adopting a multinomial model) yet retaining the ordering of the groups, is a Kolmogorov-Smirnov test: as a test statistic use the size of the largest deviation between the two empirical cdfs. This is easy and quick to compute and it would also be easy to bootstrap a p-value by pooling the two sets of results.

Specifically, let the count in bin $j$ for group $i$ be $k_{ij}$. Then the empirical cdf for group $i$ is essentially the vector $\left( 0 = m_{i0}, m_{i1}, \ldots, m_{i5}=n_i \right) / n_i$ where $m_{i,j} = m_{i-1,j} + k_{ij}, 1 \le i \le 5$. The test statistic is the sup norm of the difference of these two vectors.

Critical values ($\alpha = 0.05$) with two groups of ten individuals are going to be around 0.2 - 0.4, with the higher values occurring when the 20 values are spread evenly between the two extremes.

Answer 3

Three things come to mind:

1. Contingency table analysis using Fisher's exact test or Chi Square (but will only tell you that somewhere in the table there is a difference that is significant. You'd have to visualize your data or do post-hoc tests to know where this difference is.) Not my preferred solution.
2. A non-parametric method such the Mann Whitney test. This will rank all of your scores within each group. A good method, but may be underpowered.
3. A parametric method (such as a t test). Disadvantage is that the assumptions of this method may be violated, especially with such a small sample. Also, the difference between 0 and 1 is not likely to be the same (depending on what you're measuring) as the difference between 3 and 4. The good news is that the t test is relatively robust to the assumptions you are supposed to ensure are true before using the test. However, as I said, the sample size is fairly small.

The best bet may be the Mann Whitney test.