I have a series of observations that fall into bins (or "scores"); that is, the data can be 0, 1, 2, 3 or 4. There are two groups of such data, control and treated. I know the number of individuals with each score for each group.

What is the best way to determine whether these groups are different or not?

A colleague suggested just arranging the data as individual data points with the given score, and doing the analysis on those two columns of data. Since there are ten individuals per group, this is not difficult, but I do not believe that I am getting a valid answer.


What you are looking for seems to be a test for comparing two groups where observations are kind of ordinal data. In this case, I would suggest to apply a trend test to see if there are any differences between the CTL and TRT group.

Using a t-test would not acknowledge the fact your data are discrete, and the Gaussian assumption may be seriously violated if scores distribution isn't symmetric as is often the case with Likert scores (such as the ones you seem to report). Don't know if these data come from a case-control study or not, but you might also apply rank-based method as suggested by @propfol: If it is not a matched design, the Wilcoxon-Mann-Whitney test (wilcox.test() in R) is fine, and ask for an exact p-value although you may encounter problem with tied observations. The efficiency of the WMW test is $3/\pi$ with respect to the t-test if normality holds but it may even be better otherwise, I seem to remember.

Given your sample size, you may also consider applying a permutation test (see the perm or coin R packages).

Check also those related questions:

  • $\begingroup$ With only 10 observations in each group, ordinal logistic regression is very unlikely to fit, let alone give correct estimates of the variances. I'd suggest the permutation test. $\endgroup$ – Joris Meys Sep 1 '10 at 11:54
  • $\begingroup$ @Joris Good point! I forgot the sample size issue when I initially wrote my answer. The GLM would certainly yields poor estimates... Re-randomization test should be better, you're right. Thanks! $\endgroup$ – chl Sep 1 '10 at 14:26

Three things come to mind:

  1. Contingency table analysis using Fisher's exact test or Chi Square (but will only tell you that somewhere in the table there is a difference that is significant. You'd have to visualize your data or do post-hoc tests to know where this difference is.) Not my preferred solution.
  2. A non-parametric method such the Mann Whitney test. This will rank all of your scores within each group. A good method, but may be underpowered.
  3. A parametric method (such as a t test). Disadvantage is that the assumptions of this method may be violated, especially with such a small sample. Also, the difference between 0 and 1 is not likely to be the same (depending on what you're measuring) as the difference between 3 and 4. The good news is that the t test is relatively robust to the assumptions you are supposed to ensure are true before using the test. However, as I said, the sample size is fairly small.

The best bet may be the Mann Whitney test.

  • $\begingroup$ Thank you very much for that. I thought of the Mann-Whitney test, but many (most?) of the values will result in ties. Won't this reduce the power of the analysis? $\endgroup$ – John Aug 31 '10 at 21:38
  • 2
    $\begingroup$ Yes, of course it will; but as you're working with 10 observations per group, don't expect very much from this setting; check for yourself at statpages.org/#Power $\endgroup$ – chl Aug 31 '10 at 21:53
  • $\begingroup$ @John: I suspect the KS test I proposed would be more powerful than Mann-Whitney because it can detect various alternatives that M-W cannot (such as a change in spread) and otherwise is doing a mathematically similar calculation. This is difficult to study in full generality--in your case, each group's results has four degrees of freedom, so there are eight dimensions of differences to explore--but if you could be more specific about exactly how the two groups might differ (which is matter for science, not statistics, to decide), you would have a better basis for selecting a test. $\endgroup$ – whuber Sep 1 '10 at 15:01

This question is a little unusual because the nature of "different" is unspecified. This response is formulated in the spirit of trying to detect as many kinds of differences as possible, not just changes of location ("trend").

One approach that might have more power than most, while remaining agnostic about the relative magnitudes represented by the five groups (e.g., adopting a multinomial model) yet retaining the ordering of the groups, is a Kolmogorov-Smirnov test: as a test statistic use the size of the largest deviation between the two empirical cdfs. This is easy and quick to compute and it would also be easy to bootstrap a p-value by pooling the two sets of results.

Specifically, let the count in bin $j$ for group $i$ be $k_{ij}$. Then the empirical cdf for group $i$ is essentially the vector $\left( 0 = m_{i0}, m_{i1}, \ldots, m_{i5}=n_i \right) / n_i$ where $m_{i,j} = m_{i-1,j} + k_{ij}, 1 \le i \le 5$. The test statistic is the sup norm of the difference of these two vectors.

Critical values ($\alpha = 0.05$) with two groups of ten individuals are going to be around 0.2 - 0.4, with the higher values occurring when the 20 values are spread evenly between the two extremes.

  • $\begingroup$ Thanks everyone. I went back and tried the Mann-Whitney test, using the method of listing the score of each individual as a column. (FWIW, I use GraphPad Prism for the analysis.) The results do appear reasonable, given what we already know about the biology. Thanks very much for your help. $\endgroup$ – John Sep 1 '10 at 12:15
  • 1
    $\begingroup$ Mann-Whitney can be a good choice, but be aware that it makes some specific assumptions about the relationships among the two groups. It is good when results appear reasonable, but that cannot be the sole reason to adopt a statistical test: that would be getting into a circular argument ("we chose this test because it gives reasonable answers; our answers are significant because this test says so"). At a minimum, make sure to graph all the data in a way that promotes visual comparison of the groups. $\endgroup$ – whuber Sep 1 '10 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy