Summary: I have a population of people split randomly into seven distinct cohorts, amongst whom events occur each month with varying frequency. We have made the same environmental change across all seven cohorts, and I'd now like to see whether the frequencies since that change show a statistically-significant decrease across the population as a whole or not.

The detail: An analogy would be a year of students split randomly into seven differently-sized classes, and the variable I'm looking at is number of detentions received. I'm interested in the picture across the entire school before/after the environmental alteration, rather than changes within individual classes.

However, I'm struggling to work out how to test for that because of the way the data are set out. Points of note are:

  1. The seven cohorts are different sizes.
  2. The people are all part of the same group as a whole, but the data are arbitrarily split into these cohorts [continuing the example above, they're all students, but randomly split into different classes].
  3. The frequency of event differs between cohorts, i.e. one cohort may see on average five events per month, while another might see twenty-five. This does not bear any relation to how the population has been split into these cohorts, but is a (loosely) function of the size of cohort, as you'd expect.
  4. The number of monthly samples for each population before and after the environmental change varies between populations - some have ten months (ten measurements) before and ten after, while some have as few as four after.

Is it possible to do a single test which looks at the impact of the environmental change across all populations? Or would I need to do a test against each population and draw conclusions from those collectively?

If I'm honest, I'm struggling to even work out if the data are distributed normally - I'm not sure whether I should be looking at the frequency of events within a single cohort over time, or the total frequency across all cohorts over time. [I guess they would be normally-distributed across all cohorts, but that's as likely to be because some classes are very large, and some are very small, rather than anything to do with the environmental change - does that matter? Can you even confirm a normal distribution when n=7?]

I'm trying to revive my PhD-level stats, but failing miserably! If anyone could point me in the right direction then I can happily plough through the calculations, I'm just struggling to determine which test is appropriate given the complexity of the data. Many thanks.


A few points:

  1. The Mann-Whitney U test is as nonparametric test. As such, it doesn't rely on a particular distribution. Therefore, whether or not your data is Normally distributed is irrelevant. If your data were Normally distributed, you might have an argument for using a parametric test like a $t$-test, but I would recommend the Mann-Whitney test.

  2. Similarly with the Mann-Whitney test, you do not need balanced samples. You're able to compare larger samples to smaller samples. Though balanced samples leads to greater power, it's not prohibitive (unless, of course, your sample size is very small).

  3. Since it seems as though the grouping into different cohorts is relatively arbitrary and the results you seek are population-wide, I would attempt to pool your results. Pooling your results will cause a bit of data loss - i.e. you'd need to measure each of your cohorts at the same time, so you might only have four months before as your "pre-test" and four months after as your "post-test," but that will allow for the most uniform analysis at the population level.

  4. I might try fitting a regression model and controlling for the effect of each individual cohort. This way, you can isolate the effect of the cohort from the pre-/post-test effect you're hoping to see. This does lend itself to parametric analyses, however, so if small sample sizes is a problem, then this may not be the route to go.

Hope this helps!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.