Comparing one group to many others

Question

I have a quasi-experiment where I'm comparing one experimental group (A) to many control groups (B1, B2, B3, B4, B5). Since this is a quazi-experiment, I do not know for sure that my control groups all come from the same distribution. I want to know whether some values for the experimental group are significantly different from the control groups. For some values, I hypothesize there will be a difference, and for other that there will not be.

My current approach is to use an omnibus test across all groups (e.g. ANOVA, Kruskal-Wallis) and then post hoc pairwise tests (e.g. Tukey's HSD, Dunn test). This does allow me to detect differences between the groups, but it makes interpretation more difficult. For example, what if A > B1..B3, but there are no differences between A and B4 or B5? Or what is B1 > B2?

Is there a better test for investigating my hypotheses? Or would a modeling approach be more appropriate (e.g. HLM)?

Answer 1

The approach of using post-hoc tests is the correct one. I'm not sure what interpretational difficulties you're having; no contradictions arise from these sorts of tests. Each test is interpreted pairwise, so if you can interpret a t-test you can interpret each of the pairwise tests.

Failing to find a significant difference doesn't mean there isn't a difference; it just means you don't have enough evidence to claim there is a difference. So rejecting some null hypotheses and not others doesn't mean your treatment group is different from some control groups and not others; it means you have evidence to claim it's different from some control groups and lack evidence to claim it's different from others. Lacking evidence just means you can't say anything concrete; it means more data is required to fully answer the specific question.

One occasional potentially confusing finding is that A is significantly different from C, but neither A nor C are significantly different from B. Again, this means you have evidence to claim A is different from C, but not enough evidence to make a claim about whether B is different from A or C. It doesn't mean A is the same as B and B is the same as C and A is different from C, which would be a contradiction.