# Accounting for an unexpected predictor in post-hoc Kruskal-Wallis tests

Assume a set of ten simulated frequency distributions that are contingent on an independent variable that reflects two treatment groups: Nine of the distributions represent treatment group 1 (in blue in Fig.1), while the tenth represents treatment group 2 (in red in Fig.1). All ten distributions have the same sample size, but none is normally distributed.

Visually I confirm that mean and median of the single frequency distribution of treatment group 2 is strongly different than means and medians of the other distributions, which do display slight variation among each other (Fig.1).

To evaluate the omnibus H0, I perform a Kruskal-Wallis test, which allows me to reject the omnibus H0. To conduct individual comparisons between the distributions, I perform pairwise Kruskal-Wallis tests in the post-hoc framework of Nemenyi. These post-hoc tests indicated rejection of H0 between distribution of treatment group 2 to any of the other distributions. However, H0 can also be rejected in some of the comparisons within treatment group 1, indicating the presence of another explanatory variable.

What strategy (e.g., test design, statistical framework, etc.) would you employ to compensate for this second, unrecognized explanatory variable. All I am interested in is to test H0: distributions [plural!] of treatment group 1 identical to distribution of treatment group 2.

[More specific details about what the subgroups in treatment 1 represent should result in better-targeted responses.]

I'm not sure that any "compensation" is necessary; your post-hoc identified all the treatment2-vs-treatment1 differences you're interested in.

The important part of the story is completely illustrated in your display; clearly treatment 2 is very different from treatment 1. There may be some very small differences between groups within in treatment 1.

Indeed if you weren't interested in within-treatment-1 differences, one is left to wonder why you would test for them.

The rejection of the null of no difference within some of the treatment 1 group pairs doesn't automatically mean there's some other explanatory variable (Type I errors can happen, for example), though in very large samples, you would be able to identify that at least the differences between two apparent sub-groups may be more than chance:

Assuming that's a true difference, the question would be 'how much does that matter'?

Certainly it doesn't impact the tests for pairwise differences between treatment 1 subgroups and treatment 2. The next consideration is how important those relatively small differences are within the treatment-1 groups (i.e. how important are they?) -- statistical significance is not the same as practical importance.

If that small difference is real but not of interest, you could consider setting up a single contrast a priori that reflects what you are interested in, but it depends on how (a priori) you see such differences as arising.

How you view potential differences within the treatment1 groups may make a difference to what analyses you consider. For example, if you see the effects as due to some unspecified blocking factor (e.g. imagine you were treating some unusual disease that was only treated at a handful of different treatment centers, and those groups represent each of those treatment centers), or some uncontrolled but important variable, then you'd probably simply do as you have already done - account for that factor in the analysis.

However, in a situation where not all important variables are controllable, and you regard the resulting variations in group-locations in treatment 1 as being random selections from a population of other possible values which you want to extend your inference over, then you might look at random effects type models.