I have a multiple N (>20) variables. I want to test $H_0$ that in at least one pair of them one variable have significantly higher values then the other.

I know that I can perform a bunch of $\frac{N*(N-1)}{2}$ paired t tests with a standard multiple comparison penalty, like Holms. But such generic penalty doesn't account for correlations between the $t$ statistics between the tests, so it would give too conservative results. (I know the correlations are high, because the variables are highly correlated with each other.)

Can I use the multcomp package to do the analysis (and how?).

Or should I use some bootstrap method? I never did resample-based mutliple comparisons.


We are creating a new questionnaire. At the moment it consists of ca. 200 (proposed) items. We have several (15) judges who assess importance of each of the 200 candidate items in 3 level scale Essential, Important and Not very relevant. The scale will be treated as interval.

I need to see, if there are any pairs of questions, who are judged to be significantly different from each other. Later I will run cluster analysis on the results to group them into statistically-meaningful and disjoint groups sorted according to their importance.


1 Answer 1


More info is needed. Why do you have 20+ highly similar variables in your design in the first place? And when you say you just want to know if "at least one" is significant, does that mean you don't need to know which one?

  • $\begingroup$ Question edited. $\endgroup$ Jun 23, 2016 at 12:23
  • $\begingroup$ Seems like you need to fit some sort if mixed model with judges as a random effect, using, e. g., lme (nlme package) or lmer (lme4 package). Subsequently, you might take a look at my lsmeans package for estimating and follow-up comparisons. $\endgroup$
    – Russ Lenth
    Jun 25, 2016 at 1:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.