3
$\begingroup$

Using R's lsmeans, I'm testing the pairwise comparisons among factor1 levels for each different combination of factor2 and factor3 levels:

lsmeans(model,pairwise~factor1|factor2*factor3,adjust="holm")

(model was produced by lmer).

The last line in the output states:

P value adjustment: holm method for 3 tests

It seems that lsmeans corrects MCP only within each combination of factor2 and factor3, but not across the combinations (In my data there are 14 combinations of factor2 and factor3 and 3 levels within factor1, so the number of total tests is 42).

  1. Is this behavior reasonable? Doesn't it inflate the overall type I error? Why is this the default?
  2. How can one make lsmeans correct across combinations (i.e. for the total number of tests?)
Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
4
$\begingroup$

You can exert more control over what you get if you do the contrasts in a separate step:

lsm = lsmeans(model, ~ factor1 | factor2*factor3)
comps = pairs(lsm)
summary(comps, by = NULL, adjust = "holm")

The by = NULL part of the last statement tells it to ignore the by grouping and thus treat it as one family of tests.

To answer part of the first question, yes, it does inflate the overall type I error, if by that you mean the whole family of 42 tests. On the other hand, I could ask the same question about the results of your one-shot command that includes the least-squares means (I think there are $3\times42=126$ of them) and the 42 pairwise comparisons you requested, for a total of 168 tests: Is it perhaps necessary to protect the overall type I error for that set of 168 tests?

There is always a choice of what family of tests you are applying a multiplicity adjustment to. I am not sure if the per-group adjustment is the most reasonable; I think it is what most users expect, and that is why it is the default. At any rate, one of the design decisions was to try to state explicitly how the adjustments are made, so people don't make the wrong assumption about what they see.

Any comments of whether that is the right default?

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for this helpful answer! It's actually 42 least-squares means, (it's a 3x7x2 three-way design), but you are right that these tests have to be corrected as well. So, more generally, it all comes down to the question of what consists of a 'family' in this context. I saw here on CV decisive opinions in favor of controlling the experiment-wise type I error, but in many scientific fields this is far from the conventional practice. I wonder if someone wrote a paper about this issue. $\endgroup$
    – Trisoloriansunscreen
    Commented Jun 3, 2015 at 9:09
  • $\begingroup$ Duh - of course it's 42 LSmeans. In spite of what I said (a rhetorical point), I personally would NOT do a multiplicity adjustment for all 84 tests. It makes sense to do it for the 42 comparisons of interest -- and in some cases I think it's OK to stay with the 14 families of 3 comparisons, each separately adjusted, as long as it's clearly understood those are conditional inferences. $\endgroup$
    – Russ Lenth
    Commented Jun 3, 2015 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.