I have run a probit regression and am now trying to run post-hoc tests. I am trying to compare differences between a 3 level factor variable.

I am confused about the difference between running a 'simultaneous tests for general linear hypotheses' and running the same thing but with a 'Tukey' adjustment- they get very similar answers but is either 'better' or 'worse'? Or does it not matter? For example:

lsmeans(m1, pairwise~Name.Origin, adjust="tukey") 

summary(glht(m1, lsm(pairwise~Name.Origin)))

In addition, is there a more formal name for the first method which just fits the model?

  • $\begingroup$ What are you quoting from? Are you talking about using certain R functions? Can you specify them? $\endgroup$ – gung - Reinstate Monica Aug 16 '15 at 13:51
  • $\begingroup$ Yes, sorry I am talking about using: lsmeans(m1, pairwise~Name.Origin, adjust="tukey") versus using summary(glht(m1, lsm(pairwise ~ Name.Origin))) $\endgroup$ – Lola2000 Aug 16 '15 at 14:34
  • $\begingroup$ (m1 is the probit model that I ran) $\endgroup$ – Lola2000 Aug 16 '15 at 14:36

If you specify adjust = "mvt" in the lsmeans call, you'll get exactly the same results as the glht call (except for minor differences due to the fact that the computations are simulation-based). The difference would come if you summarize the tests in the glht object with some option other than the one-step method (which is the default). The one-step method protects the error rate for simultaneous confidence intervals, which is stronger (and hence more conservative) than the step-down methods. The mvt method is the exact one-step method when the distributional assumptions hold.

Also, in a nicely balanced experiment with homogeneous errors, there is no difference between the Tukey method and the mvt method. That is, the Tukey method is the mvt method for the particular covariance structure encountered in such a balanced design.

  • 1
    $\begingroup$ I previously posted this as an answer to stats.stackexchange.com/q/167464/52554 which is a duplicate. I'm erasing it there because I think that repost should be closed. $\endgroup$ – rvl Aug 18 '15 at 0:26
  • $\begingroup$ Thank you @rvl for your reply. I have deleted the other question. So, in conclusion, which method do you think is better to use or report that you have used? I am still slightly confused as to the current difference between the two functions I have specified. Are you saying that there is essentially no difference in these if I am getting the same results implying that the experiment is balanced with homogenous errors? $\endgroup$ – Lola2000 Aug 18 '15 at 9:16
  • $\begingroup$ The fact that the results are close does NOT imply that the experiment is balanced etc. the arrow goes the other way. The McRae results, which are the same as the one-step glht results, will be appropriate and correct. @frankharrell is saying the same thing. He just doesn't like the terminology. Frank, LS means is an unfortunate term but there are lots of unfortunate terms in common use. They are just equality weighted averages of predictions on a regular grid. $\endgroup$ – rvl Aug 18 '15 at 14:40
  • $\begingroup$ I'm really sorry but as you can probably tell I am not from a stats background so do not know what the 'McRae' results refer to (is that the first formula with adjust=mvt instead of Tukey ?) And if this is the same as the one-step glht results, why can I not just use the glht? $\endgroup$ – Lola2000 Aug 18 '15 at 14:54
  • $\begingroup$ Sorry that is some kind of auto-incorrect glitch. I meant to say the mvt results (!) $\endgroup$ – rvl Aug 18 '15 at 15:06

The silver standard is the simultaneous confidence region from the likelihood profile. A Wald method approximates that (and is exact for the Gaussian case with continuous $Y$). The R multcomp package is an excellent tool. For probit binary or ordinal regression you can use a front-end to it provided by the R rms package that makes it a bit easier, using the rms functions orm (with family='probit') and contrast.rms (with contrast(..., conf.type='simultaneous')).

  • $\begingroup$ Hi Frank, thanks for your reply. However, the functions I ran are lsmeans(m1, pairwise~Name.Origin, adjust="tukey") and summary(glht(m1, lsm(pairwise ~ Name.Origin))) - I am wondering what the differences are between the two and which is 'better' if there exists such a thing. M1 is the probit model I ran - the first is tukey but I was wondering if there was a formal name for the second and which I should use. $\endgroup$ – Lola2000 Aug 17 '15 at 10:53
  • $\begingroup$ They will have different operating characteristics for different combinations of true parameter values but I avoid anything that has the name "least squares means" attached to it. $\endgroup$ – Frank Harrell Aug 17 '15 at 12:45
  • $\begingroup$ Thanks for your reply, I have to admit that I am still slightly confused as both seem to give identical results ...I am not sure how to report the first one or what it is technically called, if anything - so I guess I should just use Tukey? $\endgroup$ – Lola2000 Aug 17 '15 at 13:13
  • $\begingroup$ @FrankHarrell can you tell me what is your specific objection to least-squares means? $\endgroup$ – rvl Aug 18 '15 at 17:34
  • $\begingroup$ (1) Maximum likelihood estimation may have nothing to do with minimizing sum of squared errors; (2) they are hard to interpret; (3) SAS invented this, with bad defaults for how they are estimated. $\endgroup$ – Frank Harrell Aug 18 '15 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.