4
$\begingroup$

My original thought was “post hoc” in a post hoc multiple comparison procedure means that the procedure is taken after an omnibus test (such as ANOVA F test) gives a significant result.

But after reading Howell's Statistical Methods for Psychology, “post hoc” in a post hoc multiple comparison procedure means the procedure is planned after data collection, and “prior” in a prior comparison means the procedure is planned before data collection.

So was my original thought wrong?

Is Tukey–Kramer method (Tukey's HSD) post hoc?

How about Scheffe method?

A table from Howell's book (here is link to Google Books, although not having the table):

enter image description here

|cite|improve this question
$\endgroup$
8
$\begingroup$

As always, your question implicitly asks for some authoritative answer that might very well not exist. Scheffé's method and Tukey's HSD are usually called post-hoc tests, used for unplanned comparisons and conducted after an omnibus test but that's not a requirement for all such methods.

The main argument for a distinction between planned and unplanned tests is that if you always intended to conduct a limited number of tests (planned contrasts), you don't necessarily need to adjust the error level. If, on the other hand, you are just reporting/testing the differences that look big (post-hoc tests), you might be “capitalizing on chance” and you should adjust not only of the tests you conduct/report but for all possible pairwise comparisons/contrasts in your design.

One issue with all this is that it makes the evaluation of the evidence and the result of a study contingent on the intentions of the experimenter, a most counter-intuitive and undesirable state of affairs. This is sometimes held as an argument against null-hypothesis significance testing as used within psychology.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks! Besides Scheffe's and Tukey's methods, is any multiple comparison method able to apply as either prior and post hoc? $\endgroup$ – Tim Jul 9 '13 at 12:10
  • $\begingroup$ Also in Table 12.8 in Howell's book, he listed both Scheffe's and Tukey's methods as post hoc. Do you know why not prior there? $\endgroup$ – Tim Jul 9 '13 at 12:22
  • 1
    $\begingroup$ (A) I didn't mean to imlpy that Tukey's HSD and Scheffé's method are not appropriate for post-hoc tests, quite the contrary. (B) I feel I have given a good description of why people make a difference between planned and unplanned tests (as good as I could at any rate) but I don't have Howell's book before me right now and I cannot comment on every word and every sentence of every book on the topic. Even if I did disagree, it could also be a slip or typo, a non-standard definition or something that makes sense in a specific context. It's best to focus on substantive issues, not text commentary. $\endgroup$ – Gala Jul 9 '13 at 12:28
  • $\begingroup$ Thanks. About (A), what I intended to ask is: is it correct, any multiple comparison method (Scheffe's, Tukey's, or any other) can be used as both prior and post hoc, unlike what is mentioned the table in Howell's book? (B) I added link to his book and a picture of the table in my post. Thanks! $\endgroup$ – Tim Jul 9 '13 at 12:33
  • 1
    $\begingroup$ It says “most commonly” and “primarily” under the table. There you have it: It's an introductory book geared toward applied researchers providing some practical guidance not a definitive theoretical treatment of these tests. $\endgroup$ – Gala Jul 9 '13 at 12:41
6
$\begingroup$

Your original thought was incorrect. Howell's is correct.

Take the simple t-test, some people just use them for planned or post hoc but adjust their p-values for multiple comparisons.

Both of the tests you mention are typically used post-hoc but could be used for planned tests if the planned tests are expected to have multiple comparison issues. For example, in an ANOVA situation where you do planned contrasts after the ANOVA you may want to test non-orthogonal or simply recognize that there are still problems with multiple comparisons. In that case you might use a traditionally post-hoc procedure.

Also, the conclusions you can reach regarding the two kinds of testing are quite different. A post-hoc test allows you to draw tentative conclusions to guide further research while the planned test speaks directly to the theories that you're examining.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks, but what do you mean by "planned" tests? In Howell's book, he contrasts "post hoc" comparisons to "prior" comparisons, and both are planned either after or before data collection. Also do you think Howell's is correct? $\endgroup$ – Tim Jul 8 '13 at 16:07
  • $\begingroup$ prior = planned...Howell's is addressed in the answer. $\endgroup$ – John Jul 8 '13 at 16:09
2
$\begingroup$

Many people--as well as SPSS--do in fact use the term "post hoc" the way you initially described. Howell's usage may be more common. But rather than argue about which definition is "right," the important thing is to know that when you see the term used it may mean different things. Becase it is so inconsistently used, it's probably best to avoid using the term yourself unless it is absolutely clear what you mean. This problem is discussed in Frane, 2015, "Planned Hypothesis Tests Are Not Necessarily Exempt From Multiplicity Adjustment"

Incidentally, Howell's classification of given multiple comparison procedures as being strictly for a priori or unplanned tests appears to be rather arbitrary.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.