0
$\begingroup$

Let's say I have many ANOVA tests on the same data using different factor variables each time. Then, I want to rank the F-statistics for all of those tests. Notice that I'm not interested in measuring significance (as is usually done with test statistics). Rather, I'm ranking the results.

Is there a concern for multiple comparisons when ranking instead of using significance?

My intuition says yes, there still is a concern. But I can't confidently back it up. It seems to me like the $\alpha$ risk of a Type I error is present each time we rank the results, except it isn't always the same value. So, even thought the value of $\alpha$ isn't always the same, it's still present. Is this correct rationale to believe that multiple comparisons is still in play?

EDIT: I just found in Elements of Statistical Learning, on page 79: "Other more traditional packages base the selection on F -statistics, adding “significant” terms, and dropping “non-significant” terms. These are out of fashion, since they do not take proper account of the multiple testing issues."

$\endgroup$
3
  • $\begingroup$ What is the purpose of doing this? Deciding, which factors best predict things / ranking them as predictors? There clearly is a multiplicity issue or perhaps rather overfitting issue then and the relative ranking will potentially not be realiable (particularly if some factors are associated with different splits of the total sample size). Certainly, once you get into the realm of wanting to say that any of these factors are relevant (or deciding on this basis how to further analyze this data), you also end up getting a (potentially very severe) traditional type I error problem. $\endgroup$
    – Björn
    May 25, 2016 at 7:28
  • $\begingroup$ Yes, ranking the predictors is the application. This also might simulate what the LASSO regression does, when the predictors are uncorrelated with each other. $\endgroup$ May 26, 2016 at 15:33
  • 1
    $\begingroup$ Okay, to wrap this up, what are the advantages/disadvantages of correcting for multiple testing and NOT correcting for multiple testing? Also, have you seen the section in Elements of Statistical Learning I edited into the question? $\endgroup$ May 30, 2016 at 23:17

1 Answer 1

-1
$\begingroup$

If you are simply ranking the tests for the purposes of deciding which comparisons are of interest, then there is no possibility of a false positive decision and so type I error rates are not a relevant consideration.

If you are ranking the contrasts in order to make claims about some of them then some people would suggest that 'correction' for multiplicity is appropriate. Note, however, that there are many who do not consider adjustments for multiplicity to be 'corrections' so much as nonsense (see these http://www.bio.sdsu.edu/pub/stuart/2012LopsidedReasoning.pdf and http://www.stat.columbia.edu/~gelman/research/published/multiple2f.pdf).

$\endgroup$
8
  • 1
    $\begingroup$ How would it not be a problem to simply rank "the tests for the purposes of deciding which comparisons are of interest, then there is no possibility of a false positive decision and so type I error rates are not a relevant consideration." ? Surely, you are then potentially conducting all the possible comparisons, but are only picking out the "most signficant" and then doing them? Surely that's the very problem one should be concerned about. $\endgroup$
    – Björn
    May 25, 2016 at 7:23
  • $\begingroup$ @Björn In general one should be concerned about making correct inferences, not just with protecting against inflation of type I errors above an arbitrary nominal rate. There certainly are some circumstances where one might consider correction for multiple comparisons, but those circumstances are not those that pertain in an exploratory study intended to rank questions of interest for future investigation. If you don't want to read the full papers I linked, then try my short commentary in the supplementary material here: tandfonline.com/doi/suppl/10.1080/00031305.2016.1154108 $\endgroup$ May 25, 2016 at 21:31
  • $\begingroup$ okay.....now I'm thoroughly confused. $\endgroup$ May 27, 2016 at 3:43
  • $\begingroup$ @Björn Buzzinolops, sorry, but I'm not surprised that you are confused. As far as I am concerned the confusion is a product of discussion between those who believe that multiplicity of comparisons should always be 'corrected' for in order to protect against inflation of type I errors, and those, like me, who feel that protection against inflated type I error rates is important in only some circumstances but always presents costs in terms of power and flexibility of design. It's a tradeoff, but not everyone sees the tradeoff as optional. I assume Björn is among the latter. $\endgroup$ May 27, 2016 at 3:52
  • $\begingroup$ All I am saying is that there is definitely a multiplicity problem. The more things you rank, the more likely it is that you get at least some of the rankings wrong. Whether you you need something like stricty familywise type I error rate control is a totally different topic, of course. However, one cannot have the cake and eat it. I.e. pick out the most "significant" contrast and treat it like the sole contrast in a pre-specified hypothesis testing situation (equivalent to claiming that because the F-statistics are sufficiently apart between this factor a and factor b to be significant). $\endgroup$
    – Björn
    May 27, 2016 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.