Let's say I have many ANOVA tests on the same data using different factor variables each time. Then, I want to rank the F-statistics for all of those tests. Notice that I'm not interested in measuring significance (as is usually done with test statistics). Rather, I'm ranking the results.
Is there a concern for multiple comparisons when ranking instead of using significance?
My intuition says yes, there still is a concern. But I can't confidently back it up. It seems to me like the $\alpha$ risk of a Type I error is present each time we rank the results, except it isn't always the same value. So, even thought the value of $\alpha$ isn't always the same, it's still present. Is this correct rationale to believe that multiple comparisons is still in play?
EDIT: I just found in Elements of Statistical Learning, on page 79: "Other more traditional packages base the selection on F -statistics, adding “significant” terms, and dropping “non-significant” terms. These are out of fashion, since they do not take proper account of the multiple testing issues."