Is there a concern for multiple comparisons when ranking instead of using significance?

Let's say I have many ANOVA tests on the same data using different factor variables each time. Then, I want to rank the F-statistics for all of those tests. Notice that I'm not interested in measuring significance (as is usually done with test statistics). Rather, I'm ranking the results.

Is there a concern for multiple comparisons when ranking instead of using significance?

My intuition says yes, there still is a concern. But I can't confidently back it up. It seems to me like the $\alpha$ risk of a Type I error is present each time we rank the results, except it isn't always the same value. So, even thought the value of $\alpha$ isn't always the same, it's still present. Is this correct rationale to believe that multiple comparisons is still in play?

EDIT: I just found in Elements of Statistical Learning, on page 79: "Other more traditional packages base the selection on F -statistics, adding “significant” terms, and dropping “non-significant” terms. These are out of fashion, since they do not take proper account of the multiple testing issues."

• What is the purpose of doing this? Deciding, which factors best predict things / ranking them as predictors? There clearly is a multiplicity issue or perhaps rather overfitting issue then and the relative ranking will potentially not be realiable (particularly if some factors are associated with different splits of the total sample size). Certainly, once you get into the realm of wanting to say that any of these factors are relevant (or deciding on this basis how to further analyze this data), you also end up getting a (potentially very severe) traditional type I error problem. May 25, 2016 at 7:28
• Yes, ranking the predictors is the application. This also might simulate what the LASSO regression does, when the predictors are uncorrelated with each other. May 26, 2016 at 15:33
• Okay, to wrap this up, what are the advantages/disadvantages of correcting for multiple testing and NOT correcting for multiple testing? Also, have you seen the section in Elements of Statistical Learning I edited into the question? May 30, 2016 at 23:17