Comparing main effects and interactions on the same significance scale

Question

When we collect genomics/metabolomics/proteomics data (many variables) in a between-by-within 2-way experimental design, we are often interested in which variables change differently between groups (interaction) and which variables change equally among groups (main effect).

A follow up question after the analysis is whether the overall effect is mostly different between groups or the same among groups i.e. how many significant interactions vs main effects.

However, main effects comparisons are more powerful than interaction tests, so equivalency across groups is more likely to be detected than interactions.

Can anyone recommend an approach to put these tests on the same playing field? Simply use a more stringent p-value (or false discovery rate) for the main effects? How would we choose the different cut-offs?

P.S. I would gladly change the unappealing title of "significance scale" if someone has a better term.

Answer 1

I would abandon the idea of testing this via significance or comparisons of significance. Indeed, I'd abandon the dichotomous nature of "which is more important".

Look at the interactions' effect sizes. Some will be large, some small. Decide, based on substantive grounds, what large and small are. Because all the variables will change differently, it's a question of how differently. You can describe these interactions in various ways.