0
$\begingroup$

When we collect genomics/metabolomics/proteomics data (many variables) in a between-by-within 2-way experimental design, we are often interested in which variables change differently between groups (interaction) and which variables change equally among groups (main effect).

A follow up question after the analysis is whether the overall effect is mostly different between groups or the same among groups i.e. how many significant interactions vs main effects.

However, main effects comparisons are more powerful than interaction tests, so equivalency across groups is more likely to be detected than interactions.

Can anyone recommend an approach to put these tests on the same playing field? Simply use a more stringent p-value (or false discovery rate) for the main effects? How would we choose the different cut-offs?

P.S. I would gladly change the unappealing title of "significance scale" if someone has a better term.

$\endgroup$
0
$\begingroup$

I would abandon the idea of testing this via significance or comparisons of significance. Indeed, I'd abandon the dichotomous nature of "which is more important".

Look at the interactions' effect sizes. Some will be large, some small. Decide, based on substantive grounds, what large and small are. Because all the variables will change differently, it's a question of how differently. You can describe these interactions in various ways.

$\endgroup$
  • $\begingroup$ Your points are well taken, Peter. However, I would still like to incorporate precision into my decisions. I don't think a large but unreliable (wide confidence interval) interaction effect should be given more weight than a smaller interaction effect with low variability. While I know declaring significance has its (perhaps severe) limitations, in the case of high dimensional data, reducing down to simple decisions makes the analysis much more manageable. $\endgroup$ – Moose Jan 22 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.