I got a paper rejected at a conference with the following comment:

The used method for the analysis is correct, but I find it strange that the authors evaluate the results as a 4-condition experiment instead of the 2x2 experiment they set up. This makes their analysis of the interaction effect a bit clumsy, and they end up testing all possible combinations of conditions (which means they should have done a Bonferroni correction). A factorial analysis would have been much more straightforward.

As I do not have the possibility to respond, can anybody explain what the reviewer meant with:

the authors evaluate the results as a 4-condition experiment instead of the 2x2 experiment they set up

A factorial analysis would have been much more straightforward.

Can anybody explain what a factorial analysis is, and how a 4-condition experiment differ from a 2x2 experiment.

My experiment consist of 2x2 within-subjects design. For example A (yes/no) x B (yes/no), where AxB (no/no) is the control condition. I chose to perform a repeated measures logistic regression analyses as I have a:

  • Within-subjects design
  • A four level categorical IV (yes/yes, yes/no, no/yes, no/no)
  • A two level categorical DV (yes/no)

To test significance I contrasted the categories against a reference. I varied the reference to asses differences between categories. I don't see the "clumsiness" the reviewer implies, as this is the way I've learned it using a regression.

| cite | improve this question | | | | |

"Factorial analysis" means analyzing your data with respect to the factors in your model.

Your experimental design has two factors (A and B), each of which has two levels (yes/no). This is usually modeled like this: $$\textrm{logit}(p )=\beta_0 + \beta_1A+\beta_2B + \beta_3(A\cdot B)$$

This corresponds to arranging your data like this enter image description here

and asking whether the row (Factor A, $\beta_1$), column (Factor B, $\beta_2$) or an interaction between them ($\beta_3$) produces differences in your outcome. This makes it easier to interpret your results because you can now argue that A and B are individually important (or not), and likewise for their interaction.

However, for your analysis, it sounds like you "exploded" your two two-level factors into a single factor with four levels and then asked whether different levels of that derived factor $D$ changed your outcome.

The model is something like: $$ \textrm{logit}( p)=\beta_0 + \beta_1D$$

Graphically, it looks like this: enter image description here

All you can say here is that one or more boxes are different from the rest. If you're going to compare them all, you ought to correct for multiple comparisons (but that's another thread).

This makes your result hard to interpret because I don't know whether A (or B) in isolation produces the effect, or if I need to measure both A and B to be sure. Suppose A="is male" and B indicates "is retired". It's much more useful to know if the effect is driven by gender or employment status, or both, rather than some nebulous combination of them.

| cite | improve this answer | | | | |

There are two possibilities here. The reviewer may be overly fixated upon this model due to strong preconceived notions of how such models should be adjusted. Or you did not sufficiently justify the model you presented. Reviewer preferences may be based upon either prior research to which you hadn't consulted thoroughly enough, or an overly critical mentality. In either case, those are beyond the scope of this answer.

Justifying an interaction effect depends highly upon the question at hand and there are circumstances when your model is correct. To explain the differences with models: the reviewer preferred the stratified model

$$\mbox{logit}(P(Y=1|X,W)) = \beta_0 + \beta_1 X + \beta_2 W$$

The interpretation of $\exp \left(\beta_1\right)$ is an odds ratio for the outcome comparing groups having $X=1$ to those having $X=0$ having the same value of $W$. A test of $\beta_1=0$ is analogous to the Mantel-Haenszel stratified odds ratio. Among $X$ and $W$, even if $W$ is experimentally controlled, we usually think of only one (e.g. $X$) variable as a main effect. Stratified models borrow information across values of $W$ and hence control for its influence upon the relationship.

What you fit was equivalent to the interaction model

$$\mbox{logit}(P(Y=1|X,W)) = \beta_0 + \beta_1 X + \beta_2 W + \gamma X W$$

where the interpretation of $\exp \left(\beta_1\right)$ is an odds ratio for the outcome comparing groups having $X=1$ to those having $X=0$ having $W=0$. If $W$ is an experimentally controlled condition, then that main effect has little practical interpretation. A test of $\gamma=0$ would be similar to a test of effect modification. If you simultaneously test $\beta_1, \beta_2, \gamma = 0$ then what you are performing is a generalization of the Pearson $\chi^2$ test for independence in $X$ and $W$. Those two tests are the only justified tests I can conceive of from the interaction model. The test of effect modification is only really justified after examining the main effects from the stratified model.

Testing the y/n vs n/n, n/y vs n/n and y/y vs n/n contrasts is not a sensible way to do such an analysis. It's difficult to interpret their meanings and adjusting for multiple comparisons becomes hairy. (what if say, y/n vs n/n and n/y vs n/n was "significant" with positive association, but y/y vs n/n had negative association?) I would agree it's clumsy.

| cite | improve this answer | | | | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.