4
$\begingroup$

Students were split into 2 groups and were both tested on their knowledge of spaced items and massed items.

The research question is testing the difference of the difference, i.e. the difference between groups in the difference between learning spaced vs massed words.

I performed glmer(Posttest ~ group × spacing + (1|subject) + (1|item))

    subject   Item    Group   Spacing
      S1       1      G1      spaced 
      S1       2      G1      spaced
      S1       3      G1      massed
      S1       4      G1      massed
      S2       1      G1      spaced 
      S2       2      G1      spaced
      S2       3      G1      massed
      S2       4      G1      massed
      S3       1      G2      spaced 
      S3       2      G2      spaced
      S3       3      G2      massed
      S3       4      G2      massed
      S4       1      G2      spaced 
      S4       2      G2      spaced
      S4       3      G2      massed
      S4       4      G2      massed
    

Result: The effect of spacing depended on group (χ²(1) = 9.71, P = .002)

                   Posttest   

       Predictors           Odds Ratios           CI      p
                  
       (Intercept)             0.22         0.07 – 0.67 0.008
       group [G2]              2.74         1.26 – 6.00 0.011
       spacing [massed]        0.81         0.34 – 1.93 0.632
       group [G2]*spacing      0.45         0.27 – 0.74 0.002
       [massed]                

BUT no example found in articles for reporting odds ratio when both variables are categorical. Is my attempt below correct?

The odds of a correct response for spaced items compared to massed items were twice as high in Group2 compared to Group1 (OR = 1/Exp(B) = 1/0.45 = 2.22, 95% CI [0.27, 0.74]). Additional analysis in which massed items were taken as reference category confirmed that the reverse was true for G2 group.

$\endgroup$
3
$\begingroup$

You have two questions. The first question is in your title.

How to report odds ratio for interaction between 2 categorical variables?

The answer will depend on how many levels these variables have. I am glad that your both variables only have two levels, because this makes things a bit easier. So you have a two by two matrix where all observations fit.

| Group\Spacing | Spaced              | Massed              |
|---------------|---------------------|---------------------|
| Group 1       | Group1_Spaced (25%) | Group1_Massed (25%) |
| Group 2       | Group2_Spaced (25%) | Group2_Massed (25%) |

The resulting distribution of the observations in these four groups, would be equivalent as if you had only one variable with four levels.

| Group_Spaces | % of cases |
|--------------|------------|
| G1_spaced    | 25%        |
| G2_spaced    | 25%        |
| G1_massed    | 25%        |
| G2_spaced    | 25%        |

The interpretation of each of the constituent parts of the interaction term will also be equivalent to what you would obtain if you add only that hypothetical variable with four levels. That is, you would have a dummy indicator variable for each level, and one of the levels (usually the first) would have to be dropped off from the model (to avoid perfect multicollinearity). The dropped level would be the reference category for all other coefficients. So, you would have odds ratios as below:

| Group_Spaces | Odds Ratio |
|--------------|------------|
| G1_spaced    | (omitted)  |
| G2_spaced    | 2.74       |
| G1_massed    | 0.81       |
| G2_spaced    | 0.45       |

It is important to remember that all coefficients are measuring the average effect of changing levels related to omitted level, that is to "G1_spaced".

Your second question:

Is my attempt below correct?

The odds of a correct response for spaced items compared to massed items were twice as high in Group2 compared to Group1 (OR = 1/Exp(B) = 1/0.45 = 2.22, 95% CI [0.27, 0.74]). Additional analysis in which massed items were taken as reference category confirmed that the reverse was true for G2 group.

I don’t think so. From your statement ("The odds of a correct response for spaced items compared to massed items were twice as high in Group2 compared to Group1"), it sounds to me like you are comparing the odds of:

$$ \frac{Group2\_spaced \div Group2\_massed}{Group1\_spaced \div Group1\_massed} $$

However, from your calculations, you are comparing the odds of:

$$ \frac{Group1\_spaced}{Group2\_massed} = \frac{1}{0.45} = 2.22 $$

If you truly wanted the first option (which I assume you stated), then it would be:

$$ \frac{2.74 \div 0.45}{1 \div 0.81} = 4.932 $$

I think the statement you are trying to make is a bit confusing, because you are trying to compare four different categories with just one number. Mathematically, it can be done, but it is hard for our human mind to understand. So perhaps, it will work better if you compare two groups at each time. From the above table, you may say:

  • Students of Group 2 have 2.74 times (or 174%) higher odds of a correct answer in spaced items than students of Group 1.
  • Students of Group 1 have 1.8 times (or 80%) higher odds of a corrected answer in massed items than students of Group 2 (Given by 0.81/0.45).
  • Students of Group 1 have 23% (or 1.23 times) higher odds of a correct response in spaced items than in massed items (Given by 1/0.81).
  • Students of Group 2 have 6.08 times higher odds of a correct response in spaced items than in massed items (Given by 2.74/0.45)

Getting the odds and the probabilities:

In odds ratio everything is relative: 2 times more may sound a lot. But if the actual odds are 1%, then it is just a 1% increase. If the actual odds are 50%, then it is a 50% increase. Our minds do not have a good reference when working with odds ratio. They are non-intuitive and hard to interpret.

In the above calculations, We considered that the odds-ratio of the reference category is 1. This makes sense, as it is related to itself. However, if both factors are coded as dummies (0 and 1), then the intercept would be equivalent to the actual odds of the reference category. This is because, as you only have these two variables, and the intercept is the expected value when all variables are equal to zero. Therefore, I would strongly suggest you coding your factor variables this way.

If this is already the case, then you could work with the actual odds of getting each item type correct for each group:

| Group\Spacing | spaced               | massed               |
|---------------|----------------------|----------------------|
| Group 1       | 0.22                 | 0.1782 (0.22 x 0.81) |
| Group 2       | 0.6028 (0.22 x 2.74) | 0.099 (0.22 x 0.45)  |

Perhaps this numbers are easier to understand. We can see that those highly different odds ratios were all referring to actual odds lower than 1. Which means that every group has less than 50% average probability of getting the items correct.

For me, even better and more intuitive would be working with average predicted probabilities. Assuming that your factors are already coded as 0 and 1, then you could calculate your average predicted probabilities also from your coefficients and intercept:

| Group\Spacing | spaced            |  massed              |
|---------------|-------------------|----------------------|
| Group 1       | 18.0% (0.22/1.22) |  15.1% (0.178/1.178) |
| Group 2       | 37.6% (0.60/1.60) |  9% (0.099/ 1.099)   |

The advantage of working with probabilities is that it also allows you to work in additive terms and have a better idea of absolute values. If you plot the averages predicted probabilities, which is the current best practice for logistic regressions, you will make much easier to see the interaction effect (probably the main reason for adding the interaction term). In group 2, item type has a huge effect, while in group 1, it has a small effect (perhaps not even statistically significant, if you ask R to compute these values).

Additional comments:

It would be worth noticing that you are running a multilevel logistic regression model, with two cross-classified second levels. I don’t know how many items or subjects you have, but if you do have only four items, it might be the case that you get a better model fit without specifying item as a second level.

$\endgroup$
  • $\begingroup$ Your interpretation does not fit my research question which is: comparing the difference of difference. i.e. compare the difference between groups in their spacing advantage. so your answer is not applicable unfortunately since it does not not serves this aims. $\endgroup$ – Acer acer Nov 8 '20 at 9:50
  • $\begingroup$ For number of items and subjects its 16 and 110, respectively. $\endgroup$ – Acer acer Nov 8 '20 at 9:53
  • $\begingroup$ @Acer, If you want to compare differences of difference, then I would strongly recommend you to work with predicted probabilities. Because they are ready to be computed in additive terms. Odds ratio are multiplicative, so it is a bit confusing to work with them in terms of differences in differences. For example, with probabilities, you could say: Group 2 have a 28.6 p p. increase in probabilities of getting corrected spaced items compared to massed items, while for Group 1 this increase was of 2.9 p.p. $\endgroup$ – LuizZ Nov 9 '20 at 1:55
  • $\begingroup$ I am not sure you are in some sort of experimental setting, but for differences-in-differences approach, I think you would need also before and after measurements $\endgroup$ – LuizZ Nov 9 '20 at 2:01
  • $\begingroup$ Maybe that's why researchers prefer not to report odds ratio in this case, because it is confusing. $\endgroup$ – Acer acer Nov 9 '20 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.