Examine significance of the effect of an attribute on the effect of another attribute

Question

I'd like to examine the effect of an attribute on the effect of another attribute on certain behaviour. To explain what I mean:

I tested two groups of 110 people on whether they did x. One of the groups had the attribute A, the other did not have A:

|---------------------|------------------|------------------|
|                     |       did x      |     didn't do x  |
|---------------------|------------------|------------------|
|     group 0         |         1        |      109         |
|---------------------|------------------|------------------|
|     group A         |         8        |      102         |
|---------------------|------------------|------------------|

I also tested x with two more groups, both of which have another attribute B, which is independent from attribute A. Again, one group had A, the other did not.

|---------------------|------------------|------------------|
|                     |       did x      |     didn't do x  |
|---------------------|------------------|------------------|
|     group B         |         7        |      103         |
|---------------------|------------------|------------------|
|     group BA        |         2        |      108         |
|---------------------|------------------|------------------|

When the subjects had the attribute B, attribute A appears to have a different (opposite) effect on whether subjects did x. I would like to test whether this difference in the effect is statistically significant. Which test could I use to do that?

In case it's relevant: The significance of each table alone is determined with Fisher's exact test (first is significant with p < 0.05, second is not). Doing x and not doing x is mutually exclusive.

I am new to statistics, simple terms and explanations would be appreciated.

Answer 1

For the most general approach, what you want to do is combine the data into one model. You can think of these data as having a dependent variable (Did / Didn't) and a couple of independent variables (0/B and A/not-A). You could model these data with logistic regression. The question you are asking is if there is an interaction between 0/B and A/not-A.

There are traditional tests for this question for 2 x 2 x k tables. Namely, the Woolf test and Breslow-Day test.

Because you have low counts in some cells (e.g. 1, 2), these approaches might be deficient.

The following example uses R. It could be run at rdrr.io/snippets.

To summarize the results here, as you suspected, there is a significant interaction of the effect of (A/not-A) with (0/B) (from Woolf test, Breslow-Day test, or the interaction p in the Anova results). (You'll note that the results of the Woolf test agree precisely with those from the Wald test of the interaction in the Anova table.)

### Install packages and read data

if(!require(vcd)){install.packages("vcd")}
if(!require(DescTools)){install.packages("DescTools")}
if(!require(car)){install.packages("car")}
if(!require(emmeans)){install.packages("emmeans")}
if(!require(ggplot2)){install.packages("ggplot2")}

Data = read.table(header=T, text="
Group1 Group2     X      Count  
0      not-A      Did      1
0      not-A      Didnt  109 
0      A          Did      8
0      A          Didnt  102

B      not-A      Did      7
B      not-A      Didnt  103 
B      A          Did      2
B      A          Didnt  108
")

Data$X      = factor(Data$X, levels = c("Didnt", "Did"))
Data$Group2 = factor(Data$Group2, levels = c("not-A", "A"))

### Woolf test and Breslow-Day test

Table = xtabs(Count ~  Group2 + X + Group1, data=Data)

library(vcd)

woolf_test(Table)

   ### Woolf-test on Homogeneity of Odds Ratios (no 3-Way assoc.)
   ### 
   ### X-squared = 6.5759, df = 1, p-value = 0.01034

library(DescTools)

BreslowDayTest(Table)

   ### Breslow-Day test on Homogeneity of Odds Ratios
   ### 
   ### X-squared = 8.4376, df = 1, p-value = 0.003675

### Model logistic regession and analysis of deviance table

model = glm(X ~ Group1*Group2, weights=Count, data=Data, family=binomial())

library(car)

Anova(model, test="Wald")

   ### Analysis of Deviance Table (Type II tests)
   ### 
   ###               Df  Chisq Pr(>Chisq)  
   ### Group1         1 0.1109    0.73908  
   ### Group2         1 0.0033    0.95417  
   ### Group1:Group2  1 6.5759    0.01034 *

### Produce estimates of probabilities and plot

library(emmeans)

marginal = emmeans(model, ~ Group1:Group2, type="response")

marginal

   ### Group1 Group2    prob      SE  df asymp.LCL asymp.UCL
   ### 0      not-A  0.00909 0.00905 Inf   0.00128    0.0617
   ### B      not-A  0.06364 0.02327 Inf   0.03064    0.1275
   ### 0      A      0.07273 0.02476 Inf   0.03679    0.1387
   ### B      A      0.01818 0.01274 Inf   0.00455    0.0698
   ### 
   ### Confidence level used: 0.95 
   ### Intervals are back-transformed from the logit scale 

library(ggplot2)

pd = position_dodge(.2)

ggplot(as.data.frame(marginal),
       aes(x     = Group1,
           y     = prob,
           color = Group2)) +

    geom_point(shape = 15,
               size  = 4,
             position = pd) +

    geom_errorbar(aes(ymin  = asymp.LCL,
                      ymax  = asymp.UCL),
                      width = 0.2,
                      size  = 0.7,
                      position = pd) +
    theme_bw() +
    theme(axis.title = element_text(face = "bold")) +

    ylab("Probability of Did")