I am going to use R to demonstrate these concepts, but they can be done in any popular statistics software.
Example Data
The example I like to use is some athletic event. Imagine we have Teams A and B will be facing off against one another. We get people to imagine how happy they will be if Team A wins or if Team B wins. We also ask them if they are a fan of Team A or Team B. Lastly, we get what hand they write with. This gives us four variables: happiness (dependent variable, DV), winning team (first independent variable, IV1), fan of team (IV2), or handedness (IV3).
I simulated data representing this using R:
# generate data ----------------------------------------------------------------
set.seed(1839)
n <- 300
fan <- sample(c("a", "b"), n, TRUE)
win <- sample(c("a", "b"), n, TRUE)
happiness <- rnorm(n, ifelse(fan == win, .5, -.5), 1)
hand <- sample(c("l", "r"), n, TRUE)
The results are programmed to align with what is obvious: The effect of who wins on happiness depends on who the person is a fan of (two-way interaction), but this doesn't depend on handedness (lack of a three-way interaction). People are happy when their team wins, but sad when their team loses.
Main Effect
The main effect can be seen as the effect of one independent variable on the dependent variable, averaging across all other variables. You could also say that it is the effect ignoring or collapsing across all other variables.
To get the main effect of which team wins (A or B) on happiness, we can do a t-test. This ignores all of the other independent variables.
# main effect ------------------------------------------------------------------
t.test(happiness ~ win)
This returns:
Welch Two Sample t-test
data: happiness by win
t = 0.12692, df = 296.16, p-value = 0.8991
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.2346082 0.2669540
sample estimates:
mean in group a mean in group b
-0.08008830 -0.09626121
There is not a significant main effect of which team wins on happiness. This is because it depends on who people are a fan of. We can do that by looking at the interaction.
Fitting Model
I fit a linear model (the same thing as an ANOVA) to look at the main effects, all two-way interactions, and the three-way interaction.
# fit model --------------------------------------------------------------------
model <- lm(happiness ~ fan * win * hand)
summary(model)
This returns a coefficients table that looks like:
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.2060 0.1612 1.278 0.202323
fanb -0.6378 0.2265 -2.815 0.005203 **
winb -0.8181 0.2238 -3.655 0.000304 ***
handr 0.1704 0.2280 0.747 0.455450
fanb:winb 1.5237 0.3275 4.652 4.99e-06 ***
fanb:handr -0.2865 0.3304 -0.867 0.386542
winb:handr -0.1571 0.3147 -0.499 0.617968
fanb:winb:handr 0.7053 0.4620 1.527 0.127927
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
The three-way interaction (fanb:winb:handr) is not significant, but one of the two-way interactions (fanb:winb) is. Now we can "probe" this interaction.
Simple Main Effects
Recall the definition of a main effect:
- The effect of IV1 on the DV, ignoring the effects of all other IVs.
We can change this to the definition of a main effect by changing what comes after the comma:
- The effect of IV1 on the DV, at a specified level of IV2.
These can also be called "conditional" main effects or "conditional average treatment effects" because it looks at the effect of IV1 on the DV given some level of IV2.
So while a main effect says something like, "IV1 does not affect DV," a simple main effect says, "When IV2 = A, then IV1 negatively affects DV." It is that "When IV2 = A" that makes it a simple main effect; it is the main effect of a variable at a given level of another variable (in this case, IV2 = A).
You do not need to fit a different model to examine this. You can estimate it from the model you have. In R, it is as simple as:
# simple main effect -----------------------------------------------------------
library(emmeans)
pairs(emmeans(model, ~ win | fan))
The ~ win | fan is saying that the DV is predicted by (~) the effect of winning, given (|) the level of fandom. This returns:
fan = a:
contrast estimate SE df t.ratio p.value
a - b 0.8966396 0.1573624 292 5.698 <.0001
fan = b:
contrast estimate SE df t.ratio p.value
a - b -0.9797111 0.1691024 292 -5.794 <.0001
So when fan = a, then Team A winning leads to an increase of happiness over Team B by about .9. On the other hand, when fan = b, the effect of Team A winning leads to a decrease of happiness by about .98. The simple main effect part is in specifying the effect of winning at different levels of fandom.
We can also look at it by handedness if we want, but since there was no significant interaction, then it won't make much of a difference:
> pairs(emmeans(model, ~ win | fan + hand))
fan = a, hand = l:
contrast estimate SE df t.ratio p.value
a - b 0.8180739 0.2237993 292 3.655 0.0003
fan = b, hand = l:
contrast estimate SE df t.ratio p.value
a - b -0.7056253 0.2391469 292 -2.951 0.0034
fan = a, hand = r:
contrast estimate SE df t.ratio p.value
a - b 0.9752053 0.2212816 292 4.407 <.0001
fan = b, hand = r:
contrast estimate SE df t.ratio p.value
a - b -1.2537969 0.2391469 292 -5.243 <.0001
