Should we conduct post-hoc tests when there is a significant interaction but no significant main effects?

Question

I have data showing outcomes of some treatment on different people grouped by sex and age (grouped into: infant, child, adult). I need to investigate if there is any difference in outcome among age groups or between sexes.

I used:

model <- aov(data$outcome ~ data$sex*data$age)
summary(model)

and found that neither sex nor age were significant on their own, but the interaction term was significant.

I read about the Tukey test so I tried:

TukeyHSD(model)

and found that none of the differences were significant. Was this the right thing to do? Should my conclusion be that there is no significant difference or is there more I should do?

Answer 1

I think your conclusion should be that there is a significant interaction but that this is not significant after accounting for multiple testing.

But your emphasis should be on the size of the interaction and what it means rather than whether p = 0.04 or 0.07 or whatever. I would calculate the predicted level of the outcome for men and women of different ages (say, the 3 quartiles of age) and see if the differences were of substantive import.

Of course, there could also be nonlinear effects, so I would plot the data in various ways. E.g. you could plot age on the x axis, outcome on the y axis and color the dots blue for males and red for females, then look and see what is going on. You could add loess lines for males and females and see if a) Those lines are roughly parallel and b) If they are roughly straight.