Preliminary t-test for moderation

Question

I am currently working on my thesis and I have run into some issues regarding interpretation of my analysis.

I wanted to find out whether relationship between my IV and DV differs based on gender (two categories - male and female). In statistics classes I have been advised to do quick t-test before running the regression analysis to find out if the two groups (male and female) differ in the DV.

So I have done the t-test and it is nonsignificant. Then I ran the moderated regression and gender did not significantly predict DV and also the interaction was not significant. But also I expected small gender differences based on literature and I do not have the appropriate sample size (N=212 and I would need twice that much) so that nonsignificant interaction might have happened due to that.

This brings me to the t-test - can I still consider the t-test results to be valid when suggesting there are no gender differences in DV?

In the discussion I would probably conclude that the interaction did not occur due to lacking in sample size but can I still interpret the t-test that is suggesting that there are no gender differences?

Answer 1

In statistics classes I have been advised to do quick t-test before running the regression analysis to find out if the two groups (male and female) differ in the DV.

You probably wouldn't get that recommendation from many of those who frequently visit this site. Any use of an outcome to decide on the structure of a model violates the assumptions for later significance testing, in a way that's difficult to control for. There was no need to do that test. Unless the design was nicely balanced between genders with respect to other outcome-associated predictors, that these might even lead you astray.

Your single model with the interaction term contains all the information you need, and it might even contain information that there is an association of gender with outcome. You say:

gender did not significantly predict DV

but I wonder if that assessment is only based on the reported single regression coefficient for gender. With gender involved in an interaction, that coefficient will be for the situation when all of the interacting predictors are at 0 or reference levels. That often is a situation of no practical importance. Furthermore, the reported p-value for that single coefficient will be whether the coefficient for gender under that situation is different from 0. That single coefficient is not a test of the overall significance of gender.

To evaluate the overall significance of gender, consider a likelihood-ratio test of two models: your full model and the same model completely without gender as a predictor. Alternatively, do a Wald test of all the coefficients involving gender in your full model, as performed for example by the Anova() function in the R car package.