2
$\begingroup$

I am modeling a continuous dependent variable with a couple of covariates (known a priori) and a variable of interest.

I ran into an issue of interpretation which I'd like to clear up. When I include a categorical predictor (Sex) my variable of interest is no longer significant. However, when I run two separate models (one for males, one for females) the variable of interest is only significant in females.

In the full model (with sex as an IV) the interaction between variable of interest * sex is non-significant.

How do I proceed? How do I know for sure I have justification to run two separate GLMs based on sex? I tried plotting the the variable of interest by sex vs. my dependent variable and observed no significant difference in the slopes (although visually these were strikingly different and the lines intersected) and significant difference in intercept.

| cite | improve this question | | | | |
$\endgroup$
  • $\begingroup$ Thanks for your response and help in the past. I will take your advice. I still have a lingering question: Let's say I have good reason to consider that there is a sexual dimorphism in my variable of interest (due to influence of sex hormones etc.). Would the chain of logic have held If the slopes were significantly different when covariate of interest are plotted against dependent variable (or its biggest determinant) grouped by sex? Meaning, if significantly different than I would have "statistical backing" to proceed with 2 separate GLMs (including underlying biological theory) $\endgroup$ – mindhabits Jul 3 '19 at 4:15
  • $\begingroup$ I ran into an issue of interpretation which I'd like to clear up. When I include a categorical predictor (Sex) my variable of interest is no longer significant. However, when I run two separate models (one for males, one for females) the variable of interest is only significant in females. THis simply suggests female factor operates as a moderator. $\endgroup$ – Subhash C. Davar Jul 3 '19 at 8:19
  • $\begingroup$ In the full model (with sex as an IV) the interaction between variable of interest*sex is non-significant. You are applying Anova model here. The Anova is based on normality assumption. The sum total of deviations is zero. And u get a nonsignificant test statistic. $\endgroup$ – Subhash C. Davar Jul 3 '19 at 8:30
  • $\begingroup$ relevant portion of your comments may be considered for inclusion in the Question to improve the quality of this question. $\endgroup$ – Subhash C. Davar Jul 4 '19 at 8:20
0
$\begingroup$

Model selection based on $p$-values will bias the coefficients of the final model towards significance. This is a form of stepwise regression and should be avoided if the goal is confirmation through $p$-values. The actual chance of a false positive will be much higher than the chosen level of significance. This has been discussed in several places on CV, most notably here.

You should therefore go with the model you originally intended to use for inference, which I'm assuming is the full model with interaction.

Even (or especially) if you have expert knowledge suggesting that gender affects the variable of interest, a single model using an interaction with gender is preferred. The interaction allows you to model gender-specific effects while having a single model has multiple advantages (most importantly: more observations).

Note that just because the effect isn't significant doesn't mean it isn't there. You just couldn't demonstrate it. This might be because (1) your sample size is limited compared to the number of parameters, (2) the effect size is small, (3) the variance of the effect is large, or a combination of these reasons. For this reason, model selection based on $p$-values usually isn't very helpful.

| cite | improve this answer | | | | |
$\endgroup$
  • $\begingroup$ most importantly: more observations). This needs to be substantiated for it seems illogical to be considered for selecttion of a Model. $\endgroup$ – Subhash C. Davar Jul 3 '19 at 23:36
  • $\begingroup$ Under the assumption that the data generating processes for male and female share any kind of similarity (e.g. intercept, slopes of other covariates, error distribution), more observations means more degrees of freedom, better estimates of your other covariates, easier checking of (and possibly even meeting) distributional assumptions. Also see here, here, and here. $\endgroup$ – Frans Rodenburg Jul 3 '19 at 23:49
  • $\begingroup$ I am not sure whether combination (single model) can yield better results. Your focus is on similarity while OP is seemingly trying check the effect of sex or gender -type on a specific variable(continuos). I can makeout much from the question becausei it is somehow vague e.g. variable of interest is not defined to be meaningful or contextual.. $\endgroup$ – Subhash C. Davar Jul 4 '19 at 3:59
  • $\begingroup$ I appreciate your commenting on my answer, but I do not understand your point of contention. Even if male and female have different slopes for the explanatory variable of interest, they still (probably) share the same error structure. Anything else they share further increases the efficiency of using a single model vs multiple models. I hope you'll agree that splitting your data reduces the sample size, and that larger sample sizes tend to result in better parameter estimates. $\endgroup$ – Frans Rodenburg Jul 4 '19 at 4:18
  • $\begingroup$ Model selection based on p-values will bias the coefficients of the final model towards significance. It is not clear to me. None-the-less Your Ans talks about step regression while OP is talking GLM $\endgroup$ – Subhash C. Davar Jul 4 '19 at 8:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.