0
$\begingroup$

I am simulating a DGP like this:

$$\operatorname{logit}[p(y) = 1] = \beta_0 + \beta_1\textrm{Treatment} + \beta_2\textrm{Treatment}\times\textrm{IsFemale} + \beta_3\textrm{ IsFemale} + \beta_4 \textrm{IsSmoker}$$ and assume I know there is an interaction effect.

library('paramtest')
library('pwr')
library('ggplot2')
library('knitr')
library('nlme')
library('lavaan')
library('dplyr')
library('lmtest')


set.seed(416)

N= 500

p_to_log_odds <- function(p){log(p / (1-p))}
b.0 <-p_to_log_odds(0.03)
b.treatment <- log(1.5)
b.is_smoker <- log(1.3)
b.is_female <- log(1.3)
b.is_female_treated <- log(1.8)

X  <- data.frame(
  treatment = c(sample(c(0,1), N, replace=TRUE)),
  is_smoker = c(sample(c(0,1), N, replace=TRUE)),
  is_female = c(sample(c(0,1), N, replace=TRUE))
)

X$lin_pred <- b.treatment * X$treatment +
            b.is_smoker * X$is_smoker + 
        b.is_female * X$is_female + 
            b.is_female_treated * X$is_female * X$treatment

X$pr <-1/(1+exp(-X$lin_pred))
X$y = rbinom(N,1,X$pr)

# MODELS 
###########
# Interaction
model.interaction <- glm(y ~ treatment + treatment*is_female + is_smoker + is_female, data=X)
print(summary(model.interaction))

When I simulate this DGP, I get a non significant interaction term.

Call:
glm(formula = y ~ treatment + treatment * is_female + is_smoker + 
    is_female, data = X)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-0.8741  -0.5327   0.2073   0.3934   0.5058  

Coefficients:
                    Estimate Std. Error t value Pr(>|t|)    
(Intercept)          0.49421    0.04918  10.048   <2e-16 ***
treatment            0.11243    0.05892   1.908   0.0569 .  
is_female            0.03848    0.06055   0.636   0.5254    
is_smoker            0.08136    0.04153   1.959   0.0507 .  
treatment:is_female  0.14760    0.08337   1.771   0.0773 .  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

But if I run models just for males and just for females I see the treatment is only significant for females.

# Subset just females      
model.fem_subset <- glm(y ~ treatment + is_smoker, data=X %>% filter(is_female==1))
print(summary(model.fem_subset))

# Subset just males
model.male_subset <- glm(y ~ treatment + is_smoker, data=X %>% filter(is_female==0))
print(summary(model.male_subset))
Call:
glm(formula = y ~ treatment + is_smoker, data = X %>% filter(is_female == 
    1))

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-0.8842  -0.5223   0.1158   0.3746   0.4777  

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  0.52234    0.04715  11.079  < 2e-16 ***
treatment    0.25880    0.05568   4.648 5.47e-06 ***
is_smoker    0.10308    0.05567   1.852   0.0653 .  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for gaussian family taken to be 0.1915177)

    Null deviance: 51.919  on 247  degrees of freedom
Residual deviance: 46.922  on 245  degrees of freedom
AIC: 298.89

Number of Fisher Scoring iterations: 2


Call:
glm(formula = y ~ treatment + is_smoker, data = X %>% filter(is_female == 
    0))

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-0.6775  -0.5653   0.3225   0.3826   0.4947  

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  0.50526    0.05650   8.942   <2e-16 ***
treatment    0.11219    0.06210   1.807    0.072 .  
is_smoker    0.06003    0.06162   0.974    0.331    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for gaussian family taken to be 0.2390443)

    Null deviance: 60.520  on 251  degrees of freedom
Residual deviance: 59.522  on 249  degrees of freedom
AIC: 359.49

Number of Fisher Scoring iterations: 2

So if this is the DGP, is it generally better to just conduct two separate regressions instead of estimating an interaction?

$\endgroup$

2 Answers 2

0
$\begingroup$

A product term in a logistic regression has an estimated value and its associated P value based on the model hypothesis. These are two different things. If the treatment effect varies by gender in stratified analyses then the overall model with the product term should have a product term coefficient different from the null value (which is what you have observed). It makes no difference whether statistical significance was observed or not as the P value only tells you how much evidence you have against the model hypothesis at your sample size and nothing more. Statistical significance is not the same thing as the estimated effect of treatment.

$\endgroup$
0
$\begingroup$

There is no conflict between these two results.

The model with the interaction term tells you that the treatment effect for males is not significantly different from the treatment effect for females.

The set of gender-specific models tells you that a) the effect size is significantly different from zero for females, and b) the effect size is not significantly different from zero for males.

In other words -The effect size for males is not significantly different from zero, or the effect size for females -The effect size for females is significantly different from zero, but not significantly different from the effect size for males.

If it helps you can think about this in terms of confidence intervals (this is not exactly accurate but may help to visualize why this isn't a contradiction). Imagine a number line with three points, running from left to right: zero, male effect, female effect. There are CIs around the two effects but not zero. The confidence intervals around male effect include both zero and the female effect. The CIs around the female effect include the male effect, but not zero.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.