Interpreting the effects of dummy interactions

Question

Warning

This question is quite long, and maybe a lot of you will think it is too long. I however thought, and hope, that if this question gets a proper answer, it will actually be a really good post to help people with getting a quite comprehensive understanding of the slightly more advanced workings of dummy variables. Related to this, please feel free to edit this question if you think some parts could be made clearer.

Background

Although I think I am getting quite up to standards when interpreting dummy interaction effects, I always get confused in some situations. For example when trying to control for a sample imbalance. Added because of a question in the comments: The problem with the sample imbalance that I perceive, is that because I have substantially more females in my sample. I am worried that the effects that I pick up (not only for the treatment, I just wanted to not over-complicate the example), are not purely attributable to the variables I include, but also to potential confounding characteristics which are more present in the female subsample.

In this example, there are three types of variables:

1. Choice, a dummy variable, which is the dependent variable in all regressions.
2. treatment, a dummy variable (treatment = 1, control = 0)
3. Gender, a dummy variable (Male or Female)

.

one <- glm(Choice ~ as.factor(gender), family = binomial(link=logit), data=data)
two <- glm(Choice ~ as.factor(treatment), family = binomial(link=logit), data=data)
three <- glm(Choice ~ treatment + as.factor(gender), family = binomial(link=logit), data=data)
four <- glm(Choice ~ as.factor(gender):as.factor(treatment), family = binomial(link=logit), data=data)  
five <- glm(Choice ~ as.factor(gender):as.factor(treatment) +  as.factor(gender), family = binomial(link=logit), data=data)  
six <- glm(Choice ~ as.factor(gender)*as.factor(treatment), family = binomial(link=logit), data=data)  
====================================================================================================
                                                             Dependent variable:                    
                                         -----------------------------------------------------------
                                                                   Choice                           
                                            (1)       (2)       (3)       (4)       (5)       (6)   
----------------------------------------------------------------------------------------------------
treatment                                                     0.200*                                
                                                              (0.120)                               
                                                                                                    
Male                                     -0.340***           -0.340***            -0.140    -0.140  
                                          (0.120)             (0.120)             (0.170)   (0.170) 
                                                                                                    
as.factor(treatment)1                               0.200*                                  0.350** 
                                                    (0.120)                                 (0.140) 
                                                                                                    
Female:as.factor(treatment)0                                            0.210                      
                                                                        (0.170)                     
                                                                                                    
Male:as.factor(treatment)0                                               0.077                      
                                                                        (0.200)                     
                                                                                                    
Female:as.factor(treatment)1                                            0.570***   0.350**           
                                                                        (0.180)   (0.140)           
                                                                                                    
Male:as.factor(treatment)1                                                        -0.077    -0.430* 
                                                                                  (0.200)   (0.240) 
                                                                                                    
Constant                                 -0.210*** -0.420*** -0.300*** -0.590*** -0.380*** -0.380***
                                          (0.071)   (0.080)   (0.090)   (0.140)   (0.099)   (0.099) 
                                                                                                    
----------------------------------------------------------------------------------------------------
Observations                               1,242     1,242     1,242     1,242     1,242     1,242  
Log Likelihood                           -840.000  -843.000  -839.000  -837.000  -837.000  -837.000 
Akaike Inf. Crit.                        1,685.000 1,690.000 1,684.000 1,683.000 1,683.000 1,683.000
====================================================================================================
Note:                                                                    *p<0.1; **p<0.05; ***p<0.01

My understanding

Regression 3: I know that normally, adding a dummy, causes an intercept change, between the base group (treatment=0). However, my experiment had a gender imbalance, so I wanted to correct for that, adding a gender dummy. Now I started to doubt the interpretation. To check, I ran the regression once with only with the gender and once with only the treatment, once with both. Since the coefficients do not change, I have to assume that BOTH dummies signify independent intercept changes. So actually I am in no way controlling for the gender imbalance.

Regression 4: Here I only interacted the gender with the treatment. As there is no coefficient on as.factor(gender)2:as.factor(treatment)1, I assume that this is the base level, which I believe means that all other coefficients are the interaction effects, compared to the base level. Thinking about this, I realised this is super uninformative, because if I understand correctly, it says nothing anymore about the effects on Choice, just how one compares to the other.

Regression 5: Compared to regression (4), I now added a separate gender dummy. It now seems that I have what I want, because I get a treatment effect for men and a treatment effect for women. The gender dummy, if I understand correctly, now simply gives me the difference between men and women for the non treatment group.

Regression 6: I thought I would get the most information by interacting the dummmies, and also add them separately, but I think this does not work, because there are only 6 different groups. The treatment effect for men in this regression is I believe, the treatment effect plus the treatment interaction, which coincides with the effect of as.factor(gender)2:as.factor(treatment)0 in regression 4.

My questions

1. Are my interpretations above correct?

2. Am I correct in saying that only regression (5), gives me the treatment effect, corrected for the sample imbalance? If so, why did it necessitate me to add a separate gender dummy?

3. Which variable/interaction, tells me whether the difference in treatment effect between men and women is significantly different? Do I need to run even another regression for this?

Answer 1

I will try to answer your question, feel free to comment if somthing is unclear or i have misunderstood you.

1) to your question: Are my interpretations above correct?

I will try to go trough the models one at a time, rather than answer wit a simple yes or no.

Lets look at model 1.

one <- glm(Choice ~ as.factor(gender), family = binomial(link=logit), data=data)

To be sure we are on the same understanding, this is the same model as

one <- glm(Choice ~ gender, family = binomial(link=logit), data=data)

When gender is a string variable. R will make sure the interpretation is correct. Below I will omit the as.factor() function if reasonable to simplify the answer.

Model one is a simple model investigating the association between an outcome Choice and the gender gender. The coefficient estimate for gender, -0.340 indicates how much the probability of Choice=1 for males diverge (in this case decrease) from females. Of cause in this model we ignore any other variables that may influence the association between gender and Choice.

In conclusion, model 1 allow us to test and interpretant the crude relation between Choice and gender.

If you are not with me so far, try to look up simple models. Otherwise lets look at model 2.

two <- glm(Choice ~ treatment, family = binomial(link=logit), data=data)

Model 2 is very similar to model 1. Only this time instead of association between Choice and gender, we now look at the relation of Choice and treatment. The above should allow you to extend to this case.

So far so good, lets move to model 3.

three <- glm(Choice ~ treatment + gender, family = binomial(link=logit), data=data)

Say we are interested in the treatment effect on Choice, but we know that gender will affect both the Choice and treatment. Then gender is a so called confounder. I expect this is indirectly what you mean when you talk about imbalance. You expect that gender may influence the model result.

The output here will give an estimate for both gender and treatment. If we are only interested in the effect of treatment on choice, we would ignore the coefficient estimate for gender and only look at the estimate for treatment. Thus we can check if treatment has an effect on the choice, when correcting/adjusting for gender. Are we on the other hand interested in the effect of gender, we would consider the estimate for gender and ignore the treatment effect. Thus we would get the effect of gender on choice when we correct for treatment.

I model 3 have no interaction-term, that is we assume that the treatment influence the choice in an equal manner for both men and women. Or vice versa if we switch treatment and gender.

So lets consider an interaction term. Model 4 is not actually reasonable because it only ad the cross term between gender and treatment but dont allow a separate effect. If this is new to you, I suggest you learn about interaction models. The internet should help you here. The same goes for model 5. Since only one of the terms gender or treatment is allowed to have a separate effect.

So let's look at model 6.

six <- glm(Choice ~ gender*treatment, family = binomial(link=logit), data=data) 

Model 6 is equivalent to the model

six <- glm(Choice ~ gender+treatment+gender:treatment, family = binomial(link=logit), data=data)

Shortly it asks the program to model the effect of treatment (or gender) on Choice when corrected for gender (or treatment) and the treatment(gender) effect is allowed to vary across levels of gender (treatment). Forget interpretation of the value of the coefficients, this usually lead to confusion. Focus on the interaction term gender:treatment. If it influence the choice significantly, a cross effect exists.

This should also answer question 2)

3) Which variable/interaction, tells me whether the difference in treatment effect between men and women is significantly different? Do I need to run even another regression for this

The interaction term should tell you this.