3
$\begingroup$

Warning

This question is quite long, and maybe a lot of you will think it is too long. I however thought, and hope, that if this question gets a proper answer, it will actually be a really good post to help people with getting a quite comprehensive understanding of the slightly more advanced workings of dummy variables. Related to this, please feel free to edit this question if you think some parts could be made clearer.

Background

Although I think I am getting quite up to standards when interpreting dummy interaction effects, I always get confused in some situations. For example when trying to control for a sample imbalance. Added because of a question in the comments: The problem with the sample imbalance that I perceive, is that because I have substantially more females in my sample. I am worried that the effects that I pick up (not only for the treatment, I just wanted to not over-complicate the example), are not purely attributable to the variables I include, but also to potential confounding characteristics which are more present in the female subsample.

In this example, there are three types of variables:

  1. Choice, a dummy variable, which is the dependent variable in all regressions.
  2. treatment, a dummy variable (treatment = 1, control = 0)
  3. Gender, a dummy variable (Male or Female)

.

one <- glm(Choice ~ as.factor(gender), family = binomial(link=logit), data=data)
two <- glm(Choice ~ as.factor(treatment), family = binomial(link=logit), data=data)
three <- glm(Choice ~ treatment + as.factor(gender), family = binomial(link=logit), data=data)
four <- glm(Choice ~ as.factor(gender):as.factor(treatment), family = binomial(link=logit), data=data)  
five <- glm(Choice ~ as.factor(gender):as.factor(treatment) +  as.factor(gender), family = binomial(link=logit), data=data)  
six <- glm(Choice ~ as.factor(gender)*as.factor(treatment), family = binomial(link=logit), data=data)  
====================================================================================================
                                                             Dependent variable:                    
                                         -----------------------------------------------------------
                                                                   Choice                           
                                            (1)       (2)       (3)       (4)       (5)       (6)   
----------------------------------------------------------------------------------------------------
treatment                                                     0.200*                                
                                                              (0.120)                               
                                                                                                    
Male                                     -0.340***           -0.340***            -0.140    -0.140  
                                          (0.120)             (0.120)             (0.170)   (0.170) 
                                                                                                    
as.factor(treatment)1                               0.200*                                  0.350** 
                                                    (0.120)                                 (0.140) 
                                                                                                    
Female:as.factor(treatment)0                                            0.210                      
                                                                        (0.170)                     
                                                                                                    
Male:as.factor(treatment)0                                               0.077                      
                                                                        (0.200)                     
                                                                                                    
Female:as.factor(treatment)1                                            0.570***   0.350**           
                                                                        (0.180)   (0.140)           
                                                                                                    
Male:as.factor(treatment)1                                                        -0.077    -0.430* 
                                                                                  (0.200)   (0.240) 
                                                                                                    
Constant                                 -0.210*** -0.420*** -0.300*** -0.590*** -0.380*** -0.380***
                                          (0.071)   (0.080)   (0.090)   (0.140)   (0.099)   (0.099) 
                                                                                                    
----------------------------------------------------------------------------------------------------
Observations                               1,242     1,242     1,242     1,242     1,242     1,242  
Log Likelihood                           -840.000  -843.000  -839.000  -837.000  -837.000  -837.000 
Akaike Inf. Crit.                        1,685.000 1,690.000 1,684.000 1,683.000 1,683.000 1,683.000
====================================================================================================
Note:                                                                    *p<0.1; **p<0.05; ***p<0.01

My understanding

Regression 3: I know that normally, adding a dummy, causes an intercept change, between the base group (treatment=0). However, my experiment had a gender imbalance, so I wanted to correct for that, adding a gender dummy. Now I started to doubt the interpretation. To check, I ran the regression once with only with the gender and once with only the treatment, once with both. Since the coefficients do not change, I have to assume that BOTH dummies signify independent intercept changes. So actually I am in no way controlling for the gender imbalance.

Regression 4: Here I only interacted the gender with the treatment. As there is no coefficient on as.factor(gender)2:as.factor(treatment)1, I assume that this is the base level, which I believe means that all other coefficients are the interaction effects, compared to the base level. Thinking about this, I realised this is super uninformative, because if I understand correctly, it says nothing anymore about the effects on Choice, just how one compares to the other.

Regression 5: Compared to regression (4), I now added a separate gender dummy. It now seems that I have what I want, because I get a treatment effect for men and a treatment effect for women. The gender dummy, if I understand correctly, now simply gives me the difference between men and women for the non treatment group.

Regression 6: I thought I would get the most information by interacting the dummmies, and also add them separately, but I think this does not work, because there are only 6 different groups. The treatment effect for men in this regression is I believe, the treatment effect plus the treatment interaction, which coincides with the effect of as.factor(gender)2:as.factor(treatment)0 in regression 4.

My questions

  1. Are my interpretations above correct?

  2. Am I correct in saying that only regression (5), gives me the treatment effect, corrected for the sample imbalance? If so, why did it necessitate me to add a separate gender dummy?

  3. Which variable/interaction, tells me whether the difference in treatment effect between men and women is significantly different? Do I need to run even another regression for this?

$\endgroup$
6
  • $\begingroup$ Why are you speaking of imbalance? Adding gender is adding a second factor to control for and test for, it seems nothing to do with imbalance by gender. $\endgroup$
    – ttnphns
    Commented Jul 28, 2021 at 11:24
  • $\begingroup$ It looks like you are using dummy variables as categorical factors, aren't you? Dummies are quantitative variables in which you encode, recode your initial categorical variables, and they should enter the model as (sets of) quantitative predictors. covariates. $\endgroup$
    – ttnphns
    Commented Jul 28, 2021 at 11:29
  • $\begingroup$ @ttnphns Thank you for your comment. The problem is that because I have substantially more females in my sample. As a result, I am worried that the effects that I pick up (not only for the treatment, I just wanted to not over-complicate the example), are not purely attributable, to the variables I include, but also confounding characteristics which are more present in the female subsample. $\endgroup$
    – Tom
    Commented Jul 28, 2021 at 11:31
  • $\begingroup$ @ttnphns I think I understand what you are getting at, but not completely. Is it possible to provide me with a basic example, and a little explanation of how that actually changes the approach? $\endgroup$
    – Tom
    Commented Jul 28, 2021 at 11:33
  • $\begingroup$ Your total examle isn't very good, didactically. Because both your factors, treatment and gender, are dichotomous, two-level, and coded as binary. In such a case, it makes no difference whether to enter them as categorical factors or quantitative regressors (in the form of dummies). Still, your question title reveals that you are interested in the behaviour of dummies. I might recommend to post a question with two factors, both 3-level or one is 2-level and the other is 3-level. If your goal is to get understanding of how dummies behave. $\endgroup$
    – ttnphns
    Commented Jul 28, 2021 at 11:53

1 Answer 1

2
$\begingroup$

I will try to answer your question, feel free to comment if somthing is unclear or i have misunderstood you.

1) to your question: Are my interpretations above correct?

I will try to go trough the models one at a time, rather than answer wit a simple yes or no.

Lets look at model 1.

one <- glm(Choice ~ as.factor(gender), family = binomial(link=logit), data=data)

To be sure we are on the same understanding, this is the same model as

one <- glm(Choice ~ gender, family = binomial(link=logit), data=data)

When gender is a string variable. R will make sure the interpretation is correct. Below I will omit the as.factor() function if reasonable to simplify the answer.

Model one is a simple model investigating the association between an outcome Choice and the gender gender. The coefficient estimate for gender, -0.340 indicates how much the probability of Choice=1 for males diverge (in this case decrease) from females. Of cause in this model we ignore any other variables that may influence the association between gender and Choice.

In conclusion, model 1 allow us to test and interpretant the crude relation between Choice and gender.

If you are not with me so far, try to look up simple models. Otherwise lets look at model 2.

two <- glm(Choice ~ treatment, family = binomial(link=logit), data=data)

Model 2 is very similar to model 1. Only this time instead of association between Choice and gender, we now look at the relation of Choice and treatment. The above should allow you to extend to this case.

So far so good, lets move to model 3.

three <- glm(Choice ~ treatment + gender, family = binomial(link=logit), data=data)

Say we are interested in the treatment effect on Choice, but we know that gender will affect both the Choice and treatment. Then gender is a so called confounder. I expect this is indirectly what you mean when you talk about imbalance. You expect that gender may influence the model result.

The output here will give an estimate for both gender and treatment. If we are only interested in the effect of treatment on choice, we would ignore the coefficient estimate for gender and only look at the estimate for treatment. Thus we can check if treatment has an effect on the choice, when correcting/adjusting for gender. Are we on the other hand interested in the effect of gender, we would consider the estimate for gender and ignore the treatment effect. Thus we would get the effect of gender on choice when we correct for treatment.

I model 3 have no interaction-term, that is we assume that the treatment influence the choice in an equal manner for both men and women. Or vice versa if we switch treatment and gender.

So lets consider an interaction term. Model 4 is not actually reasonable because it only ad the cross term between gender and treatment but dont allow a separate effect. If this is new to you, I suggest you learn about interaction models. The internet should help you here. The same goes for model 5. Since only one of the terms gender or treatment is allowed to have a separate effect.

So let's look at model 6.

six <- glm(Choice ~ gender*treatment, family = binomial(link=logit), data=data) 

Model 6 is equivalent to the model

six <- glm(Choice ~ gender+treatment+gender:treatment, family = binomial(link=logit), data=data)

Shortly it asks the program to model the effect of treatment (or gender) on Choice when corrected for gender (or treatment) and the treatment(gender) effect is allowed to vary across levels of gender (treatment). Forget interpretation of the value of the coefficients, this usually lead to confusion. Focus on the interaction term gender:treatment. If it influence the choice significantly, a cross effect exists.

This should also answer question 2)

3) Which variable/interaction, tells me whether the difference in treatment effect between men and women is significantly different? Do I need to run even another regression for this

The interaction term should tell you this.

$\endgroup$
6
  • $\begingroup$ Kirsten, thank you so much for your very elaborate answer! It is enormously appreciated! I have only very few questions left. If I understand everything correctly, only regression six, properly accounts for the "imbalance" (the fact that I have much more females in the sample than males). Is that correct? For the last question, you say that the interaction of male*treatment tells me whether there is a significant difference between the treatment effect for men and women. Is this because the separate treatment effect only refers to the female subsample (this confuses me sometimes... $\endgroup$
    – Tom
    Commented Jul 28, 2021 at 12:14
  • $\begingroup$ because it obviously just says treatment, which might "linguistically" suggest that it applies to the whole sample. $\endgroup$
    – Tom
    Commented Jul 28, 2021 at 12:16
  • $\begingroup$ First, yes. Second, the interaction term treatment:gender in an ANOVA for binominal regression will tell you if the interaction term is significant. If yes, there is a significant difference in treatment effect on Choice across gender. If no the relation is non-significant. But yes you can also see it directly in the male*treatment in your table. $\endgroup$
    – Kirsten
    Commented Jul 28, 2021 at 12:32
  • 1
    $\begingroup$ I am sorry for the confusion. I was giving options. You can read it directly in Male:as.factor(treatment)1 for model 6. You dont need another regression. $\endgroup$
    – Kirsten
    Commented Jul 28, 2021 at 13:06
  • 1
    $\begingroup$ Thank you for completely making my day! $\endgroup$
    – Tom
    Commented Jul 28, 2021 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.