5
$\begingroup$

When an explanatory variable in logistic regression is binary, the interpretation is relatively straightforward.

For example, when your response variable is college admission (binary: yes or no) and explanatory variable is gender (with two levels: female and male), you can say that being male increases (or decreases) the odds of being admitted by a factor of n, holding other variables constant.

But what is the explanatory variable has multiple levels / categories? For instance, your response variable is college admission, but explanatory variable is ethnicity (with levels: Caucasian, African_American, Hispanic, and Asian). R (or any other software) will still run the model, automatically choosing one one of the levels as reference (in R it will be African_American unless specified otherwise). The results will tell us how not being African_American (or vide versa) affects the odds of admission, but says nothing for being Causasian, Hispanic, or Asian.

What's the best way of dealing with this problem? Should one create dummy variables for each ethnic group, or is there a better way?

$\endgroup$
4
$\begingroup$

The interpretation for categorical variables with more than 2 levels is very similar to the binary case you mention; for a k-level categorical variable, you will have k-1 regression coefficients each of which compare the odds of the outcome to the reference group. For the example you state, ethnicity (Caucasian, African-American, Hispanic, and Asian), let us assume your referent (baseline) group is African-American. Many software packages for logistic regressions will give you 3 Odds ratios (for a 4-level categorical predictor) once you run the regression. Let us quickly look at how this is done in R based on simulated dataset:

###########Simulate Data###########
set.seed(123) # set seed if you want to re-produce simulation results
x1 <- sample(c("AF","AS","HI","CA"),10000,replace = T) #Caucasian (CA), African-American(AA), Hispanic(HI), and Asian(AA)
x1 <- factor(x1,levels =c("AF","AS","HI","CA")) # ensure the ordering by setting AF as reference

x1.fac <- model.matrix(~x1) # generate dummy variables for simulation purposes (in practice you may not need to do this)
betas <- c(.2,.5,.53) # log odds comparing the three groups to the refernt level of AF (these are just made up values for illustration and simulation purposes!)
xbeta <- x1.fac[,-1]%*%betas #need only k-1 dummies for a variable with k-levels
y<-rbinom(n = 10000,size = 1,prob = exp(xbeta)/(1+exp(xbeta))) # Simulate outcome (Y)

Finally we have the following sample data:

example_data <- data.frame(y,x1)

####Run regression of outcome against ethnicity
model1 <- glm(y~x1,family = binomial,data = example_data)
exp(coef(model1))[-1] ###Odds Ratios comparing each group with the reference group of AF
x1AS     x1HI     x1CA 
1.229610 1.800985 1.796416 

So what does the odds ratio of 1.23 for Asians mean? This means, compared to African-Americans Asians had 23% higher odds of the outcome. Equivalently, you can interpret as Asians have 1.23 times the odds of the outcome compared to the referent group of African Americans. The odds of 1.800 and 1.796 for Caucasians and Hispanics, respectively, are interpreted in the same manner. The most important part of modeling categorical variables is identifying the proper referent group. You can always change the reference group by using the relevel() command in R. See example here.

In order to make comparison between two groups where one of them is not a referent group, there are a few ways to go:

  1. Use relevel() function and re-run the regression changing the reference group to your variable of interest (not my favorite approach when there are many levels in your categorical predictor)

  2. Use already built in packages to do this comparison.

I am not sure how this is done in Stata or SAS (probably contrast statement for SAS) but you can easily do this in R using the car package. For example, if you want to test if the odds of the outcome differ between Caucasians and Hispanics, use the following commands:

library(car)
linearHypothesis(model1,c("x1CA - x1HI = 0"))

Linear hypothesis test

Hypothesis:
- x1HI  + x1CA = 0

Model 1: restricted model
Model 2: y ~ x1

  Res.Df Df  Chisq Pr(>Chisq)
1   9997                     
2   9996  1 0.0018     0.9658

In this case, we fail to reject the null hypothesis of no difference in the odds of the outcome between Caucasians and Hispanics (p-value=0.9658).

$\endgroup$
1
$\begingroup$

For a categorical variable that is not nominal, a logistic regression will output coefficients to a One Hot Encoded version of it.

Therefore the logic remains the same: there will be a coefficient for "Caucasian" / "Not caucasian", another for "Hispanic" / "Not hispanic" and so on. The encoding makes it impossible to have "Caucasian" and "Hispanic" at the same time.

$\endgroup$
  • 1
    $\begingroup$ I disagree. Most software creates a dummy variable encoding (num categories -1), and the comparison for each category is to a reference category. $\endgroup$ – Glen Jun 13 '18 at 20:08
  • $\begingroup$ I'm not sure if I like the one-hot, because biracial folks would be excluded. :) There are cases where more than one category applies, though that should show up in the data capture, before the analysis. $\endgroup$ – EngrStudent Jun 13 '18 at 20:34
  • $\begingroup$ @Glen My bad then, I'm a python user and I apply a One Hot Encoder myself :) I don't know about most other softwares. Rather do it yourself so you know what you're dealing with imo. The logic remains in this context. @ EngrStudent if there is only one categorical variable it cannot show that, except if there is a "Caucasian and hispanic" level for instance. $\endgroup$ – Pierre Gourseaud Jun 13 '18 at 20:50
1
$\begingroup$

This is a good question, I've had it myself. Note if you only have one categorical variable then the intercept term corresponds to the reference category. If you have more than one categorical variable in your model then it becomes tricky.

One way is to rerun the model with different reference levels (clunky).

A better way is to create a dummy variables for each category (one hot encoding) as you mention. However, if you estimate the model this way you need to remove the intercept term or else it is overparameterized.

$\endgroup$
  • $\begingroup$ Thanks for this. Am I understanding correctly that, if I want to estimate glm(admission~ethnicity+gender...., then I should recode ethnicity as separate dummy variables (in contrast to @user3487564 above)? If I'm not all that interested in the effect of both ethnicity and gender, isn't it easier to run two models separately [glm(admission~ethnicity... and glm(admission~gender...] and follow @user3487564 's solution? $\endgroup$ – KaC Jun 13 '18 at 20:34
  • 1
    $\begingroup$ I suggest keeping both variables in the model instead of separate analyses. Try one hot encoding ethnicity and then remove the intercept from the model (you don't need to one hot encode binary variables). Your estimate for a specific gender/race from this model should be the same when estimating the model without one hot encoding so be sure to check this. $\endgroup$ – Glen Jun 13 '18 at 20:44
  • $\begingroup$ Indeed. Thank you again, Glen. A quick search suggests that there are different methods of removing the intercept in R. Is any of them in any way superior than the obvious glm(admission~ethnicity+gender-1...? $\endgroup$ – KaC Jun 13 '18 at 20:53
  • 1
    $\begingroup$ Not that I know of. $\endgroup$ – Glen Jun 13 '18 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.