What is the general guideline for dropping dummy variables in a regression model? [duplicate]

Question

For an ordinary least square (OLS) regression problem of the form $y = \beta + w_1 x_1 + \ldots + w_n x_n$, let's say I have 3 categorical variables.

• gender: male, female
• type: office, field, manager
• ethnicity: white, black, other

And let's also say I have some other continuous variables (not made explicit here). I know that when I dummy encode (e.g. one-hot encode) the categorical variables, I will have the following variables.

• gender_male, gender_female
• type_office, type_field, type_manager
• ethnicity_white, ethnicity_black, ethnicity_other

I know that I should drop one of the gender dummy variables; let's say, gender_female. What about the other dummy variables? Should I drop one from each too? Or can I leave them all in?

I found one post that seem to suggest that I should drop 1 dummy variable and leave the rest alone. Is this the right approach?

Let's say I decide to drop one dummy variable derived from each of the categorical ones. My model might look like the following.

$y = 1.0 + 0.8 \mathrm{gender_{male}} + 0.7 \mathrm{type_{field}} + 0.2\mathrm{type_{manager}} - 0.8 \mathrm{ethnicity_{white}} + 0.5 \mathrm{ethnicity_{black}} + w_6 x_6 + \ldots + w_n x_n$

With the exception of gender, which is binary, how do I account and explain for the dummy variables that I dropped out? (e.g. type_office and ethnicity_other).

I guess I am having problems reconciling what to do to avoid mathematical problems (e.g. singularities) and then dealing with the side-effects of how to explain or interpret the model. In this post, one reply suggest to leave everything in for OLS regression if regularization is used.

Answer 1

Rather than thinking about "dropping" a dummy variable, think about the categorical variable you start out with.

For any such categorical variable, when you use dummy coding what you are essentially saying is:

 - I will set aside one of the categories of the variable and treat it   
 as the reference level; 

 - I will compare the mean value of y corresponding to each of the    
 remaining categories against the mean value of y corresponding to the 
 reference level (controlling for everything else in the model). 

Let's say the categorical variable is gender - if we set aside the male category, all we need to do is create a dummy variable for the female category (such that dummy = 1 when gender = female and dummy = 0 else) and include it in our model. The coefficient of this dummy variable will represent the difference in the mean values of y for females and males workers/employees having the same type of job and the same ethnicity.

How did we specify the effects of type of job and ethnicity in our model?

For type of job, we set aside the office category as the reference category and created two dummy variables - one for field and one for manager. The coefficient of the dummy variable for field represents the difference in the mean values of y for field and office workers who have the same gender and the same ethnicity. The coefficient of the dummy variable for manager represents the difference in the mean values of y for manager and office workers who have the same gender and the same ethnicity.

For ethnicity, we set aside the white category as the reference category and created two dummy variables - one for black and one for other ethnicity. The coefficient of the dummy variable for black represents the difference in the mean values of y for black and white workers who have the same gender and the same type of job. The coefficient of the dummy variable for other ethnicity represents the difference in the mean values of y for other ethnicity and white ethnicity workers who have the same gender and the same type of job.

When we do model selection, we need to remove ALL dummy variables used to encode the effect of a categorical variable (e.g., ethnicity).

Often, people will set aside the category which is most populated or one which acts as a natural reference point for the other categories.