Dummy variable trap issues

Question

I am running a large OLS regression where all the independent variables (around 400) are dummy variables. If all are included, there is perfect multicollinearity (the dummy variable trap), so I have to omit one of the variables before running the regression.

My first question is, which variable should be omitted? I have read that it is better to omit a variable that is present in many of the observations rather than one that is present in only a few (e.g. if almost all observations are "male" or "female" and just a few are "unknown", omit either "male" or "female"). Is this justified?

After running the regression with a variable omitted, I am able to estimate the coefficient value of the omitted variable because I know that the overall mean of all my independent variables should be 0. So I use this fact to shift the coefficient values for all the included variables, and get an estimate for the omitted variable. My next question is whether there is some similar technique that can be used to estimate the standard error for the coefficient value of the omitted variable. As it is I have to re-run the regression omitting a different variable (and including the variable I had omitted in the first regression) in order to acquire a standard error estimate for the coefficient of the originally omitted variable.

Finally, I notice that the coefficient estimates I get (after re-centering around zero) vary slightly depending on which variable is omitted. In theory, would it be better to run several regressions, each omitting a different variable, and then average the coefficient estimates from all the regressions?

Answer 1

You should get the "same" estimates no matter which variable you omit; the coefficients may be different, but the estimates of particular quantities or expectations should be the same across all the models.

In a simple case, let $x_i=1$ for men and 0 for women. Then, we have the model: $$\begin{align*} E[y_i \mid x_i] &= x_iE[y_i \mid x_i = 1] + (1 - x_i)E[y_i \mid x_i = 0] \\ &= E[y_i \mid x_i=0] + \left[E[y_i \mid x_i= 1] - E[y_i \mid x_i=0]\right]x_i \\ &= \beta_0 + \beta_1 x_i. \end{align*}$$ Now, let $z_i=1$ for women. Then $$\begin{align*} E[y_i \mid z_i] &= z_iE[y_i \mid z_i = 1] + (1 - z_i)E[y_i \mid z_i = 0] \\ &= E[y_i \mid z_i=0] + \left[E[y_i \mid z_i= 1] - E[y_i \mid z_i=0]\right]z_i \\ &= \gamma_0 + \gamma_1 z_i . \end{align*}$$ The expected value of $y$ for women is $\beta_0$ and also $\gamma_0 + \gamma_1$. For men, it is $\beta_0 + \beta_1$ and $\gamma_0$.

These results show how the coefficients from the two models are related. For example, $\beta_1 = -\gamma_1$. A similar exercise using your data should show that the "different" coefficients that you get are just sums and differences of one another.

Answer 2

James, first of all why regression analysis, but not ANOVA (there are many specialists in this kind of analysis that could help you)? The pros for ANOVA is that all you actually interested in are differences in the means of different groups described by combinations of dummy variables (unique categories, or profiles). Well, if you do study impacts of each of categorical variable you include, you may run regression as well.

I think the type of the data you do have here is described in the sense of conjoint analysis: many attributes of the object (gender, age, education, etc.) each having several categories, thus you omit the whole largest profile, not just one dummy variable. A common practise is to code the categories within the attribute as follows (this link may be useful, you probably do not do conjoint analysis here, but coding is similar): suppose you have $n$ categories (three, as you suggested, male, female, unknown) then, first two are coded as usual you do include two dummies (male, female), giving $(1, 0)$ if male, $(0, 1)$ if female, and $(-1, -1)$ if unknown. In this way the results indeed will be placed around intercept term. You may code in a different way, however, but will lose the mentioned interpretation advantage. To sum up, you drop one category from each category, and code your observations in the described way. You do include intercept term also.

Well to omit the largest profile's categories seems good for me, though not so important, at least it is not empty I think. Since you code the variables in specific manner, joint statistical significance of included dummy variables (both male female, could be tested by F test) imply the significance of the omitted one.

It may happen that the results slightly different, but may be it is the wrong coding that influence this?

Answer 3

Without knowing the exact nature of your analysis, have you considered effects coding? This way each variable would represent the effect of that trait/attribute vs the overall grand mean rather than some particular omitted category. I believe you'll still be missing a coefficient for one of the categories/attributes - the one you assign a -1 to. Still, with this many dummies, I would think that the grand mean would make a more meaningful comparison group than any particular omitted category.