1
$\begingroup$

We're given a regression equation with two dummy variables which are perfectly collinear. $$ y_i = \beta_1 D1_i + \beta_2 D2_i + e_i$$ where $ D2_i = 1-D1_i$. Can we estimate this model using least squares?

I thought we couldn't because multicollinearity, but my teacher said we could because the model didn't have a intercept term. I don't understand the implication of having / not having an intercept term on whether or not we can run the regression. Here is what my teacher said:

Yes, you can estimate this model using least squares because there's no intercept term and hence running this regression won't lead to the problem of multicollinearity. It is equivalent to running two separate regressions $y$ on $D_1$ and $y$ on $D_2$ because $D_1$ and $D_2$ are orthogonal.

$\endgroup$
2
$\begingroup$

\begin{aligned} y_i &= \beta_1D1_i + \beta_2D2_i +e_i\\ &=\beta_2 + (\beta_1-\beta_2)D1_i+e_i \end{aligned}

In this sense, you will be able to uniquely estimate $\beta_2$ and $\beta_1 - \beta_2$ and hence $\beta_1$. You won't be able to do this if you specify a intercept term.

For a more formal discussion, you can google Estimability. Because I think your question is more about estimability, instead of multicolinearity.

$\endgroup$
1
$\begingroup$

You are right that if you include both an intercept and an indicator variable for each group, you will have perfect multicollinearity (see here). However, your teacher is also right that you will be fine if you suppress the intercept.

Including the intercept and indicators for $k-1$ levels of a categorical variable is called "reference level coding". Suppressing the intercept and including indicator variables for all $k$ levels is called "level means coding". Both are perfectly acceptable ways to represent categorical variables (although reference level coding is the most common). It is important to realize, however, that if you use level means coding, the meaning of the statistical tests that come with your output is different from that of reference level coding (see here).

We can demonstrate these issues with a simple R simulation (let me know if the code isn't self-explanatory):

set.seed(5489)  # this makes the simulation exactly reproducible
# these are the indicator variables for the two levels:
i1 = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
i2 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1)
# I will simulate data using reference level coding
# the population mean of i1 is 3, & the difference b/t i1 & 12 is 1
#   (i.e., the popultion mean of i2 is 4), then there is some normal random error
y  = 3 + 1*i2 + rnorm(20, mean=0, sd=1)
# let's check the sample means
mean(y[i1==1])  # 3.123994
mean(y[i2==1])  # 4.37477
# now, I'll  fit a model using reference level coding:
m.rlc = lm(y~i2)
summary(m.rlc)  # the intercept = the mean of i1 above, & + the dif = mean of i2
# ...
# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)   3.1240     0.3306   9.450 2.12e-08 ***
# i2            1.2508     0.4675   2.675   0.0154 *  
# ...
coef(m.rlc)[1]+coef(m.rlc)[2]  # 4.37477

# Now I'll try including both the intercept & each indicator variable:
m.mcl = lm(y~i1+i2)
summary(m.mcl)  # R automatically drops i2 b/c of the multicollinearity
# ...
# Coefficients: (1 not defined because of singularities)
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)   4.3748     0.3306  13.233 1.03e-10 ***
# i1           -1.2508     0.4675  -2.675   0.0154 *  
# i2                NA         NA      NA       NA    
# ...
coef(m.mcl)[1]+coef(m.mcl)[2]  # 3.123994
# b/c i2 was dropped, intercept = mean of i2, & + difference = mean of i1

# Here is the model fit using level means coding:
m.lmc = lm(y~0+i1+i2)  # "0+" suppresses the intercept
summary(m.lmc)
# ...
# Coefficients:
#    Estimate Std. Error t value Pr(>|t|)    
# i1   3.1240     0.3306    9.45 2.12e-08 ***
# i2   4.3748     0.3306   13.23 1.03e-10 ***
# ...
# Now both coefficients are the means of the respective levels
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.