Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite

What can go wrong if I include two categorical variables and intercept in linear regression?

With:

y~x1+x2

Both x1 and x2 are categorical variables, lets say x1 has 3 levels, x2 has 2 levels.

I encoded them as the following:

x1 corresponds to a design matrix of three columns, each column has 0-1 values indicating whether the observation belong to that level or not.

x2 corresponds to a design matrix of two columns, each column has 0-1 values indicating whether the observation belong to that level or not.

I wanted the create a case which shows "multi-collinearity".

But both linear regressions seem to work fine below.

x1=factor(rep(1:3, 100))
x2=factor(rep(1:2, 150))

y=rnorm(300)

summary(lm(y~x1+x2+1))
summary(lm(y~x1+x2-1))
share|improve this question
multicollinearity is what you get when two or more predictors are highly correlated with each other, but it appears you are generating variables which are not correlated at all. – Michael Bishop Jul 13 '12 at 18:42
But if we put a design matrix for categorical variable and an intercept together, the combined design matrix will be of reduced rank, i.e. rank-difficient. Am I right? – Luna Jul 13 '12 at 23:43
2  
R has outsmarted you :), it treats the two regressions differently. In the first case, it takes the first factor and re-frames it as a smaller number of factors, each representing the difference between a level and the first level (so there are 2 coefficients to estimate in your case instead of 3.) It then does the same for the second. Hence you get an intercept and 3 total coefficients. In the second case, it knows there isn't an intercept, so it doesn't re-frame the first factor, instead leaving it as is. It still re-frames the second factor, though, to avoid the multicollinearity. – jbowman Jul 14 '12 at 0:19
@jbowman, that's a great catch, & I think, exactly the answer to this question. Why don't you turn the comment into an answer? – gung Jul 15 '12 at 1:00
@gung - thanks for the suggestion, I couldn't think of a good reason why not, so I did :) – jbowman Jul 15 '12 at 1:24

1 Answer

up vote 5 down vote

When setting up the two regressions, R treats them differently. In the first (with intercept) case, it takes the first factor and re-frames it as a smaller number of factors, each representing the difference between a level and the first level (so there are 2 coefficients to estimate in your case instead of 3.) This avoids the multicollinearity that would be caused by the "full" representation + an intercept. It then does the same for the second. Hence you get an intercept and 3 total coefficients.

In the second case, it knows there isn't an intercept, so it doesn't re-frame the first factor, instead leaving it as is. It still re-frames the second factor, though, to avoid the multicollinearity.

Here's your example; we can look at the coefficient values to see what's happening:

x1=factor(rep(1:3, 100))
x2=factor(rep(1:2, 150))
y=rnorm(300)

summary(lm(y~x1+x2+1))
 ... stuff ... 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept)  0.13537    0.11742   1.153   0.2499  
x12         -0.20787    0.14381  -1.445   0.1494  
x13         -0.28883    0.14381  -2.008   0.0455 *
x22          0.05656    0.11742   0.482   0.6304  
---

summary(lm(y~x1+x2-1))
 ... stuff removed ...

Coefficients:
    Estimate Std. Error t value Pr(>|t|)
x11  0.13537    0.11742   1.153    0.250
x12 -0.07250    0.11742  -0.617    0.537
x13 -0.15346    0.11742  -1.307    0.192
x22  0.05656    0.11742   0.482    0.630

You can see that the Intercept in the first regression has the same value as x11 in the second. x12 in the first equals x12-x11 in the second, and x13 in the first equals x13-x11 in the second, both as a consequence of representing the factors as differences in the first regression. x22 is the same in both, because including all the levels of the second factor will result in multicollinearity in both regressions.

share|improve this answer
Thanks a lot jbowman. I still don't understand this one "It still re-frames the second factor, though, to avoid the multicollinearity." I thought x2 doesn't need to reframed and should be fine, right? – Luna Jul 17 '12 at 15:06
So the intercept "(Intercept) 0.13537" in case 1 has both x11 and x21 elements in it, because x2 also got reframed? Thank you! – Luna Jul 17 '12 at 15:07
If you have a factor with two levels, you can think of the regression as being run with two 0-1 dummy variables, one for each level - let's call them z1 and z2. Unfortunately, if you add the two dummy variables, you get a vector of all ones, which is the same as the intercept vector - so you have multicollinearity. What R does is create a single dummy equal to z1 - z2 and includes the intercept in the regression. The coefficient of the intercept (in the one-factor example) then becomes that of z1 and the coefficient of the single dummy is the difference between those of z1 and z2. – jbowman Jul 17 '12 at 15:13
(+1) This is good to know. One question: do you know why the standard errors for the x12 and x13 different in the two models? It seems that the models, as you've described them, are mathematically equivalent so I'm not sure why the precision of the estimates of the mean of y|x1=2 and y|x1=3 would differ in the two models. Any ideas? – Macro Aug 16 '12 at 1:53
@Macro - the x12 and x13 in the first model are the same as x12 - x11 and x13 - x11 in the second; it would be nice if R would make explicit in the output that the variables are not the same across the two formulations, although the overall model is mathematically equivalent as you say. – jbowman Aug 16 '12 at 17:05
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.