2
$\begingroup$

When performing regression with categorical variables, in order to avoid multicollinearity, it is necessary to drop one level. This is clear in fact:
Let's assume I have a binary categorical variable (A, B) and the following data:

id, cat, y
1    A  y_1
2    B  y_2
3    B  y_3
4    A  y_4
...

where id is just an index a y the target variable, the design matrix (with the intercept), the design matrix will look like the following (dropping A and setting it as the base level):

id, intercept, cat_B, y
1    1          0  y_1
2    1          1  y_2
3    1          1  y_3
4    1          0  y_4
...

This makes perfect sense as, if we keep both the indicator for A and B, the column for A (cat_A) will be linearly dependent to the column (cat_B) and the matrix is not invertible.

What I do not understand is that if we do not have an intercept we do not have to drop one level and we have the following:

  id, cat_A, cat_B, y
    1    1      0  y_1
    2    0      1  y_2
    3    0      1  y_3
    4    1      0  y_4
    ...

and now the linear regression can be performed. Cat_A column and Cat_B column are still linearly dependent. Why this is not a problem anymore?

Improve this question
$\endgroup$
6
  • $\begingroup$ At the end, clearly those two columns are not linearly dependent. What is your definition of "linearly dependent," then? $\endgroup$
    – whuber ♦
    Commented Dec 6, 2023 at 15:18
  • $\begingroup$ what do you mean? cat_A = 1 - cat_B. it seems they are, right? $\endgroup$
    – Marco De Virgilis
    Commented Dec 7, 2023 at 14:02
  • 1
    $\begingroup$ That's not linear dependence! That is an affine relationship. You have to make two assumptions to arrive at your conclusion: first, that "1" refers to the constant vector whose components are 1s; second, that this vector is in your design matrix. The latter is the case when an intercept is included in the model, but it is usually not the case otherwise. $\endgroup$
    – whuber ♦
    Commented Dec 7, 2023 at 14:42
  • 1
    $\begingroup$ en.wikipedia.org/wiki/Linear_independence $\endgroup$
    – Christoph Hanck
    Commented Dec 8, 2023 at 14:54
  • 2
    $\begingroup$ For the record, a nonzero linear combination of a $X\subset\mathbb V$ of vectors $S$ in in a vector space $\mathbb V$ is a sum $$\sum_{\mathbf v\in X}\alpha_v\,\mathbf v$$ where (i) only finitely many of the coefficients $\alpha_v$ are nonzero and (ii) at least one coefficient is nonzero. $X$ is called linearly dependent when there exists a nonzero linear combination equal to $\mathbf 0.$ $X$ is linearly independent when it is not linearly dependent. In your first comment, if $\mathbf 1$ is not in the column space of the design matrix, then $1-\text{cat}_B$ is not a linear combination. $\endgroup$
    – whuber ♦
    Commented Dec 8, 2023 at 15:35

1 Answer 1

Reset to default
5
$\begingroup$

What matters for multicollinearity is whether or not we can linearly combine some columns into another one, or equivalently combine all of them nontrivially into the zero vector, see e.g. Perfect multicollinearity with a cubic term in the model?. If that is possible, the regressor matrix does not have full column rank anymore.

Here, we would have cat_A+cat_b=intercept and hence multicollinearity in that case.

On the other hand, if we drop the intercept like in your last display, the column rank of the resulting matrix is two and thus equal to the number of regressors, hence no problem anymore.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.