3
$\begingroup$

I'm trying to understand how multiple linear regression with only categorical predictors works. In lectures we were told that, given a data set $(c_1, y_1), (c_2, y_2), ..., (c_n, y_n)$, $c_i \in \{1, 2, ..., K\}$. We can define a model, $j = 1, 2, ..., K$ such that:

$x_{i, j} = 1\ \ if\ \ c_i = j$

$x_{i, j} = 0\ \ otherwise$

The lecturer told us that the two models

$Y_{i} = \beta_0 + \beta_2x_{i, 2} + \beta_3x_{i, 3} + \dots + \beta_K x_{i, K} + \epsilon_i\ \ \ \ \ \ \ \ \ (1)$

and

$Y_i = \alpha_1x_{i, 1} + \alpha_2x_{i, 2} + \dots + \alpha_Kx_{i, K} + \epsilon_i\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ (2)$

are equivalent. Here I assume the usual assumptions on $\epsilon$ are made, i.e. $\epsilon_i\ i.i.d.\ N(0, 1)$. If I understand this model correctly, each row in the design matrix will only have two 1s (one from the column corresponding to the intercept parameter, and one from another column where $c_i = j$). ̃Intuitively, the fact that they are equivalent makes sense, as both systems have $n$ equations and $K$ unknowns (since one parameter is "eliminated"). I also get that when it comes to linear regression with categorical predictors you choose one variable as a reference level. However, how does one prove this equivalence mathematically? Is it possible to express $\beta$ in terms of $\alpha$? I figure this has to do with proving that the linear systems are equivalent, and hence one can be written as a linear combination as the other.

Furthermore, how can one interpret the predictors in these models? I get that, since $Y_i$ is a qualitative response, the parameters say something about how much each category contribute to the response. I understand that this will depend on which variable is the reference level. Additionally, how does this interpretation change if we do not eliminate a predictor and include all from, say $\beta_0$ to $\beta_K$?

I understand this is a lengthy question, and I am truly grateful for any answers to this even if every question is not answered.

$\endgroup$
1
$\begingroup$

There are several questions in your post. As regards the first, I think the easiest way to see that both models (with and without the intercept term) are equivalent is to notice that the columns in both design matrices (the matrices made of predictor values) generate the same vector space.

As for the second question, if $Y_i$ is qualitative you probably should not be fitting an ordinary linear model, but rather a logistic or generalized linear model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.