3
$\begingroup$

In my question (Cox model on bank customers) regarding the estimation process in regression with categorical variables, @Scortchi write the following:

Any coefficient in a multiple regression model represents the relationship between a predictor & the response holding constant all the other predictors in the model. So not only the point estimate, p-value, &c. of a coefficient change when you add in other terms, but also its interpretation.

This, I understand, in terms of interpreting the coefficients. Holding every predictor constant but one, and varying this one predictor, gives us the relationship with the response. He kindly gave me some links to other posts, and especially this one (Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?) peaked my interest. @gung explains how one are controlling for a second variable when regressing on a first variable (if I formulated this correctly).

Now, I understand Ordinary Least Squares in the following way: $y=X\beta$, where $X$ is the $n\times m$ matrix of coefficients. This equation does not have a solution (y is not in the column space of X), so we must project y down on the column space of X - to get $\hat{y}$ - which is the OLS solution (with $m=2$):

Projection of Y into Col(x)

This gives us $\hat{\beta}=(X^TX)^{-1}X^Ty$.

Now I am having difficulties understanding how this is done with categorical variables. I have read about dummy coding, and that a categorical variable with $k$ levels is divided into $k-1$ dummy variables and so on, but I fail to see how this is actually implemented with regard to the actual OLS estimation (formulas above). How would the matrix of coefficients above look, if we are dealing with a categorical variable and dummy coding?

In @gung's answer (link above) he shows us how controlling for a second variable means we are fitting a plane, instead of a line. Now this plane he mentions, is this the (imagined) tilted (upwards and to the right) plane connecting the red, blue and green marks in his second plot(below)?

Gungs plot

(OBS: this plot is from @gung here: Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?)

How does this (imagined) tilted plane relate with the OLS estimation formula $\hat{\beta}=(X^TX)^{-1}X^Ty$?

Also, I asked @gung the following in a comment:

... Regarding your first plot, we have three OLS lines, each giving us a value for $\beta_1$. What happens if these three values of $\beta_1$ are different?...

I imagine such a plane, if I understand anything at all, would be "bent" (and some extra dimension would come into play). He's answer is:

...Briefly, when you have 2 X variables, you are fitting a plane instead of a line. If the appropriate slope of the y~x2 relationship changes as x1 increases, that means there is an interaction b/t x1&x2.

$\endgroup$
4
  • $\begingroup$ Would this (imagined) tilted plane be the equivalent to the two dimensional plane in my OLS picture above I'm not sure I interpreted your first picture right but it looks much like a representation of variables in "the subject space": vectors are variables. Your second plot is subjects in "variable space", a usual scatterplot. NO, the plane X of the first and the prediction plane Y-hat of the second are different planes. Here one answer uses "subject space" representation and the other uses "variable space" in discussing the same phenomenon. $\endgroup$
    – ttnphns
    Apr 17 '15 at 10:49
  • $\begingroup$ Yes I saw this question was very ill-posed, and just edited it. Indeed two different spaces. Thank you for commenting on that! $\endgroup$
    – Erosennin
    Apr 17 '15 at 10:54
  • 1
    $\begingroup$ How does this (imagined) tilted plane relate with the OLS estimation formula Perform regression of Y on X1 for slice X2=1 and draw the prediction line (parallel to X1). Do likewise for X2=2, draw the line. Do likewise for X2=3, draw the line. Look at the three lines from the front (from X1): they will be at somewhat different angle with X1 axis. Do (weighted) averaging of the 3 lines. The resultant line's angle with X1 represents regression coefficient for X1 (b1) in the multiple regression Y~X1+X2, and it is the angle of inclination of the plane Y-hat you are speaking about to X1. $\endgroup$
    – ttnphns
    Apr 17 '15 at 11:24
  • $\begingroup$ If the angle of inclination was at the same angle with the X1 axis, for all X2's, then weighted average would not be necessary, correct? This is what I took from the post of @gung, at least. And the inclination would be of the imagined tilted plane I was talking about - to X1. Is this imagined plane called Y-hat? $\endgroup$
    – Erosennin
    Apr 17 '15 at 13:31
3
$\begingroup$

Just to answer one part of your question:

Now I am having difficulties understanding how this is done with categorical variables. I have read about dummy coding, and that a categorical variable with k levels is divided into k−1 dummy variables and so on, but I fail to see how this is actually implemented with regard to the actual OLS estimation (formulas above). How would the matrix of coefficients above look, if we are dealing with a categorical variable and dummy coding?

Hopefully this block of code will help. Look at the X matrix:

set.seed(123987)
n    <- 6
df   <- data.frame(x=runif(n), categorical=factor(letters[1:3]))
df$y <- rnorm(n) + df$x + ifelse(df$categorical == "a", 0,
                                 ifelse(df$categorical == "b", 2, 10))
fit  <- lm(y ~ x + categorical, data=df)
fit$coefficients  # Around -0.1, 2.5, 1.1 and 10.3

X      <- matrix(1, nrow=n, ncol=length(fit$coefficients))
X[, 2] <- df$x
X[, 3] <- 1*(df$categorical == "b")
X[, 4] <- 1*(df$categorical == "c")
colnames(X) <- c("constant", "x", "indicator for b", "indicator for c")  # Aka dummies
Y <- matrix(df$y, ncol=1)

beta_hat <- as.vector(solve(t(X) %*% X) %*% t(X) %*% Y)
max(abs(beta_hat - fit$coefficients))          # Very small -- essentially equal
isTRUE(all.equal(beta_hat, fit$coefficients))  # ...well, not equal enough for all.equal

The matrix X has one column of 1s (the constant); a column of df$x (a continuous predictor); a column that is 1 when the categorical variable equals "b", and zero otherwise; and similarly for "c". The value "a" is omitted, since we have a constant.

Edit: spacing in the code block is messed up for some reason, not sure why.

$\endgroup$
1
  • 1
    $\begingroup$ The code block is messed up because the dollar signs in the code interact w/ the Mathjax running in the background (see here). $\endgroup$ Apr 17 '15 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.