3
$\begingroup$

I am familiar with how to interpret linear regression coefficients when the independent variables are dummy coded and one of them is dropped. And this question helped me understand how to interpret the coefficients when the intercept is dropped instead.

I am also aware of the dummy variable trap, and why it is necessary to drop one of the dummy coded categories. ( $X^TX$ will not be invertible )

However, I've found that by using regularization $X^TX + \lambda I$ is never singular and therefore invertible. This seems to allow me to not have to drop any dummy variable nor the intercept.

The problem is, I couldn't find a way to interpret the coefficients when neither of those columns are dropped.

e.g.:

Let the dataset be talking about height measurements in males and females and the gender column is dummy coded into $x_f$ and $x_m$.

Let the resulting ridge regression be $\hat{y} = \beta_0 + \beta_fx_f + \beta_m x_m$

How can I interpret the values of $\beta_0$, $\beta_f$ and $\beta_m$?

Does it change if my category has more then 2 possible values?

Does it change if I have multiple categories?

Is this even a valid way of doing regression?

$\endgroup$
  • 1
    $\begingroup$ Why should the interpretation depend on the procedure you used to fit the model? The underlying model hasn't changed. Thus, the predicted value for males is $\beta_0+\beta_m$ while the predicted value for females is $\beta_0+\beta_f,$ just as always (assuming indicator coding for the dummies). $\endgroup$ – whuber Mar 21 '18 at 22:03
  • $\begingroup$ Even though the underlying model hasn't changed, the interpretation of the meaning of the values of the beta coefficients varies drastically depending on how the model came to be. For instance, when you drop one of the dummy variables, your coefficients are now referencing changes from that variable. $\endgroup$ – wmaciel Mar 21 '18 at 23:05
  • 1
    $\begingroup$ That may be so, but it doesn't create any new difficulties. The interpretation of any coefficient remains the same: plug in two values for one regressor (such as gender), compute the corresponding predictions, and subtract: that's how the prediction changes with a change in the regressor. Without the ridge regression your model would be unidentifiable, but it could still be fit and the coefficient interpretations would be exactly the same. $\endgroup$ – whuber Mar 21 '18 at 23:14
  • $\begingroup$ Your comment was quite helpful in solving my underlying problem of how to explain the rate of change caused by the independent variables. If you write an answer about it I would totally upvote it. However, it doesn't really answer the core of the question which is if the $\beta$ coefficients have any real meaning. I highly appreciate you taking the time to comment. $\endgroup$ – wmaciel Mar 21 '18 at 23:54
  • 1
    $\begingroup$ That is a good point, and I agree with you now. I think your comment answers my question well. In the sense that the $\beta$ coefficients have no inherent meaning in this case, and should only be "interpreted" in a difference scenario. $\endgroup$ – wmaciel Mar 22 '18 at 0:54
6
$\begingroup$

It is certainly a valid way to run a regression. The interpretation of the coefficients in your example ridge regression is simple:

  • If you are a male, then your predicted value is $\widehat{y}=\beta_0+\beta_m$
  • If you are a female, then your predicted value is $\widehat{y}=\beta_0+\beta_f$
  • The predicted difference between males and females is $\beta_f-\beta_m$

Then the interpretation is completely analogous if you have more than one categorical value, or if your variable has more than two categories. For example, if gender had a "not given" category that you wanted to include in your model with $x_n$, then you would simply add that:

  • If gender is not given, then your predicted value is $\widehat{y}=\beta_0+\beta_n$
  • The predicted difference between males and "not given"'s is $\beta_m-\beta_n$
  • The predicted difference between females and "not given"'s is $\beta_f-\beta_n$

And you can keep adding similar examples. There's no limitation on the interpretation of the coefficients because of the intercept/dummy issue.

$\endgroup$
  • $\begingroup$ I understand that $\hat{y} = \beta_0 + \beta_m$ and so forth, it makes sense. What I am struggling to understand is how to interpret the $\beta$s by themselves. Especially $\beta_0$. e.g. When you drop the male dummy variables $\beta_0$ is the average $\hat{y}$ when the $x_f = 0$. And $\beta_f$ is the average variation in $\hat{y}$ when $x_f = 1$. $\endgroup$ – wmaciel Mar 21 '18 at 23:02
  • 1
    $\begingroup$ If people can be only female or male, such that $x_m+x_f=1$, then $\beta_0$ on its own has no clear interpretation. This does not mean in the least that the model is incorrectly specified or anything like that. It just means the number on its own means nothing. Think of it this way: if I tell you that that the price of a ticket is $Price=10+2*(Male)+4*(Female)$ (i.e., males pay \$12 and females pay \$14), what does the 10 represent? It's a perfectly valid equation and yet the 10 doesn't have a clear interpretation. $\endgroup$ – user3208442 Mar 22 '18 at 0:32
  • $\begingroup$ @user3208442 excellent job pointing out how two rank-dependent(nonsingular) covariates can be combined to estimate the usual "slope" term. One thought, how does one get the variance of ridge estimates? I'm curious if the post-hoc estimate yields similar variances to the full-rank solution. $\endgroup$ – AdamO Mar 22 '18 at 16:48
  • $\begingroup$ I had to look this up because I've never needed to get the variance of a ridge coefficient :-) Here's a link to some slides with the formula; check slide 7: web.as.uky.edu/statistics/users/pbreheny/764-F11/notes/9-1.pdf. $\endgroup$ – user3208442 Mar 22 '18 at 17:05
0
$\begingroup$

A trace plot can help elucidate how and why this works-ish

set.seed(123)
n <- 100
x <- rbinom(n, 1, 0.4)
y <- rnorm(n, -3 + 0.5*x)

X <- cbind(1, model.matrix(~-1+factor(x)))
I <- diag(3)
ridge <- function(w) solve(t(X)%*%X + w*I, t(X)%*%y)

matplot(t(sapply(seq(0, 100, by=1)[-1], ridge)), type='l', xlab='Ridge weight', ylab='Parm estimate')
legend('bottomright', lty=1:3, col=1:3, c('Intercept', 'X0', 'X1'))

Gives:

enter image description here

Taking a very small penalty tells us what might be one of the pseudoinverse solutions to the rank deficient solution to the least squares regressors: $\beta = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^TY$.

> ridge(0.001)
                 [,1]
           -1.8752108
factor(x)0 -1.1421614
factor(x)1 -0.7330494

This approximates the left limits of the trace plot I showed above. Note also some empirical statistics about the sample:

> mean(y[x==0])
[1] -3.017391
> mean(y[x==1])
[1] -2.608278
> mean(y[x==0]*mean(x==0))
[1] -1.810435
> mean(y[x==0]*mean(x==1))
[1] -1.206956

You can see the "quasi-intercept" is heading toward about -1.8. This is the prevalence of $\neg X$ multiplied by the mean of $Y$ when $X=0$. e.g. $E[Y|X=0]E[1-X]$ (see the third command). The "quasi-X0" term is the prevalence of $X$ multiplied by the mean of $Y$ when $X=0$ e.g. $E[Y|X=0]E[X]$ (see the fourth command). Together they add up to $E[Y|X=0]$ (see the first command). The "quasi-X1" term is the mean of $Y$ when X=1 minus the intercept term. That is $E[Y|X=1]-E[Y|X=0]E[X=0]$, so subtract the second term from the third term = -2.5 + 1.8 = -0.7.

So, in my opinion these contrasts are utter nonsense! They are redundant and don't tell us anything useful that a simple slope/intercept would.

$\endgroup$
  • $\begingroup$ I am sorry, I don't quite understand what you mean. - Could you elaborate on what is going on in that R snippet? - What are the "prevalence" comments you had about where the intercept was heading. I am very lost in your answer, sorry. $\endgroup$ – wmaciel Mar 21 '18 at 21:49
  • $\begingroup$ @wmaciel I edited to include more detail. If you're still unclear, please say more. $\endgroup$ – AdamO Mar 21 '18 at 21:59
  • $\begingroup$ I've followed your calculations, I think I understand what you did and I highly appreciate your time on this. However, I don't see how you concluded that the coefficients don't tell us anything from the math you showed on your answer. $\endgroup$ – wmaciel Mar 21 '18 at 23:17
  • $\begingroup$ @wmaciel thanks for clarifying. I was trying to point out that the interpretation is very strange. The X1 coefficient has a negative slope when X has a positive trend with Y. To summarize, the coefficients do have an interpretation. But it is far afield from normal parametrizations. $\endgroup$ – AdamO Mar 22 '18 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.