How does one interpret regression coefficients when no dummy variables nor intercept are dropped?

Question

I am familiar with how to interpret linear regression coefficients when the independent variables are dummy coded and one of them is dropped. And this question helped me understand how to interpret the coefficients when the intercept is dropped instead.

I am also aware of the dummy variable trap, and why it is necessary to drop one of the dummy coded categories. ( $X^TX$ will not be invertible )

However, I've found that by using regularization $X^TX + \lambda I$ is never singular and therefore invertible. This seems to allow me to not have to drop any dummy variable nor the intercept.

The problem is, I couldn't find a way to interpret the coefficients when neither of those columns are dropped.

e.g.:

Let the dataset be talking about height measurements in males and females and the gender column is dummy coded into $x_f$ and $x_m$.

Let the resulting ridge regression be $\hat{y} = \beta_0 + \beta_fx_f + \beta_m x_m$

How can I interpret the values of $\beta_0$, $\beta_f$ and $\beta_m$?

Does it change if my category has more then 2 possible values?

Does it change if I have multiple categories?

Is this even a valid way of doing regression?

Answer 1

It is certainly a valid way to run a regression. The interpretation of the coefficients in your example ridge regression is simple:

• If you are a male, then your predicted value is $\widehat{y}=\beta_0+\beta_m$
• If you are a female, then your predicted value is $\widehat{y}=\beta_0+\beta_f$
• The predicted difference between males and females is $\beta_f-\beta_m$

Then the interpretation is completely analogous if you have more than one categorical value, or if your variable has more than two categories. For example, if gender had a "not given" category that you wanted to include in your model with $x_n$, then you would simply add that:

• If gender is not given, then your predicted value is $\widehat{y}=\beta_0+\beta_n$
• The predicted difference between males and "not given"'s is $\beta_m-\beta_n$
• The predicted difference between females and "not given"'s is $\beta_f-\beta_n$

And you can keep adding similar examples. There's no limitation on the interpretation of the coefficients because of the intercept/dummy issue.

Answer 2

A trace plot can help elucidate how and why this works-ish

set.seed(123)
n <- 100
x <- rbinom(n, 1, 0.4)
y <- rnorm(n, -3 + 0.5*x)

X <- cbind(1, model.matrix(~-1+factor(x)))
I <- diag(3)
ridge <- function(w) solve(t(X)%*%X + w*I, t(X)%*%y)

matplot(t(sapply(seq(0, 100, by=1)[-1], ridge)), type='l', xlab='Ridge weight', ylab='Parm estimate')
legend('bottomright', lty=1:3, col=1:3, c('Intercept', 'X0', 'X1'))

Gives:

Taking a very small penalty tells us what might be one of the pseudoinverse solutions to the rank deficient solution to the least squares regressors: $\beta = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^TY$.

> ridge(0.001)
                 [,1]
           -1.8752108
factor(x)0 -1.1421614
factor(x)1 -0.7330494

This approximates the left limits of the trace plot I showed above. Note also some empirical statistics about the sample:

> mean(y[x==0])
[1] -3.017391
> mean(y[x==1])
[1] -2.608278
> mean(y[x==0]*mean(x==0))
[1] -1.810435
> mean(y[x==0]*mean(x==1))
[1] -1.206956

You can see the "quasi-intercept" is heading toward about -1.8. This is the prevalence of $\neg X$ multiplied by the mean of $Y$ when $X=0$. e.g. $E[Y|X=0]E[1-X]$ (see the third command). The "quasi-X0" term is the prevalence of $X$ multiplied by the mean of $Y$ when $X=0$ e.g. $E[Y|X=0]E[X]$ (see the fourth command). Together they add up to $E[Y|X=0]$ (see the first command). The "quasi-X1" term is the mean of $Y$ when X=1 minus the intercept term. That is $E[Y|X=1]-E[Y|X=0]E[X=0]$, so subtract the second term from the third term = -2.5 + 1.8 = -0.7.

So, in my opinion these contrasts are utter nonsense! They are redundant and don't tell us anything useful that a simple slope/intercept would.