# Do linear models with shrinkage maintain the "controlling for" property of the predictors?

Suppose I run a linear model with shrinkage, say elastic net, and only 2 coefficients remain. The model then looks like

$$y = \beta_1 x_1 + \beta_2 x_2 + \epsilon$$

Can I say that $\beta_1$ is the impact of $x_1$ controlling for $x_2$, like I would if this was an ordinary least squares fit? I feel like because of all the shrinkage it's no longer an orthogonal projection so that interpretation is not accurate.

• Without any question, this model "controls for" $x_2$. The question appears ambiguous, though, because it seems to be asking about the method of fitting the model rather than the model itself. Do you conceive of the $\beta_i$ as taking on the values estimated by the Elastic Net or do you re-estimate them using least squares after having selected the two variables? Your language suggests the former, but your reference to the model makes one wonder whether it's the latter you have in mind.
– whuber
Commented Jul 25, 2017 at 19:55
• What I meant is that after fitting with an elastic net, if the only two nonzero coefficients are $\beta_1$ and $\beta_2$, do they have the same interpretation as if I were to fit an OLS regression with only these two variables? Commented Jul 25, 2017 at 22:36
• Well, one would think that the interpretation ought to depend on the model rather than the fitting procedure. This suggests that we write down and compare the two models--one for Elastic Net, another for OLS--if indeed any difference at all can be discerned between them. This could be done in many ways. Which ones do you have in mind?
– whuber
Commented Jul 25, 2017 at 22:41
• Under OLS if you want to account for the effect of a second predictor you add it to the model. Then the effect of the second predictor on the first is removed and so you can claim that you are now controlling for the effects of the second predictor. Suppose now that you have an elastic net with many predictors but only two remain. Can you make the same claim - that the effect of the second predictor on the first has been removed? Commented Jul 25, 2017 at 22:56
• I don't see why not. With the ElasticNet solution, all that differs from OLS (where it appears you feel comfortable making the claim that effects have been controlled) is that your estimates of the effects might differ from estimates obtained by minimizing the sum of squared residuals. It would appear that both methods have valid claims to "removing" the effects of one variable from the other. The non-orthogonal projection in the EN does suggest you should take care, when explaining or interpreting "removing," not to appeal to properties unique to the OLS solution, though.
– whuber
Commented Jul 26, 2017 at 13:42

$$\newcommand{\l}{\lambda}$$Consider a linear model $$Y = X\beta + \varepsilon$$ where $$X$$ is full rank. We all know the standard interpretation of $$\hat \beta_j$$: it is the estimated effect on the response for a unit increase in $$X_j$$ when the other predictors are held fixed.

First I will show where in the math this interpretation comes from, and then I will apply that analysis to ridge regression (I'm not considering the lasso right now).

We know $$\hat \beta = (X^T X)^{-1}X^T Y$$. Consider a partition of $$X$$ so that we have $$X = \left(Z \ \vert\ W \right)$$ where $$Z \in \mathbb R^n$$ is the first column of $$X$$ and $$W \in \mathbb R^{n \times (p-1)}$$ is the remaining $$p-1$$ columns. We could shuffle $$X$$ before this partitioning so in effect $$Z$$ represents an arbitrary column. We will work out the estimated coefficient $$\hat \beta_Z$$ of $$Z$$.

From its block form we know that $$X^TX = \left[ \begin{array}{c|c} Z^TZ & Z^TW \\ \hline W^TZ & W^TW \end{array} \right]$$ By assumption $$X$$ is full column rank so $$X^TX$$ is invertible, and this means that $$W$$ is full column rank too so $$W^TW$$ is also invertible.

Let $$H = W(W^TW)^{-1}W^T$$ be the projection matrix onto the column space of $$W$$. Then by the inverse of a block diagonal matrix we have $$(X^TX)^{-1} = \left[ \begin{array}{c|c} (Z^T(I-H)Z)^{-1} & -(Z^T(I-H)Z)^{-1}Z^TW(W^TW)^{-1} \\ \hline C & D \end{array} \right].$$ We only care about the first row of $$(X^TX)^{-1}$$ since that's all we'll need for $$\hat \beta_Z$$ so we omit the details of $$C$$ and $$D$$.

Note that $$Z^T(I-H)Z = 0$$ iff $$Z$$ lies completely in the column space of $$W$$, which is impossible by the assumption that $$X$$ is full rank.

This means that $$\hat \beta_Z = \frac{Z^TY}{Z^T(I-H)Z} - \frac{Z^THY}{Z^T(I-H)Z} = \frac{Z^T(I-H)Y}{Z^T(I-H)Z} = \frac{e^TY}{e^Te}$$ where $$e := (I-H)Z$$ is the residual vector from regressing $$Z$$ on $$W$$. I'm using the idempotence of $$H$$ to conclude that $$Z^T(I-H)Z = \|(I- H)Z\|^2 = \|e\|^2$$. All together, we see that $$\hat \beta_Z$$ is equal to the result of a simple linear regression of $$Y$$ on the part of $$Z$$ that is not explained by $$W$$. This is where we get the interpretation that the coefficient $$\hat \beta_Z$$ is the effect of $$Z$$ after controlling for the other predictors.

Now that we have a mathematical understanding of where this comes from, we can apply this to ridge regression to see how things change. The ridge regression estimate is

$$\hat \beta^{(\lambda)} = (X^T X + \lambda I)^{-1}X^T Y.$$

We can recompute $$(X^TX + \lambda I)^{-1}$$ in the same manner as before. Let $$H_\lambda = W(W^TW + \lambda I)^{-1}W^T$$ be the hat matrix for a ridge regression onto the columns of $$W$$. We find that $$(X^TX + \lambda I)^{-1} = \left[ \begin{array}{c|c} \left(Z^T(I - H_\lambda)Z + \lambda\right)^{-1} & -\left(Z^T(I - H_\lambda)Z + \lambda\right)^{-1}Z^TW(W^TW + \lambda I)^{-1} \\ \hline C' & D' \end{array} \right].$$ where again $$C'$$ and $$D'$$ don't matter. From this we find $$\hat \beta_Z^{(\lambda)} = \frac{Z^T(I - H_\lambda) Y}{Z^T(I-H_\lambda)Z + \lambda}.$$ The interpretation here isn't quite so clean because now $$H_\lambda$$ and in turn $$I - H_\lambda$$ are no longer idempotent, so $$Z^T(I-H_\lambda)Z$$ is not exactly the norm of the residuals of a ridge regression of $$Z$$ on $$W$$. It's still similar enough though that we could just not worry about the loss of idempotence, but an alternative interpretation is to note that this is the result of a generalized least squares simple linear ridge regression weighted with $$(I-H_\lambda)^{-1}$$. I can use the Woodbury matrix identity to rearrange this as $$(I - H_\lambda)^{-1} = \lambda^{-1}\left(WW^T + \lambda I\right)$$ so letting $$\Omega_\l = WW^T + \l I$$ I have $$\hat\beta_Z^{(\l)} = \frac{Z^T\Omega_\l^{-1} Y}{Z^T\Omega_\l^{-1}Z + \lambda}.$$ What I like about this form is that this is the solution to the penalized Mahalanobis distance problem $$\underset{b \in \mathbb R}{\text{argmin }} (Y - b Z)^T\Omega_\l^{-1}(Y - bZ) + \l b^2$$ and the GLS part of this is what I'd get from having an error term with covariance matrix $$WW^T + \l I$$ which is like a similarity matrix (this is also the sort of covariance matrix that arises from a 1-way random intercepts model).

Let $$W = U\tilde DV^T$$ be the "wide" SVD of $$W$$ so that $$U$$ is $$n\times n$$ and actually is orthonormal, and $$\tilde D = {D \choose \mathbf 0}$$ is $$n\times p$$. Then $$WW^T + \l I = U\left(\begin{array}{c|c}D^2 + \l I_p& \mathbf 0 \\ \hline \mathbf 0 & \l I_{n-p}\end{array}\right)U^T$$ so for a particular $$b \in \mathbb R$$ under consideration I'll get a working residual vector $$r = Y - b Z$$ and I measure its goodness of fit via $$r^T\Omega_\l^{-1} r = (U^Tr)^T \left(\begin{array}{c|c}D^2 + \l I_p& \mathbf 0 \\ \hline \mathbf 0 & \l I_{n-p}\end{array}\right)^{-1} (U^Tr) \\ = \sum_{i=1}^p \frac{c_i^2}{d_i^2 + \l} + \l^{-1}\sum_{j=p+1}^n c_i^2$$ where $$c = U^T r$$ is $$r$$ expressed w.r.t. the basis $$U$$. This will be small when $$c$$ has most of its "mass" in the first few coordinates since those have the largest $$d_i^2$$, so a good $$b$$ is not one that makes $$\|Y - b Z\|^2$$ small but rather one that gets $$Y - b Z$$ mostly into the span of the first few columns of $$U$$. From the low-rank approximation view of the SVD I can think of these vectors as being the ones that best approximate the column space of $$W$$ which is what linear regression is all about.

Thus to summarize all this, this Mahalanobis interpretation lets us see $$W$$ as being "controlled for" since $$Z$$ explaining $$Y$$ actually just requires explaining the part of $$Y$$ that's not well within the column space of $$W$$. And then that coefficient is further shrunken.

You ask whether or not orthogonal projections play a role here and my many references to idempotence indicate that they do indeed matter. When $$\lambda=0$$ so that $$I - H_\lambda$$ really is an orthogonal projection, the interpretation of $$(I-H)Z$$ is crystal clear: we know that we can decompose $$Z = HZ + (I-H)Z$$, so we are just taking the part of $$Z$$ that is completely orthogonal to the column space of $$W$$. When $$\lambda > 0$$ we still have $$Z = H_\lambda Z + (I-H_\lambda)Z$$ but this doesn't correspond to an orthogonal decomposition, so $$(I-H_\lambda)Z$$ can lie a little bit in the column space of $$W$$. We still can view this as controlling for $$W$$, because we're controlling for the effect of $$W$$ as estimated with this model, but it's not as tidy as when we really are taking the part of $$Z$$ that is completely orthogonal to $$W$$ and the residuals increasingly resemble $$Z$$ so the less flexible our model is the less we're able to control, but it is still happening.

Hope this helps!