2
$\begingroup$

Background

Exercise 3.12 of "Elements of Statistical Learning" by by Hastie, Tibshirani, and Friedman reads as follows:

Show that the ridge regression estimates can be obtained by ordinary least squares regression on an augmented data set. We augment the centered matrix $X$ with $p$ additional rows $\sqrt{\lambda}I_p$, and augment $\vec{y}$ with $p$ zeros. By introducing artificial data having response value zero, the fitting procedure is forced to shrink the coefficients toward zero. This is related to the idea of hints due to Abu-Mostafa (1995), where model constraints are implemented by adding artificial data examples that satisfy them.

The solution to this exercise is just a computation: it turns out that $ X_{\textrm{aug}}^\top X_{\textrm{aug}} = X^\top X + \lambda I_p$ and $X_{\textrm{aug}}^\top \vec{y}_{\textrm{aug}} = X^\top \vec{y}$, so that

$$ (X_{\textrm{aug}}^\top X_{\textrm{aug}})^{-1}X_{\textrm{aug}}^\top \vec{y}_{\textrm{aug}} = (X^\top X + \lambda I_p)^{-1} X^\top \vec{y} $$

which is the explicit formula for the ridge regression coefficients.

Questions

This "data augmentation" perspective on ridge regression suggests some generalizations.

  1. Instead of inventing a "fake observation" for every feature, we could instead only invent "fake observations" for features which exhibit multicolinearity.
  2. Instead of choosing the constant value $\sqrt{\lambda}$ along the diagonal, we might choose smaller values for features we deem "more important" and larger values for features we deem "less important".

Questions:

  1. Have either of these ideas been utilized in practice?
  2. Ridge regression is usually formulated as the solution minimizing the modified loss function $$\ell_\alpha = |\vec{y} - X\vec{\beta}|^2 + \alpha |\vec{\beta}|^2$$ Can either of my two generalizations be similarly seen as solutions minimizing some modified loss?
$\endgroup$
5
  • 1
    $\begingroup$ yes totally; in fact your proposition corresponds to only penalizing select elements of $\beta$; i.e. replace $\Vert\beta\Vert_2 = \sum_{i=1}^N\alpha \beta^2_i$ with $\sum_{i\in\mathcal{I}} \alpha \beta_i$. Your second proposition corresponds to replacing the constant $\alpha$ with an index dependent $\alpha_i$. $\endgroup$ Oct 4, 2023 at 17:11
  • $\begingroup$ @JohnMadden Oh ya, I should have seen that! Feeling a little embarrassed. If you do have any references for people actually using these I would be happy to accept your answer! $\endgroup$ Oct 4, 2023 at 17:13
  • 1
    $\begingroup$ I don't know of any direct application off the top of my head; however I'll mention that if you throw half-cauchy hyperpriors on the $\alpha_i$'s, you get an incredibly useful model called the HorseShoe proceedings.mlr.press/v5/carvalho09a/carvalho09a.pdf Incidentally, if you were interested in variable coefficient Lasso rather than Ridge, I'd be able to point you to tons of stuff... it's good for my productivity today that you weren't :) $\endgroup$ Oct 4, 2023 at 17:24
  • 1
    $\begingroup$ @JohnMadden Very cool paper! It also lead me to this: jmlr.org/papers/volume20/19-236/19-236.pdf $\endgroup$ Oct 4, 2023 at 17:32
  • 1
    $\begingroup$ @JohnMadden Your hint showed me how to generalize: augmenting by any $m \times p$ matrix $M$ is equivalent to adding a penalty term of $|M\vec{\beta}|^2$ to the squared error loss function. $\endgroup$ Oct 4, 2023 at 18:00

1 Answer 1

1
$\begingroup$

I am only answering question 2.

Consider the following weighted regularization problem:

Let $X$ be an arbitrary $n \times p$ matrix and $M$ be an arbitrary $m \times p$ matrix. Let $\vec{y} \in \mathbb{R}^n$. We want to minimize

$$\ell_M (\vec{\beta}) = |\vec{y} - X \vec{\beta}|^2 + |M \vec{\beta}|^2 $$

The gradient is

$$ \begin{align*} \nabla \ell_M &= -2X^\top (\vec{y} - X \vec{\beta}) + 2M^\top M \vec{\beta}\\ &= 2(X^\top X + M^\top M) \vec{\beta} - 2X^\top \vec{y} \end{align*} $$

and so we get (assuming invertibility of $X^\top X + M^\top M$)

$$\vec{\beta}_{M} = (X^\top X + M^\top M)^{-1} (X^\top \vec{y})$$

Note that $X^\top X + M^\top M$ is a symmteric positive semi-definite matrix, so choosing $M$ with $M^\top M$ positive definite is especially attractive.

This corresponds to ordinary least squares regression where we let

$$ X_{\textrm{aug}} = \begin{bmatrix} X \\ M\end{bmatrix} \hphantom{sdds}, \hphantom{sdds} \vec{y}_{\textrm{aug}} = \begin{bmatrix} \vec{y} \\\vec{0}_p\end{bmatrix} $$

using exactly the same computation as in the OP.

My "generalization 1" corresponds to taking $M$ to be a diagonal matrix with entries $\sqrt{\lambda}$ in the columns of interest and $0$ otherwise. My "generalization 2" corresponds to taking $M$ to be a positive diagonal matrix.

I am still looking for examples of this kind of thing "in the wild" and would be happy to accept an answer showcasing such!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.