How to implement L2 regularization towards an arbitrary point in space?

Question

Here is something I read in Ian Goodfellow's book Deep Learning.

In the context of neural networks, "the L2 parameter norm penalty is commonly known as weight decay. This regularization strategy drives the weights closer to the origin [...]. More generally, we could regularize the parameters to be near any specific point in space" but it is far more common to regularize the model parameters toward zero. (Deep Learning, Goodfellow et al.)

I'm just curious. I understand that by simply adding a regularizing term to our cost function, and that by minimizing this total cost $J$ we can influence the model's parameters to remain small :

$$J(\boldsymbol{\Theta}, \boldsymbol{X}, \boldsymbol{y}) = L(\boldsymbol{\Theta}, \boldsymbol{X}, \boldsymbol{y}) + \lambda||\boldsymbol{w}||_{2}^{2}$$

But how would one implement a version of this regularization strategy that would lead the parameters towards any arbitrary point? (say we want the norm to tend towards 5)

Answer 1

You actually ask two different questions.

1. Having the norm tend to 5 implies that you want the weights to be near the surface of a hypersphere centered at the origin with radius 5. This regularization looks something like

$$J(\Theta, X, y) = L(\Theta, X, y) + \lambda (||w||_2^2-5)^2$$

But you could instead use something like $\lambda \cdot\text{abs}(||w||_2^2-5)$, I suppose.

2. On the other hand, if you want to tend towards an arbitrary point, you just need to use that point as the center $c$.

$$J(\Theta, X, y) = L(\Theta, X, y) + \lambda ||w-c||_2^2$$

Answer 2

Define $$\hat w_\lambda = \arg\min_w L(\Theta, X, y) + \lambda \|w\|_2^2.$$ We know that $\lim_{\lambda \to \infty} \hat w_\lambda = 0$, due to the penalty $w \mapsto \|w\|_2^2$ having the origin as its minimizer.

Sycorax points out that, similarly, $\lim_{\lambda \to \infty} \left\{ \arg\min_w L(\Theta, X, y) + \lambda \|w-c\|_2^2 \right\} = c.$ This successful generalization may lead us to propose the estimator $$\tilde w_\lambda = \arg\min_w L(\Theta, X, y) + \lambda \mathrm{pen}(w),$$ where $\mathrm{pen}$ is a function whose minimizer satisfies some property that we seek. Indeed, Sycorax takes $\mathrm{pen}(w) = g(\|w\|_2^2 - 5)$, where $g$ is (uniquely) minimized at the origin, and, in particular, $g \in \{|\cdot|, \, (\cdot)^2\}$. Therefore $\lim_{\lambda \to \infty} \|\tilde w_\lambda \|_2^2 = 5$, as desired. Unfortunately, though, both of the choices of $g$ lead to penalties which are nonconvex, leading the estimator to be difficult to compute.

The above analysis seems to be the best solution (maybe up to the choice of $g$, for which I have no better one to suggest) if we insist on $\lambda \to \infty$ as being the unique interpretation of "tends to" described in the question. However, assuming that $\|\arg\min_w L(\Theta, X, y) \|_2^2 \geq 5$, there exists some $\Lambda$ so that the minimizer $\hat w_\Lambda$ of OP's problem satsifes $\|\hat w_\Lambda\|_2^2 = 5$. Therefore $$\lim_{\lambda \to \Lambda} \left\| \hat w_\lambda \right\|_2^2 = 5,$$ without needing to change the objective function. If no such $\Lambda$ exists, then the problem of computing $\arg\min_{w : \|w\|_2^2 = 5} L(\Theta, X, y)$ is intrinsically difficult. Indeed, there's no need to consider any estimator besides $\hat w_\lambda$ when trying to encourage natural properties of $\|\hat w_\lambda\|_2^2$.

(To enforce that a penalized estimator attains a value of the penalty which is not achieved by the unpenalized estimator seems highly unnatural to me. If anyone is aware of any places where this is in fact desired, please do comment!)

Answer 3

For appropriate $L$ it is possible to view it as negative log-likelihood and appropriate regularization $J$ can be viewed as negative log-likelihood for prior distribution. This approach is called Maximum A Posteriori (MAP).

It should be easy to see Sycorax's examples in the light of MAP.

For details of MAP you can look at these notes. From my experience googling 'maximum a posteriori regularization' gives good results.