Here is something I read in Ian Goodfellow's book Deep Learning.

In the context of neural networks, "the L2 parameter norm penalty is commonly known as weight decay. This regularization strategy drives the weights closer to the origin [...]. More generally, we could regularize the parameters to be near any specific point in space" but it is far more common to regularize the model parameters toward zero. (Deep Learning, Goodfellow et al.)

I'm just curious. I understand that by simply adding a regularizing term to our cost function, and that by minimizing this total cost $J$ we can influence the model's parameters to remain small :

$$J(\boldsymbol{\Theta}, \boldsymbol{X}, \boldsymbol{y}) = L(\boldsymbol{\Theta}, \boldsymbol{X}, \boldsymbol{y}) + \lambda||\boldsymbol{w}||_{2}^{2}$$

But how would one implement a version of this regularization strategy that would lead the parameters towards any arbitrary point? (say we want the norm to tend towards 5)


You actually ask two different questions.

  1. Having the norm tend to 5 implies that you want the weights to be near the surface of a hypersphere centered at the origin with radius 5. This regularization looks something like

$$J(\Theta, X, y) = L(\Theta, X, y) + \lambda (||w||_2^2-5)^2$$

But you could instead use something like $\lambda \cdot\text{abs}(||w||_2^2-5)$, I suppose.

  1. On the other hand, if you want to tend towards an arbitrary point, you just need to use that point as the center $c$.

$$J(\Theta, X, y) = L(\Theta, X, y) + \lambda ||w-c||_2^2$$

  • $\begingroup$ (+1) I think a fruitful way to think about the "norm tending to five" could be through the choice of the tuning parameter in the version of $J$ given by OP (rather than changing the function) $\endgroup$ – user795305 Aug 31 '17 at 1:10
  • $\begingroup$ (I've written a short answer to clarify what I mean by above. Thank you, by the way, for clarifying the distinction of the two questions asked!) $\endgroup$ – user795305 Aug 31 '17 at 16:45
  • $\begingroup$ a common (practical) goal when doing that is to regularize towards some known operating point e.g. the previous model that you want to replace but for which you'd like a "smooth" transition $\endgroup$ – oDDsKooL Sep 6 '17 at 8:49

Define $$\hat w_\lambda = \arg\min_w L(\Theta, X, y) + \lambda \|w\|_2^2.$$ We know that $\lim_{\lambda \to \infty} \hat w_\lambda = 0$, due to the penalty $w \mapsto \|w\|_2^2$ having the origin as its minimizer.

Sycorax points out that, similarly, $\lim_{\lambda \to \infty} \left\{ \arg\min_w L(\Theta, X, y) + \lambda \|w-c\|_2^2 \right\} = c.$ This successful generalization may lead us to propose the estimator $$\tilde w_\lambda = \arg\min_w L(\Theta, X, y) + \lambda \mathrm{pen}(w),$$ where $\mathrm{pen}$ is a function whose minimizer satisfies some property that we seek. Indeed, Sycorax takes $\mathrm{pen}(w) = g(\|w\|_2^2 - 5)$, where $g$ is (uniquely) minimized at the origin, and, in particular, $g \in \{|\cdot|, \, (\cdot)^2\}$. Therefore $\lim_{\lambda \to \infty} \|\tilde w_\lambda \|_2^2 = 5$, as desired. Unfortunately, though, both of the choices of $g$ lead to penalties which are nonconvex, leading the estimator to be difficult to compute.

The above analysis seems to be the best solution (maybe up to the choice of $g$, for which I have no better one to suggest) if we insist on $\lambda \to \infty$ as being the unique interpretation of "tends to" described in the question. However, assuming that $\|\arg\min_w L(\Theta, X, y) \|_2^2 \geq 5$, there exists some $\Lambda$ so that the minimizer $\hat w_\Lambda$ of OP's problem satsifes $\|\hat w_\Lambda\|_2^2 = 5$. Therefore $$\lim_{\lambda \to \Lambda} \left\| \hat w_\lambda \right\|_2^2 = 5,$$ without needing to change the objective function. If no such $\Lambda$ exists, then the problem of computing $\arg\min_{w : \|w\|_2^2 = 5} L(\Theta, X, y)$ is intrinsically difficult. Indeed, there's no need to consider any estimator besides $\hat w_\lambda$ when trying to encourage natural properties of $\|\hat w_\lambda\|_2^2$.

(To enforce that a penalized estimator attains a value of the penalty which is not achieved by the unpenalized estimator seems highly unnatural to me. If anyone is aware of any places where this is in fact desired, please do comment!)

  • 1
    $\begingroup$ This is an excellent addition. +1 $\endgroup$ – Sycorax Aug 31 '17 at 17:32

For appropriate $L$ it is possible to view it as negative log-likelihood and appropriate regularization $J$ can be viewed as negative log-likelihood for prior distribution. This approach is called Maximum A Posteriori (MAP).

It should be easy to see Sycorax's examples in the light of MAP.

For details of MAP you can look at these notes. From my experience googling 'maximum a posteriori regularization' gives good results.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.