9
$\begingroup$

In the week 3 lecture notes of Andrew Ng's Coursera Machine Learning class, a term is added to the cost function to implement regularisation:

$$J^+(\theta) = J(\theta) + \frac{\lambda}{2m} \sum_{j=1}^n \theta_j^2$$

The lecture notes say:

We could also regularize all of our theta parameters in a single summation:

$$min_\theta\ \dfrac{1}{2m}\ \left[ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})^2 + \lambda\ \sum_{j=1}^n \theta_j^2 \right] $$

$\frac 1 {2m}$ is later applied to the regularisation term of neural networks:

Recall that the cost function for regularized logistic regression was:

$$J(\theta) = - \frac{1}{m} \sum_{i=1}^m [ y^{(i)}\ \log (h_\theta (x^{(i)})) + (1 - y^{(i)})\ \log (1 - h_\theta(x^{(i)}))] + \frac{\lambda}{2m}\sum_{j=1}^n \theta_j^2$$

For neural networks, it is going to be slightly more complicated: $$\begin{gather*} J(\Theta) = - \frac{1}{m} \sum_{i=1}^m \sum_{k=1}^K \left[y^{(i)}_k \log ((h_\Theta (x^{(i)}))_k) + (1 - y^{(i)}_k)\log (1 - (h_\Theta(x^{(i)}))_k)\right] + \frac{\lambda}{2m}\sum_{l=1}^{L-1} \sum_{i=1}^{s_l} \sum_{j=1}^{s_{l+1}} ( \Theta_{j,i}^{(l)})^2\end{gather*} $$

  • Why is the constant one-half used here? So that it is cancelled in the derivative $J'$?
  • Why the division by $m$ training examples? How does the amount of training examples affect things?
$\endgroup$
  • $\begingroup$ are you sure the 1/m is on the regularisation and not on the J(theta) AFAIK @DikranMarsupial 's answer is making that assumption...... or does J(theta) itself have a 1/m term? $\endgroup$ – seanv507 Jul 16 '17 at 13:29
  • $\begingroup$ That assumption is incorrect - $ 1 \over 2m$ is applied to both the un-regularised cost function and the regularisation term. I've updated the question to give the full formulae. $\endgroup$ – Tom Hale Jul 23 '17 at 10:35
5
$\begingroup$

Let's suppose you have 10 examples and you don't divide a L2 regularization cost by number of examples m. Then a "dominance" of the L2 regularization cost compared to a cross-entropy cost will be like 10:1, because each training example can contribute to the overall cost proportionally to 1/m = 1/10.

If you have more examples, let's say, 100, then the "dominance" of the L2 regularization cost will be something like 100:1, so you need to decrease a λ accordingly, which is inconvenient. It's better to have λ constant regardless of a batch size.

Update: To make this argument more strong I created a jupyter notebook.

$\endgroup$
  • 1
    $\begingroup$ Hmm, but isn't the purpose of the 1/m factor before the cost function that each training example contributes equally to the cost? So, since we are already averaging the individual costs, this should not be the cause of dominance of the L2 term. However, I do see from your great simulation that the 1/m factor also before the L2 term does help. I just don't get the intuition behind it (yet). $\endgroup$ – Milania Jun 4 '18 at 18:57
  • $\begingroup$ Why is it inconvenient?? it is simple to divide L2 cost by the number of samples. I think maybe you phrased it the wrong way. I think you meant to say it's inconvenient to manually scale L2 cost every time, it's better to divide by the number of samples as part of the formula to scale it automatically. $\endgroup$ – SpaceMonkey Jun 6 '18 at 0:23
6
$\begingroup$

The loss function on the training set $J(\theta)$ is generally a sum over the patterns comprising the training set, so as the training set gets larger, the first term scales essentially linearly with $m$. We can narrow the range for seraching for a good value of $\lambda$ a fair bit if we first divide the regularisation term by $m$ to offset the dependence of $J(\theta)$ on $m$. The 2 of course is indeed in the denominator to simplify the derivatives needed for the opimisation algorithm used to determine the optimal $\theta$.

$\endgroup$
  • $\begingroup$ Thanks for explaining the non-regularised cost scaling with $m$. I still don't get how dividing by $m$ will help a single $\lambda$ value work better with widely differing values of $m$. The non-regularised cost is already heavily dependent on $m$, so why care about the regularisation term which is dependent on $n$ parameters, not $m$ examples? Is it because with more training examples, the variance will decrease given the same number of parameters? $\endgroup$ – Tom Hale Jun 29 '17 at 8:26
  • $\begingroup$ The loss function in the question is an average over all examples (i.e. it is divided by m), not a sum, so I don't really see how this answer works. $\endgroup$ – Denziloe Jul 12 '19 at 22:28
  • $\begingroup$ @Denziloe it is applied to the regularisation term as well. $\endgroup$ – Dikran Marsupial Jul 13 '19 at 17:35
2
$\begingroup$

I wondered about the exact same thing when taking this course, and ended up researching this a bit. I'll give a short answer here, but you can read a more detailed overview in a blog post I wrote about it.

I believe that at least part of the reason for those scaling coefficients is that L² regularization probably entered the field of deep learning through the introduction of the related, but not identical, concept of weight decay.

The 0.5 factor is then there to get a nice λ-only coefficient for the weight decay in the gradient, and the scaling by m... well, there are at least 5 different motivations that I have found or came up with:

  1. A side-effect of batch gradient descent: When a single iteration of gradient descent is instead formalized over the entire training set, resulting in the algorithm sometimes called batch gradient descent, the scaling factor of 1/m, introduced to make the cost function comparable across different size datasets, gets automatically applied to the weight decay term.
  2. Rescale to the weight of a single example: See grez's interesting intuition.
  3. Training set representativeness: It makes sense to scale down regularization as the size of the training set grows, as statistically, its representativeness of the overall distribution also grows. Basically, the more data we have, the less regularization is needed.
  4. Making λ comparable: By hopefully mitigating the need to change λ when m changes, this scaling makes λ itself comparable across different size datasets. This make λ a more representative estimator of the actual degree of regularization required by a specific model on a specific learning problem.
  5. Empirical value: The great notebook by grez demonstrates that this improves performance in practice.
$\endgroup$
0
$\begingroup$

I was also confused about this, but then in a lecture for deeplearning.ai Andrew suggests this is just a scaling constant:

http://www.youtube.com/watch?v=6g0t3Phly2M&t=2m50s

Perhaps there is a deeper reason for using 1/2m but I suspect it is simply a hyperparameter.

$\endgroup$
  • $\begingroup$ This does not answer the question. $\endgroup$ – Michael R. Chernick Nov 25 '19 at 1:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.