10
$\begingroup$

In the week 3 lecture notes of Andrew Ng's Coursera Machine Learning class, a term is added to the cost function to implement regularisation:

$$J^+(\theta) = J(\theta) + \frac{\lambda}{2m} \sum_{j=1}^n \theta_j^2$$

The lecture notes say:

We could also regularize all of our theta parameters in a single summation:

$$min_\theta\ \dfrac{1}{2m}\ \left[ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})^2 + \lambda\ \sum_{j=1}^n \theta_j^2 \right] $$

$\frac 1 {2m}$ is later applied to the regularisation term of neural networks:

Recall that the cost function for regularized logistic regression was:

$$J(\theta) = - \frac{1}{m} \sum_{i=1}^m [ y^{(i)}\ \log (h_\theta (x^{(i)})) + (1 - y^{(i)})\ \log (1 - h_\theta(x^{(i)}))] + \frac{\lambda}{2m}\sum_{j=1}^n \theta_j^2$$

For neural networks, it is going to be slightly more complicated: $$\begin{gather*} J(\Theta) = - \frac{1}{m} \sum_{i=1}^m \sum_{k=1}^K \left[y^{(i)}_k \log ((h_\Theta (x^{(i)}))_k) + (1 - y^{(i)}_k)\log (1 - (h_\Theta(x^{(i)}))_k)\right] + \frac{\lambda}{2m}\sum_{l=1}^{L-1} \sum_{i=1}^{s_l} \sum_{j=1}^{s_{l+1}} ( \Theta_{j,i}^{(l)})^2\end{gather*} $$

  • Why is the constant one-half used here? So that it is cancelled in the derivative $J'$?
  • Why the division by $m$ training examples? How does the amount of training examples affect things?
$\endgroup$
2
  • $\begingroup$ are you sure the 1/m is on the regularisation and not on the J(theta) AFAIK @DikranMarsupial 's answer is making that assumption...... or does J(theta) itself have a 1/m term? $\endgroup$
    – seanv507
    Jul 16, 2017 at 13:29
  • $\begingroup$ That assumption is incorrect - $ 1 \over 2m$ is applied to both the un-regularised cost function and the regularisation term. I've updated the question to give the full formulae. $\endgroup$
    – Tom Hale
    Jul 23, 2017 at 10:35

4 Answers 4

5
$\begingroup$

Let's suppose you have 10 examples and you don't divide a L2 regularization cost by number of examples m. Then a "dominance" of the L2 regularization cost compared to a cross-entropy cost will be like 10:1, because each training example can contribute to the overall cost proportionally to 1/m = 1/10.

If you have more examples, let's say, 100, then the "dominance" of the L2 regularization cost will be something like 100:1, so you need to decrease a λ accordingly, which is inconvenient. It's better to have λ constant regardless of a batch size.

Update: To make this argument more strong I created a jupyter notebook.

$\endgroup$
3
  • 2
    $\begingroup$ Hmm, but isn't the purpose of the 1/m factor before the cost function that each training example contributes equally to the cost? So, since we are already averaging the individual costs, this should not be the cause of dominance of the L2 term. However, I do see from your great simulation that the 1/m factor also before the L2 term does help. I just don't get the intuition behind it (yet). $\endgroup$
    – Milania
    Jun 4, 2018 at 18:57
  • $\begingroup$ Why is it inconvenient?? it is simple to divide L2 cost by the number of samples. I think maybe you phrased it the wrong way. I think you meant to say it's inconvenient to manually scale L2 cost every time, it's better to divide by the number of samples as part of the formula to scale it automatically. $\endgroup$ Jun 6, 2018 at 0:23
  • $\begingroup$ In your example you process all data in 1 batch. Let's say you wanted to split the data up in multiple batches, would you divide the Regularization Term by the amount of all data, or by the size of the current batch? $\endgroup$
    – csch
    Feb 11, 2021 at 9:30
6
$\begingroup$

The loss function on the training set $J(\theta)$ is generally a sum over the patterns comprising the training set, so as the training set gets larger, the first term scales essentially linearly with $m$. We can narrow the range for seraching for a good value of $\lambda$ a fair bit if we first divide the regularisation term by $m$ to offset the dependence of $J(\theta)$ on $m$. The 2 of course is indeed in the denominator to simplify the derivatives needed for the opimisation algorithm used to determine the optimal $\theta$.

$\endgroup$
3
  • $\begingroup$ Thanks for explaining the non-regularised cost scaling with $m$. I still don't get how dividing by $m$ will help a single $\lambda$ value work better with widely differing values of $m$. The non-regularised cost is already heavily dependent on $m$, so why care about the regularisation term which is dependent on $n$ parameters, not $m$ examples? Is it because with more training examples, the variance will decrease given the same number of parameters? $\endgroup$
    – Tom Hale
    Jun 29, 2017 at 8:26
  • $\begingroup$ The loss function in the question is an average over all examples (i.e. it is divided by m), not a sum, so I don't really see how this answer works. $\endgroup$
    – Denziloe
    Jul 12, 2019 at 22:28
  • $\begingroup$ @Denziloe it is applied to the regularisation term as well. $\endgroup$ Jul 13, 2019 at 17:35
2
$\begingroup$

I wondered about the exact same thing when taking this course, and ended up researching this a bit. I'll give a short answer here, but you can read a more detailed overview in a blog post I wrote about it.

I believe that at least part of the reason for those scaling coefficients is that L² regularization probably entered the field of deep learning through the introduction of the related, but not identical, concept of weight decay.

The 0.5 factor is then there to get a nice λ-only coefficient for the weight decay in the gradient, and the scaling by m... well, there are at least 5 different motivations that I have found or came up with:

  1. A side-effect of batch gradient descent: When a single iteration of gradient descent is instead formalized over the entire training set, resulting in the algorithm sometimes called batch gradient descent, the scaling factor of 1/m, introduced to make the cost function comparable across different size datasets, gets automatically applied to the weight decay term.
  2. Rescale to the weight of a single example: See grez's interesting intuition.
  3. Training set representativeness: It makes sense to scale down regularization as the size of the training set grows, as statistically, its representativeness of the overall distribution also grows. Basically, the more data we have, the less regularization is needed.
  4. Making λ comparable: By hopefully mitigating the need to change λ when m changes, this scaling makes λ itself comparable across different size datasets. This make λ a more representative estimator of the actual degree of regularization required by a specific model on a specific learning problem.
  5. Empirical value: The great notebook by grez demonstrates that this improves performance in practice.
$\endgroup$
0
$\begingroup$

I was also confused about this, but then in a lecture for deeplearning.ai Andrew suggests this is just a scaling constant:

http://www.youtube.com/watch?v=6g0t3Phly2M&t=2m50s

Perhaps there is a deeper reason for using 1/2m but I suspect it is simply a hyperparameter.

$\endgroup$
1
  • $\begingroup$ This does not answer the question. $\endgroup$ Nov 25, 2019 at 1:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.