Regularisation: why multiply by 1/2m?

Question

In the week 3 lecture notes of Andrew Ng's Coursera Machine Learning class, a term is added to the cost function to implement regularisation:

$$J^+(\theta) = J(\theta) + \frac{\lambda}{2m} \sum_{j=1}^n \theta_j^2$$

The lecture notes say:

We could also regularize all of our theta parameters in a single summation:

$$min_\theta\ \dfrac{1}{2m}\ \left[ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})^2 + \lambda\ \sum_{j=1}^n \theta_j^2 \right] $$

$\frac 1 {2m}$ is later applied to the regularisation term of neural networks:

Recall that the cost function for regularized logistic regression was:

$$J(\theta) = - \frac{1}{m} \sum_{i=1}^m [ y^{(i)}\ \log (h_\theta (x^{(i)})) + (1 - y^{(i)})\ \log (1 - h_\theta(x^{(i)}))] + \frac{\lambda}{2m}\sum_{j=1}^n \theta_j^2$$

For neural networks, it is going to be slightly more complicated: $$\begin{gather*} J(\Theta) = - \frac{1}{m} \sum_{i=1}^m \sum_{k=1}^K \left[y^{(i)}_k \log ((h_\Theta (x^{(i)}))_k) + (1 - y^{(i)}_k)\log (1 - (h_\Theta(x^{(i)}))_k)\right] + \frac{\lambda}{2m}\sum_{l=1}^{L-1} \sum_{i=1}^{s_l} \sum_{j=1}^{s_{l+1}} ( \Theta_{j,i}^{(l)})^2\end{gather*} $$

• Why is the constant one-half used here? So that it is cancelled in the derivative $J'$?
• Why the division by $m$ training examples? How does the amount of training examples affect things?

Answer 1

Let's suppose you have 10 examples and you don't divide a L2 regularization cost by number of examples m. Then a "dominance" of the L2 regularization cost compared to a cross-entropy cost will be like 10:1, because each training example can contribute to the overall cost proportionally to 1/m = 1/10.

If you have more examples, let's say, 100, then the "dominance" of the L2 regularization cost will be something like 100:1, so you need to decrease a λ accordingly, which is inconvenient. It's better to have λ constant regardless of a batch size.

Update: To make this argument more strong I created a jupyter notebook.

Answer 2

The loss function on the training set $J(\theta)$ is generally a sum over the patterns comprising the training set, so as the training set gets larger, the first term scales essentially linearly with $m$. We can narrow the range for seraching for a good value of $\lambda$ a fair bit if we first divide the regularisation term by $m$ to offset the dependence of $J(\theta)$ on $m$. The 2 of course is indeed in the denominator to simplify the derivatives needed for the opimisation algorithm used to determine the optimal $\theta$.

Answer 3

I wondered about the exact same thing when taking this course, and ended up researching this a bit. I'll give a short answer here, but you can read a more detailed overview in a blog post I wrote about it.

I believe that at least part of the reason for those scaling coefficients is that L² regularization probably entered the field of deep learning through the introduction of the related, but not identical, concept of weight decay.

The 0.5 factor is then there to get a nice λ-only coefficient for the weight decay in the gradient, and the scaling by m... well, there are at least 5 different motivations that I have found or came up with:

1. A side-effect of batch gradient descent: When a single iteration of gradient descent is instead formalized over the entire training set, resulting in the algorithm sometimes called batch gradient descent, the scaling factor of 1/m, introduced to make the cost function comparable across different size datasets, gets automatically applied to the weight decay term.
2. Rescale to the weight of a single example: See grez's interesting intuition.
3. Training set representativeness: It makes sense to scale down regularization as the size of the training set grows, as statistically, its representativeness of the overall distribution also grows. Basically, the more data we have, the less regularization is needed.
4. Making λ comparable: By hopefully mitigating the need to change λ when m changes, this scaling makes λ itself comparable across different size datasets. This make λ a more representative estimator of the actual degree of regularization required by a specific model on a specific learning problem.
5. Empirical value: The great notebook by grez demonstrates that this improves performance in practice.

Answer 4

I was also confused about this, but then in a lecture for deeplearning.ai Andrew suggests this is just a scaling constant:

http://www.youtube.com/watch?v=6g0t3Phly2M&t=2m50s

Perhaps there is a deeper reason for using 1/2m but I suspect it is simply a hyperparameter.