0
$\begingroup$

Equation 93 of Chapter 3 of Michael Nielsen's neural networks book describes the stochastic gradient descent update rule as the following:

$w \leftarrow (1-\frac{\eta\lambda}{n})w - \frac{\eta}{m}\sum_x \frac{\partial C_x}{\partial w}$

for a mini-batch of size $m$ and individual example cost $C_x$. His cost function is of the form

$C = C_0 + \frac{\lambda}{2n}\sum_w w^2$ where $C_0$ is the original unregularized cost function.

My question is, why is there a $n$ in the update rule for stochastic gradient descent? Why does the update rule not have $(1-\frac{\eta\lambda}{m})$ instead since we only have $m$ examples? Or is this not really a big deal because that's just the way we defined $C$ and the $\lambda$ term can take care of the difference?

$\endgroup$
0
$\begingroup$

We seek to minimize the cost on the whole training dataset (for example, consider the regression loss, classification loss is the same, as well as many other losses)

$$ C = \frac{1}{2n} \sum_{i=1}^n \|y_i - a^L(x_i)\|^2 + \frac{\lambda}{2n} \sum_{w} w^2 $$

Note that this is equivalent to

$$ C = \frac{1}{n} \sum_{i=1}^n \left[ \frac{1}{2} \|y_i - a^L(x_i)\|^2 + \frac{\lambda}{2n} \sum_{w} w^2 \right] = \mathbb{E}_{i \sim U[1, \dots, n]} \left[ \frac{1}{2} \|y_i - a^L(x_i)\|^2 + \frac{\lambda}{2n} \sum_{w} w^2 \right] $$

That is, $C$ is an expectation over random index of a training sample, $i$, uniformly distributed from 1 to n.

Stochastic Gradient Descent is used to optimize objectives of this kind when we only have the gradient of the expression under the expectation evaluated at and averaged over for some batch (however small) of indices $i$. That is, we "approximate" the true gradient with

$$ \tilde g = \frac{1}{m} \sum_{i=1}^m \frac{\partial}{\partial w} \left[ \frac{1}{2} \|y_i - a^L(x_i)\|^2 + \frac{\lambda}{2n} \sum_{w} w^2 \right] = \frac{1}{m} \sum_{i=1}^m \frac{\partial}{\partial w} \left[ \frac{1}{2} \|y_i - a^L(x_i)\|^2 \right] + \frac{\lambda}{n} w $$

Now, to your questions: No, batch size $m$ does not affect the strength of regularization, as it does not depend of your training data. You can think of it as of an average of $m$ identical samples with a weight of $\lambda/2n$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.