0
$\begingroup$

When performing stochastic gradient descent, it is necessary for the training loss to decrease

a) between iterations in an epoch? (I think the answer is no)

b) between epochs? (I think the answer is yes)

The training loss is always defined using the entire dataset.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ no to both q's. $\endgroup$
    – shimao
    Commented Feb 20, 2019 at 19:06

1 Answer 1

Reset to default
2
$\begingroup$

In both cases, it is not guaranteed because you're not using the full batch, i.e. stochastic gradient descent only approximates the gradient to be found by the full batch. However, between epochs (b), it's much more likely to decrease compared to between iterations (a), because the entire training dataset have been inputted to the network and weights are updated several times towards a local minimum.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.