3
$\begingroup$

Assume we have a neural network with stochastic gradient descent used for backpropagation, and therefore each element in the training set is used once to calculate the error, and then to adjust the weights (assume each element in the training set is used only once).

Assume we're doing a simple regression without regularization, and with 1 or 2 hidden layers.

I can't quite understand why such an algorithm doesn't, through learning at training point N, "undo" learning that it did with training point N-1, N-2, etc.

Is such an algorithm even supposed to converge to the optimal parameters? My intuition would say that it would "hops back and forth", because it doesn't take into account all the data all at once.

Improve this question
$\endgroup$
1
  • $\begingroup$ whether it hops back and forth depends on the learning rate - by making it small enough that doesn't happen (and it is almost like using all the data at once) $\endgroup$
    – seanv507
    Commented Aug 11, 2016 at 16:20

1 Answer 1

Reset to default
3
$\begingroup$

Model collapse and catastrophic forgetting can both happen to some neural networks. Setting aside these somewhat niche scenarios, the trajectory of SGD can "make hops back and forth" while making excruciatingly slow progress towards the minimum. See: https://stats.stackexchange.com/a/367459/22311 Tuning the learning rate and using momentum and preconditioning can ameliorate this somewhat.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.