If I have virtually endless training data (it's synthesized) is there still purpose in having epochs? I.e. training on the same samples multiple times?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
If you have access to unlimited data, unless the order in which that data is generated has an influence on the data, it is better not to have epochs and just train for as long as desired while synthetising examples on the fly. (You want each of your examples to be iid).
However, if your data generating process is biased, for example if a variable X taking values in $\{0,1\}$ has its $0$ values generated first, it will quite probably be better to generate a finite sample and shuffle it, and then train with several epochs (possibly even shuffling between epochs. Otherwise, you will always "show" your $0$ examples before the $1$ examples, and after a long period of learning, the network will only have seen $1$'s in its recent past and thus never fitted to the $0$'s.
$\begingroup$
$\endgroup$
For this issue you can use Online learning. Defenition you can find at Wikipedia. More information about online learning in NN you can check at this article.
