Andrew Ng in his deep learning course on Coursera.org states that there is a boundary on sample size where machine learning algorithms stop improving and such boundary is nonexistent for the deep learning algorithms, as they always improve when feeded with more data. Could you point any reference that goes into more details and describes actual research on the phenomenon?
$\begingroup$
$\endgroup$
4
-
1$\begingroup$ I don't believe what he says is true if he is talking about a predetermined neuron count. If you build a very big network, obviously it will take a lot of data points to saturate it, but it will still saturate eventually. If we are talking about throwing in more neurons as we get more data, then it is more likely to be true. $\endgroup$– Cagdas OzgencCommented Oct 31, 2017 at 12:14
-
1$\begingroup$ @CagdasOzgenc: and of course, if we allow the model to increase in complexity as we feed in more data, then non-DL models may also hit no hard improvement limits. $\endgroup$– Stephan KolassaCommented Oct 31, 2017 at 12:39
-
$\begingroup$ @Tim I think this reference can be useful. pdfs.semanticscholar.org/1759/…. Rational weighted recurrent neural network with sigmoid activation and Turing Machine equivalency. I would post it to Computer Theory group to get more information as it is related to limits of computation rather than statistics. $\endgroup$– Cagdas OzgencCommented Oct 31, 2017 at 14:03
-
$\begingroup$ @CagdasOzgenc thanks, but Andrew Ng talked only about model performance, not computational poewer and this is the only aspect of interest for me. $\endgroup$– TimCommented Oct 31, 2017 at 14:23
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era Shows performance of RNNs has roughly linear relationship with the log amount of data. See also Deep Speech 2: End-to-End Speech Recognition in English and Mandarin, table 10. Regularization techniques for fine-tuning in neural machine translation and Scaling Recurrent Neural Network Language Models seem to say the same thing. I couldn't find any sources, but simpler models with limited number of parameters, at least in my experience, have a much lower "hard-limit" of performance.
-
$\begingroup$ The "Unreasonable Effectiveness of Data" by Pereira et al. was such an opinionated and enjoyable paper when it came out, thank you for bringing attention to one of its intellectual descendants. (The new paper is a bit too lax in my opinion; all the examples are based on Comp. Vision which clearly is not everything big data are used for but OK... Thanks again! +1) $\endgroup$ Commented Nov 12, 2017 at 1:42