Andrew Ng in his deep learning course on Coursera.org states that there is a boundary on sample size where machine learning algorithms stop improving and such boundary is nonexistent for the deep learning algorithms, as they always improve when feeded with more data. Could you point any reference that goes into more details and describes actual research on the phenomenon?
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era Shows performance of RNNs has roughly linear relationship with the log amount of data. See also Deep Speech 2: End-to-End Speech Recognition in English and Mandarin, table 10. Regularization techniques for fine-tuning in neural machine translation and Scaling Recurrent Neural Network Language Models seem to say the same thing. I couldn't find any sources, but simpler models with limited number of parameters, at least in my experience, have a much lower "hard-limit" of performance.