Large neural networks have low bias and high variance. Training on large datasets greatly reduces the variance allowing them to fit complicated functions. My question is why they seem to have much lower bias than other machine learning models, regardless of how flexible you make the other models. *
It's purported that if you plot amount of data vs performance, neural networks tend to not to plateau as quickly. One could argue that this has something to do with being a universal function approximator, but this apparently applies to other models like kernel methods as well.
What about neural networks makes them especially good at working in regimes with large amounts of data compared to more traditional models which can also be made to be arbitrarily flexible?
I can only speculate it might have something to do with the following (but I can't find definitive sources):
- weaker inductive bias
- real-world data (distribution of images, language, etc.) being particularly amenable to NN operations
- having intermediate/hierarchical representations representations is somehow important (Yann LeCun talks about a Kuhnian [or Le-Kuhnian? 😆] paradigm shift on this point among researchers).
*[EDIT thanks for comments - framing in terms of bias variance is murky because it's not clear how much of the performance improvement could be attributed to lower variance due to the double descent phenomenon].