This is a very deep question, because neural networks are very mysterious in this regard compared to classic learning algorithms. Modern applications of deep learning tend to use an enormous number of parameters, often much higher than the number of observations. As such, they will typically learn the training data exactly, and will achieve 0 error on the training set.
Intuitively these models should overfit, but in practice they dont and the generalisation error tends to be very low. As far as I know, noone has really managed to explain this yet, and its a very active area of research. What makes things is extremely strange is that the likelihood surface being minimised is usually multimodal and there will often be several modes (i.e. multiple different sets of parameters) that all give 0 training set error, however some of these modes will generalise well to the test set, while others dont. Stochastic Gradient Descent usually manages to find a mode that generalises well, even if there is no obvious reason why this should be the case. This means that the performance of deep neural networks perhaps cant be analysed separately from the optimisation algorithm used to fit them, which is highly counterintuitive.
That said, neural networks do often get fit with various regularisation techniques. A fairly standard one is drop-out, where some nodes randomly get their weights set to 0: https://en.wikipedia.org/wiki/Dilution_(neural_networks)
Some references, to look through:
https://arxiv.org/abs/1906.11300
https://arxiv.org/abs/1812.11118
https://arxiv.org/abs/1703.11008
There is also an upcoming Annals of Statistics paper on this (https://imstat.org/wp-content/uploads/2019/12/AOS1875.pdf) with a really excellent reply/criticism by Omad Shamir which is quite readable and maybe a good starting point: https://www.e-publications.org/ims/submission/AOS/user/submissionFile/41983?confirm=b1fc57b2
For other machine learning settings than deep learning, the trade-off between overfitting and flexible models is more understood and agreed on. Many machine learning algorithms use regulisastion (eg lasso in logistic regression, or SVMs doing automatic implicit regularisation) which essentially reduces the number of parameters. Also the way machine learning algorithms are fitted to data will often implicity do regularisation - a very standard approach is to use an overparameterised model but to monitor the performance on a separate validation set while the training set error is being minimised (rather than monitoring the training set error), and to stop the training early as soon as the validation error starts increasing. This is again an implicit type of regularisation since the training set error will always be decreasing during the estimating/optimisation phase, whereas the validation set error will not. While this approach is very intuitive and works well in practice, its difficult to put it inside a formal mathematical/statistical framework since (unlike standard regulisation) its happening inside the optimisation algorithm, rather than being a feature of the model itself.
I think there is a big philosophical/cultural difference between statistics and machine learning here. Statisticians tend to start with simple models with few parameters which they know wont be flexible enough to capture real world behavior (the famous Box quote "all models are wrong") and then cautiously expand them to give more complex models only when this is justified. On the other hand, machine learners typically start by fitting complex models which are flexible enough to capture the 'true' model whatever it may be, and then aggressively regularise to prevent overfitting, even if this means hacking around inside the optimisation algorithms themselves.
