Can we solve overfitting by adding more parameters? What is the state of the art knowledge on how generalization in interpolating models looks with respect to the number of parameters?
Does it look like this: 
(Picture from Mikhail Belkin's talk on https://www.youtube.com/watch?v=OBCciGnOJVs&t=1185s)
In other words, can overfitting always be overcome with adding more parameters?
Let's say we don't use regularization, but train only for some natural-looking interpolation loss.
I'm mainly interested in what is true for neural networks with several layers, but anything goes.
 A: According to recent works on the Double Descent phenomena, specially Belkin's, yes, you may be able to fix overfitting with more parameters.
That happens because, according to their hypothesis, if you have just enough parameters to interpolate training data, the solution space becomes constrained, precluding you from achieving a lower norm solution.
Adding more parameters (to the limit at infinity) "opens up" solution space again, allowing for, still interpolating, smaller norm solutions.
The interesting part is that in the interpolating regime smaller loss is often achieved than in the non-interpolating regime.
That helps to explain how absurdly over-parametrized deep networks work in practice, as stochastic optimization is inherently regularized and SGD will converge to minimum norm solutions in the over-parametrized regime.
A: Adding parameters will lead to more overfitting. The more parameters, the more models you can represent. The more models, the more likely you'll find one that fits your training data exactly.
To avoid overfitting, choose the simplest model that does not underfit, and use cross-validation to make sure.
