In artificial neural networks, is weight decay regularization the same sort of regularization that would reducing the capacity of the model be?
I've learned that applying weight decay shrinks the weights towards smaller values, and this tends to avoid overfitting. Okey, I see that. In the figure below it is clear that the overfitted model has regions of large curvature, and this could be a consequence of having high values of the weights.
I've learned that reducing the model's capacity, i.e. reducing the number of layers or the number of units per layer, also avoids overfitting (depends also on the complexity of the task), because the more flexible the model the more it will tend to overfit. This can also be seen in the figure below, where the model that overfits chose that particular solution but could have chosen many others that would still fit the data, but not the underlying "data generation process" (expression that I don't get to fully grasp).
So again, my question is, are weight decay and model-capacity-reduction achieving the same sort of regularization? Or they achieve something different? And if so, how?
(Figure taken from www.deeplearningbook.org by Ian Goodfellow et al. )