I don't think there is a definite answer to your questions. But I think the conventional wisdom goes as following:
Basically, as the hypothesis space of a learning algorithm grows, the algorithm can learn richer and richer structures. But at the same time, the algorithm becomes more prone to overfitting and its generalization error is likely to increase.
So ultimately, for any given dataset, it's advisable to work with the minimal model that has enough capacity to learn the real structure of the data. But this is a very hand-wavy advice, since usually the "real structure of the data" is unknown, and often even the capacities of the candidate models are only vaguely understood.
When it comes to neural networks, the size of the hypothesis space is controlled by the number of parameters. And it seems that for a fixed number of parameters (or a fixed order of magnitude), going deeper allows the models to capture richer structures (e.g. this paper).
This may partially explain the success of deeper models with fewer parameters: VGGNet (from 2014) has 16 layers with ~140M parameters, while ResNet (from 2015) beat it with 152 layers but only ~2M parameters
(as a side, smaller models may be computationally easier to train - but I don't think it's a major factor by itself - since depth actually complicates the training)
Note that this trend (more depth, less parameters) is mostly present in vision-related tasks and convolutional networks, and this calls for a domain-specific explanation. So here's another perspective:
Each "neuron" in a convolutional layer has a "receptive field", which is the size and shape of the inputs that effects each output. Intuitively, each kernel captures some kind of a relation between nearby inputs. And small kernels (which are common and preferable) have a small receptive field, so they can provide information only regarding local relations.
But as you go deeper, the receptive field of each neuron with respect to a some earlier layer becomes larger. So deep layers can provide features with global semantic meaning and abstract details (relations of relations ... of relations of objects), while using only small kernels (which regularize the relations the network learns, and helps it converge and generalize).
So the usefulness of deep convolutional networks in computer vision may be partially explained by the spatial structure of images and videos. It's possible that time will tell that for different types of problems, or for non-convolutional architectures, depth actually doesn't work well.