I think the previous answers do a good job of making important points:
- Parsimonious models tend to have better generalization characteristics.
- Parsimony is not truly a gold standard, but just a consideration.
I want to add a few comments that come out of my day to day job experience.
The generalization of predictive accuracy argument is, of course, strong, but is academically bias in its focus. In general, when producing a statistical model, the economies are not such that predictive performance is a completely dominant consideration. Very often there are large outside constraints on what a useful model looks like for a given application:
- The model must be implementable within an existing framework or system.
- The model must be understandable by a non-technical entity.
- The model must be efficient computationally.
- The model must be documentable.
- The model must pass regulatory constraints.
In real application domains, many if not all of these considerations come before, not after, predictive performance - and the optimization of model form and parameters is constrained by these desires. Each of these constraints biases the scientist towards parsimony.
It may be true that in many domains these constraints are being gradually lifted. But it is the lucky scientist indeed that gets to ignore them are focus purely on minimizing generalization error.
This can be very frustrating for the first time scientist, fresh out of school (it definitely was for me, and continues to be when I feel that the constraints placed on my work are not justified). But in the end, working hard to produce an unacceptable product is a waste, and that feels worse than the sting to your scientific pride.