Should parsimony really still be the gold standard?

Question

Just a thought:

Parsimonious models have always been the default go-to in model selection, but to what degree is this approach outdated? I'm curious about how much our tendency toward parsimony is a relic of a time of abaci and slide rules (or, more seriously, non-modern computers). Today's computing power enables us to build increasingly complex models with ever-greater ability for prediction. As a result of this increasing ceiling in computing power, do we really still need to gravitate toward simplicity?

Sure, simpler models are easier to understand and interpret, but in the age of ever-growing data sets with greater numbers of variables and a shift towards a greater focus on prediction capability, this might no longer even be achievable or necessary.

Thoughts?

Answer 1

@Matt's original answer does a great job of describing one of the benefits of parsimony but I don't think it actually answers your question. In reality, parsimony isn't the gold standard. Not now nor has it ever been. A "gold standard" related to parsimony is generalization error. We would like to develop models that don't overfit. That are as useful for prediction (or as interpretable or with minimum error) out of sample as they are in sample. It turns out (because of things laid out above) that parsimony is actually quite a good proxy for generalization error but it's by no means the only one.

Really, think about why we use cross validation or bootstrapping or train/test sets. The goal is to create models with good generalization accuracy. A lot of the time, these ways of estimating out of sample performance do end up choosing models with lower complexity but not always. As an extreme example imagine the oracle hands us the true but extremely complex model and a poor but parsimonious model. If parsimony was really our goal then we would choose the second but in reality, the first is what we would like to learn if we could. Unfortunately a lot of the time that last sentence is the kicker, "if we could".

Answer 2

Parsimonious models are desirable not just due to computing requirements, but also for generalization performance. It's impossible to achieve the ideal of infinite data that completely and accurately covers the sample space, meaning that non-parsimonious models have the potential to overfit and model noise or idiosyncrasies in the sample population.

It's certainly possible to build a model with millions of variables, but you'd be using variables that have no impact on the output to model the system. You could achieve great predictive performance on your training dataset, but those irrelevant variables will more than likely decrease your performance on an unseen test set.

If an output variable truly is the result of a million input variables, then you would do well to put them all in your predictive model, but only if you have enough data. To accurately build a model of this size, you'd need several million data points, at minimum. Parsimonious models are nice because in many real-world systems, a dataset of this size simply isn't available, and furthermore, the output is largely determined by a relatively small number of variables.

Answer 3

I think the previous answers do a good job of making important points:

• Parsimonious models tend to have better generalization characteristics.
• Parsimony is not truly a gold standard, but just a consideration.

I want to add a few comments that come out of my day to day job experience.

The generalization of predictive accuracy argument is, of course, strong, but is academically bias in its focus. In general, when producing a statistical model, the economies are not such that predictive performance is a completely dominant consideration. Very often there are large outside constraints on what a useful model looks like for a given application:

• The model must be implementable within an existing framework or system.
• The model must be understandable by a non-technical entity.
• The model must be efficient computationally.
• The model must be documentable.
• The model must pass regulatory constraints.

In real application domains, many if not all of these considerations come before, not after, predictive performance - and the optimization of model form and parameters is constrained by these desires. Each of these constraints biases the scientist towards parsimony.

It may be true that in many domains these constraints are being gradually lifted. But it is the lucky scientist indeed that gets to ignore them are focus purely on minimizing generalization error.

This can be very frustrating for the first time scientist, fresh out of school (it definitely was for me, and continues to be when I feel that the constraints placed on my work are not justified). But in the end, working hard to produce an unacceptable product is a waste, and that feels worse than the sting to your scientific pride.

Answer 4

I think this is a very good question. In my opinion parsimony is overrated. Nature is rarely parsimonious, and so we shouldn't necessarily expect accurate predictive or descriptive models to be so either. Regarding the question of interpretability, if you choose a simpler model that only modestly conforms to reality merely because you can understand it, what exactly are you understanding? Assuming a more complex model had better predictive power, it would appear to be closer to the actual facts anyways.

Answer 5

Perhaps have a review of the Akaike Information Criterion, a concept that I only discovered by serendipity yesterday. The AIC seeks to identify which model and how many parameters are the best explanation for the observations at hand, rather than any basic Occam's Razor, or parsimony approach.

Answer 6

Parsimony is not a golden start. It's an aspect in modeling. Modeling and especially forecasting can not be scripted, i.e. you can't just hand a script to a modeler to follow. You rather define principles upon which the modeling process must be based. So, the parsimony is one of these principles, application of which can not be scripted (again!). A modeler will consider the complexity when a selecting model.

Computational power has little to do with this. If you're in the industry your models will be consumed by business folks, product people, whoever you call them. You have to explain your model to them, it should make a sense to them. Having parsimonious models helps in this regard.

For instance, you're forecasting product sales. You should be able to describe what are the drivers of sales, and how they work. These must be related to concepts with which business operates, and the correlations must be understood and accepted by business. With complex models it could be very difficult to interpret the results of the model or attribute the differences with actuals. If you can't explain your models to business, you will not be valued by it.

One more thing that is particularly important for forecasting. Let's say your model is dependent on N exogenous variables. This means that you have to first obtain the forecasts of these variables in order to forecast your dependent variable. Having smaller N makes your life easier, so a simpler model is easier to use.

Answer 7

Regarding Neural Networks, this topic has been very nicely covered in a recent NeurIPS 2020 paper entitled

The Pitfalls of Simplicity Bias in Neural Networks (Shah et al.).

I totally recommend reading the paper, I think it is very nicely structured and rigorous. Here is an attempt to summarize its main ideas:

---

• The core of the problem is that the field seems to associate simplicity bias to "justify why NNs work well", but not to their "lack of robustness".

• The authors tackle this problem via the design of datasets that comprise multiple predictive features with varying levels of simplicity. These will serve as control variables.

• Following a series of theoretical analyses and targeted experiments, they found that the simplicity bias given by Stochastic Gradient Descent can be extreme, that common approaches to overcome this can fail, and in general, that there is still quite a way to go in order to better understand SB in NNs.

The proposed datasets and methods can serve as a basis for future work in that direction.

---

This principle of SB for NNs is very much embedded in the Deep Learning culture: See e.g. this video from the Udacity "Deep Learning" course (Vanhoucke et al.), where they use the "stretch pants" analogy. The idea is that instead of trying to guess the right pant size, you start with several extra sizes and "stretch them down" until they fit perfectly. This way, training is seen as reducing the capacity of the model wherever there is room to do so.

The paper above questions the general validity of this principle, showing that when trained with SGD, the pants tend to fit the simpler legs, no matter how many dimensions they have.

So whereas the original question is from 2015, this topic is still extremely current. Particularly, the following OP sentence is right on point:

Today's computing power enables us to build increasingly complex models with ever-greater ability for prediction. As a result of this increasing ceiling in computing power, do we really still need to gravitate toward simplicity?

Since the paper above shows that the ever-growing NN architectures still invariably gravitate towards simpler features when trained with SGD.

---

I would also like to finish commenting about another bias related to SB, that is Inductive Bias, "the set of assumptions that the learner uses to predict outputs of given inputs that it has not encountered".

Instead of assuming simplicity, inductive bias assumes any other arbitrary property. For example, when training a NN for transcription of piano music, it is safe to assume that all keyboard strokes generate mainly harmonic waveforms that have an onset and decay exponentially. In terms of "no free lunch", correctly applying an inductive bias would translate to getting lunch for a good prize. Physical models are known for their extremely precise and rigorous application of inductive biases.

Another amazing ICLR 2020 paper that treats the systematic introduction of Inductive Bias into NN design and training is

DDSP: DIFFERENTIABLE DIGITAL SIGNAL PROCESSING (Engel et al.)

With application of DDSP techniques it can be observed that introducing inductive biases helps the models satisfactorily solve complex tasks with less capacity, also the SGD convergence is faster and requires less training data.

So again coming back to the OP: While the No Free Lunch theorem states that no algorithm will work better than other for all kinds of problems, inductive biases are a good mechanism to gain performance without sacrificing much interpretability, if you are willing to reduce the set of targeted problems.