38
$\begingroup$

Just a thought:

Parsimonious models have always been the default go-to in model selection, but to what degree is this approach outdated? I'm curious about how much our tendency toward parsimony is a relic of a time of abaci and slide rules (or, more seriously, non-modern computers). Today's computing power enables us to build increasingly complex models with ever-greater ability for prediction. As a result of this increasing ceiling in computing power, do we really still need to gravitate toward simplicity?

Sure, simpler models are easier to understand and interpret, but in the age of ever-growing data sets with greater numbers of variables and a shift towards a greater focus on prediction capability, this might no longer even be achievable or necessary.

Thoughts?

$\endgroup$
7
  • 5
    $\begingroup$ With apologies to Richard Hamming: The purpose of modeling is insight, not numbers. Complicated models impede insight. $\endgroup$ Jul 28, 2015 at 15:47
  • 15
    $\begingroup$ Models that are oversimplified impede insight even more. $\endgroup$ Jul 28, 2015 at 16:35
  • 6
    $\begingroup$ It may depend on the application; in physics, I think the argument for parsimony will have a strong basis. However, many applications will have a host of small effects that can't be eliminated (consider models for political preferences, for example) . A number of workers suggest that use of regularization (such as methods that lead to shrinkage or in many applications shrinkage of differences, or both) rather than elimination of variables makes more sense; others lean toward some selection and some shrinkage (LASSO for example, does both). $\endgroup$
    – Glen_b
    Jul 28, 2015 at 17:50
  • 4
    $\begingroup$ Parsimonious models are not the "go-to" in model selection. Otherwise we would always model everything with its sample mean and call it a day. $\endgroup$ Jul 29, 2015 at 16:29
  • 2
    $\begingroup$ Also, some food for thought: Mease and Wyner (2008) recommend richer learners in AdaBoost, which is a little unintuitive. An open question in that line of research seems to be whether parsimonious base learners actually lead to parsimonious ensembles. $\endgroup$ Jul 29, 2015 at 16:33

7 Answers 7

30
$\begingroup$

@Matt's original answer does a great job of describing one of the benefits of parsimony but I don't think it actually answers your question. In reality, parsimony isn't the gold standard. Not now nor has it ever been. A "gold standard" related to parsimony is generalization error. We would like to develop models that don't overfit. That are as useful for prediction (or as interpretable or with minimum error) out of sample as they are in sample. It turns out (because of things laid out above) that parsimony is actually quite a good proxy for generalization error but it's by no means the only one.

Really, think about why we use cross validation or bootstrapping or train/test sets. The goal is to create models with good generalization accuracy. A lot of the time, these ways of estimating out of sample performance do end up choosing models with lower complexity but not always. As an extreme example imagine the oracle hands us the true but extremely complex model and a poor but parsimonious model. If parsimony was really our goal then we would choose the second but in reality, the first is what we would like to learn if we could. Unfortunately a lot of the time that last sentence is the kicker, "if we could".

$\endgroup$
0
25
$\begingroup$

Parsimonious models are desirable not just due to computing requirements, but also for generalization performance. It's impossible to achieve the ideal of infinite data that completely and accurately covers the sample space, meaning that non-parsimonious models have the potential to overfit and model noise or idiosyncrasies in the sample population.

It's certainly possible to build a model with millions of variables, but you'd be using variables that have no impact on the output to model the system. You could achieve great predictive performance on your training dataset, but those irrelevant variables will more than likely decrease your performance on an unseen test set.

If an output variable truly is the result of a million input variables, then you would do well to put them all in your predictive model, but only if you have enough data. To accurately build a model of this size, you'd need several million data points, at minimum. Parsimonious models are nice because in many real-world systems, a dataset of this size simply isn't available, and furthermore, the output is largely determined by a relatively small number of variables.

$\endgroup$
4
  • 5
    $\begingroup$ +1. I suggest reading The Elements of Statistical Learning (freely available on the web), which discusses this problem in depth. $\endgroup$ Jul 28, 2015 at 14:42
  • 3
    $\begingroup$ On the other hand, when you have millions of variables and few objects, it is likely that purely by chance some variables are better at explaining outcome that the true interaction. In such case parsimony-based modelling will be more susceptible to overfitting than a brute-force approach. $\endgroup$
    – user88
    Jul 28, 2015 at 19:36
  • $\begingroup$ @CagdasOzgenc For instance a large random subspace ensemble. $\endgroup$
    – user88
    Jul 28, 2015 at 19:53
  • $\begingroup$ I feel like something like a Lasso approach could apply here. $\endgroup$ Jan 13, 2016 at 20:24
19
$\begingroup$

I think the previous answers do a good job of making important points:

  • Parsimonious models tend to have better generalization characteristics.
  • Parsimony is not truly a gold standard, but just a consideration.

I want to add a few comments that come out of my day to day job experience.

The generalization of predictive accuracy argument is, of course, strong, but is academically bias in its focus. In general, when producing a statistical model, the economies are not such that predictive performance is a completely dominant consideration. Very often there are large outside constraints on what a useful model looks like for a given application:

  • The model must be implementable within an existing framework or system.
  • The model must be understandable by a non-technical entity.
  • The model must be efficient computationally.
  • The model must be documentable.
  • The model must pass regulatory constraints.

In real application domains, many if not all of these considerations come before, not after, predictive performance - and the optimization of model form and parameters is constrained by these desires. Each of these constraints biases the scientist towards parsimony.

It may be true that in many domains these constraints are being gradually lifted. But it is the lucky scientist indeed that gets to ignore them are focus purely on minimizing generalization error.

This can be very frustrating for the first time scientist, fresh out of school (it definitely was for me, and continues to be when I feel that the constraints placed on my work are not justified). But in the end, working hard to produce an unacceptable product is a waste, and that feels worse than the sting to your scientific pride.

$\endgroup$
6
  • 2
    $\begingroup$ No parsimony is not a consideration. A sound inference procedure MUST rank a parsimonious model over a non-parsimonious model if they explain the data equally well. Otherwise total compressed codelength of the model and the data encoded by the model will not be the smallest. So yes it is a gold standard. $\endgroup$ Jul 28, 2015 at 16:47
  • 3
    $\begingroup$ Parsimony is NOT a "gold standard"! That statement is preposterous. If it were true, then why don't we always build models that fit nothing but the unconditional mean? We trade off bias and variance with reference to either a test set or, better still, completely new observations, and we do so within the constraints of our field, organization, and the law. Sometimes you only have enough information to make naive predictions. Sometimes you've got enough to add complexity. $\endgroup$ Jul 29, 2015 at 4:45
  • 1
    $\begingroup$ @BrashEquilibrium I think what Cagdas is saying is, given the choice between equally predictive models, one should choose the most parsimonious one. $\endgroup$ Jul 29, 2015 at 5:31
  • 2
    $\begingroup$ Ah. That's a different thing. Yes, in that case choose the most parsimonious model. I still don't think that amounts to parsimony being a "gold standard" though. $\endgroup$ Jul 29, 2015 at 13:30
  • 1
    $\begingroup$ @MatthewDrury Brash, Cagdas. Interesting. Perhaps, parsimony is just one component of the gold standard; which is probably (or ought to be) better based around the notion of encompassing. A good exposition of this idea is provided in the following astrophysics lecture from Yale: oyc.yale.edu/astronomy/astr-160/lecture-11. 7:04 onwards. The idea also features in the econometric/forecasting literature by David Hendry and Grayham Mizon. They argue that encompassing is part of a progressive research strategy, of which parsimony is a single aspect. $\endgroup$ Jan 14, 2016 at 1:03
14
$\begingroup$

I think this is a very good question. In my opinion parsimony is overrated. Nature is rarely parsimonious, and so we shouldn't necessarily expect accurate predictive or descriptive models to be so either. Regarding the question of interpretability, if you choose a simpler model that only modestly conforms to reality merely because you can understand it, what exactly are you understanding? Assuming a more complex model had better predictive power, it would appear to be closer to the actual facts anyways.

$\endgroup$
4
  • 8
    $\begingroup$ Well said @dsaxton. There is a great misunderstanding of parsimony and a great under-appreciation of how volatile feature selection is. Parsimony is nice when it results from pre-specification. Most parsimony that results from data dredging is misleading and is only understood because it's wrong. $\endgroup$ Jul 28, 2015 at 16:48
  • 2
    $\begingroup$ @FrankHarrell Would you elaborate on "only understood because it's wrong", or perhaps link to something you wrote previously about this? This is an interesting point that I would like to make sure I understand. $\endgroup$
    – gui11aume
    Jul 28, 2015 at 17:24
  • 9
    $\begingroup$ This is an extreme example but people who engage in racial profiling think they understand, with a single feature (e.g., skin color), what value someone has. To them the answer is simple. They only understand it because they are making a wrong judgment by oversimplifying. Parsimony is usually an illusion (except in Newtonian mechanics and a few other areas). $\endgroup$ Jul 28, 2015 at 18:05
  • 1
    $\begingroup$ "Nature is rarely parsimonious": and one point where nature is particularly non-parsimonious is individuals (as opposed to our typical sample sizes!). Evolution uses a whole new population of new individuals each generation... IMHO the parsimony (Frank Harrell's pre-specified type - allowing any n of m available features into the model is in fact a very complex model - even if n << m, this is a non-so-small fraction of the original search space) is how we try to get at least something out of our far-too-small data sets. $\endgroup$ Dec 23, 2017 at 13:54
3
$\begingroup$

Perhaps have a review of the Akaike Information Criterion, a concept that I only discovered by serendipity yesterday. The AIC seeks to identify which model and how many parameters are the best explanation for the observations at hand, rather than any basic Occam's Razor, or parsimony approach.

$\endgroup$
3
$\begingroup$

Parsimony is not a golden start. It's an aspect in modeling. Modeling and especially forecasting can not be scripted, i.e. you can't just hand a script to a modeler to follow. You rather define principles upon which the modeling process must be based. So, the parsimony is one of these principles, application of which can not be scripted (again!). A modeler will consider the complexity when a selecting model.

Computational power has little to do with this. If you're in the industry your models will be consumed by business folks, product people, whoever you call them. You have to explain your model to them, it should make a sense to them. Having parsimonious models helps in this regard.

For instance, you're forecasting product sales. You should be able to describe what are the drivers of sales, and how they work. These must be related to concepts with which business operates, and the correlations must be understood and accepted by business. With complex models it could be very difficult to interpret the results of the model or attribute the differences with actuals. If you can't explain your models to business, you will not be valued by it.

One more thing that is particularly important for forecasting. Let's say your model is dependent on N exogenous variables. This means that you have to first obtain the forecasts of these variables in order to forecast your dependent variable. Having smaller N makes your life easier, so a simpler model is easier to use.

$\endgroup$
2
  • $\begingroup$ Although you mention forecasting, most of your answer seems to apply only to explanatory modeling. $\endgroup$
    – rolando2
    Jan 13, 2016 at 22:44
  • $\begingroup$ @rolando2, it sounds like that because in my domain you can't simply hand the forecast to users. We have to explain the forecast, link it to drivers etc. When you get weather forecast you don't normally ask the forecaster to explain you why exactly they think it's going to rain with 50% chance. In my case I not only have to do it, but do it in a way that my consumers understand the results by linking it to business drivers that they deal with daily. That's why parsimony is valuable in its own right $\endgroup$
    – Aksakal
    Jan 14, 2016 at 0:32
2
$\begingroup$

Regarding Neural Networks, this topic has been very nicely covered in a recent NeurIPS 2020 paper entitled

The Pitfalls of Simplicity Bias in Neural Networks (Shah et al.).

I totally recommend reading the paper, I think it is very nicely structured and rigorous. Here is an attempt to summarize its main ideas:


  • The core of the problem is that the field seems to associate simplicity bias to "justify why NNs work well", but not to their "lack of robustness".

  • The authors tackle this problem via the design of datasets that comprise multiple predictive features with varying levels of simplicity. These will serve as control variables.

  • Following a series of theoretical analyses and targeted experiments, they found that the simplicity bias given by Stochastic Gradient Descent can be extreme, that common approaches to overcome this can fail, and in general, that there is still quite a way to go in order to better understand SB in NNs.

The proposed datasets and methods can serve as a basis for future work in that direction.


This principle of SB for NNs is very much embedded in the Deep Learning culture: See e.g. this video from the Udacity "Deep Learning" course (Vanhoucke et al.), where they use the "stretch pants" analogy. The idea is that instead of trying to guess the right pant size, you start with several extra sizes and "stretch them down" until they fit perfectly. This way, training is seen as reducing the capacity of the model wherever there is room to do so.

The paper above questions the general validity of this principle, showing that when trained with SGD, the pants tend to fit the simpler legs, no matter how many dimensions they have.

So whereas the original question is from 2015, this topic is still extremely current. Particularly, the following OP sentence is right on point:

Today's computing power enables us to build increasingly complex models with ever-greater ability for prediction. As a result of this increasing ceiling in computing power, do we really still need to gravitate toward simplicity?

Since the paper above shows that the ever-growing NN architectures still invariably gravitate towards simpler features when trained with SGD.


I would also like to finish commenting about another bias related to SB, that is Inductive Bias, "the set of assumptions that the learner uses to predict outputs of given inputs that it has not encountered".

Instead of assuming simplicity, inductive bias assumes any other arbitrary property. For example, when training a NN for transcription of piano music, it is safe to assume that all keyboard strokes generate mainly harmonic waveforms that have an onset and decay exponentially. In terms of "no free lunch", correctly applying an inductive bias would translate to getting lunch for a good prize. Physical models are known for their extremely precise and rigorous application of inductive biases.

Another amazing ICLR 2020 paper that treats the systematic introduction of Inductive Bias into NN design and training is

DDSP: DIFFERENTIABLE DIGITAL SIGNAL PROCESSING (Engel et al.)

With application of DDSP techniques it can be observed that introducing inductive biases helps the models satisfactorily solve complex tasks with less capacity, also the SGD convergence is faster and requires less training data.

So again coming back to the OP: While the No Free Lunch theorem states that no algorithm will work better than other for all kinds of problems, inductive biases are a good mechanism to gain performance without sacrificing much interpretability, if you are willing to reduce the set of targeted problems.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.