31
$\begingroup$

Just a thought:

Parsimonious models have always been the default go-to in model selection, but to what degree is this approach outdated? I'm curious about how much our tendency toward parsimony is a relic of a time of abaci and slide rules (or, more seriously, non-modern computers). Today's computing power enables us to build increasingly complex models with ever-greater ability for prediction. As a result of this increasing ceiling in computing power, do we really still need to gravitate toward simplicity?

Sure, simpler models are easier to understand and interpret, but in the age of ever-growing data sets with greater numbers of variables and a shift towards a greater focus on prediction capability, this might no longer even be achievable or necessary.

Thoughts?

$\endgroup$
  • 4
    $\begingroup$ With apologies to Richard Hamming: The purpose of modeling is insight, not numbers. Complicated models impede insight. $\endgroup$ – Eric Towers Jul 28 '15 at 15:47
  • 12
    $\begingroup$ Models that are oversimplified impede insight even more. $\endgroup$ – Frank Harrell Jul 28 '15 at 16:35
  • 6
    $\begingroup$ It may depend on the application; in physics, I think the argument for parsimony will have a strong basis. However, many applications will have a host of small effects that can't be eliminated (consider models for political preferences, for example) . A number of workers suggest that use of regularization (such as methods that lead to shrinkage or in many applications shrinkage of differences, or both) rather than elimination of variables makes more sense; others lean toward some selection and some shrinkage (LASSO for example, does both). $\endgroup$ – Glen_b Jul 28 '15 at 17:50
  • 3
    $\begingroup$ Parsimonious models are not the "go-to" in model selection. Otherwise we would always model everything with its sample mean and call it a day. $\endgroup$ – shadowtalker Jul 29 '15 at 16:29
  • 1
    $\begingroup$ Also, some food for thought: Mease and Wyner (2008) recommend richer learners in AdaBoost, which is a little unintuitive. An open question in that line of research seems to be whether parsimonious base learners actually lead to parsimonious ensembles. $\endgroup$ – shadowtalker Jul 29 '15 at 16:33
25
$\begingroup$

@Matt's original answer does a great job of describing one of the benefits of parsimony but I don't think it actually answers your question. In reality, parsimony isn't the gold standard. Not now nor has it ever been. A "gold standard" related to parsimony is generalization error. We would like to develop models that don't overfit. That are as useful for prediction (or as interpretable or with minimum error) out of sample as they are in sample. It turns out (because of things laid out above) that parsimony is actually quite a good proxy for generalization error but it's by no means the only one.

Really, think about why we use cross validation or bootstrapping or train/test sets. The goal is to create models with good generalization accuracy. A lot of the time, these ways of estimating out of sample performance do end up choosing models with lower complexity but not always. As an extreme example imagine the oracle hands us the true but extremely complex model and a poor but parsimonious model. If parsimony was really our goal then we would choose the second but in reality, the first is what we would like to learn if we could. Unfortunately a lot of the time that last sentence is the kicker, "if we could".

$\endgroup$
  • $\begingroup$ Which is "the original answer"? $\endgroup$ – mattdm Jul 29 '15 at 21:54
  • $\begingroup$ :) fair enough. Matt's comment. $\endgroup$ – Nick Thieme Jul 29 '15 at 22:32
22
$\begingroup$

Parsimonious models are desirable not just due to computing requirements, but also for generalization performance. It's impossible to achieve the ideal of infinite data that completely and accurately covers the sample space, meaning that non-parsimonious models have the potential to overfit and model noise or idiosyncrasies in the sample population.

It's certainly possible to build a model with millions of variables, but you'd be using variables that have no impact on the output to model the system. You could achieve great predictive performance on your training dataset, but those irrelevant variables will more than likely decrease your performance on an unseen test set.

If an output variable truly is the result of a million input variables, then you would do well to put them all in your predictive model, but only if you have enough data. To accurately build a model of this size, you'd need several million data points, at minimum. Parsimonious models are nice because in many real-world systems, a dataset of this size simply isn't available, and furthermore, the output is largely determined by a relatively small number of variables.

$\endgroup$
  • 5
    $\begingroup$ +1. I suggest reading The Elements of Statistical Learning (freely available on the web), which discusses this problem in depth. $\endgroup$ – Stephan Kolassa Jul 28 '15 at 14:42
  • 3
    $\begingroup$ On the other hand, when you have millions of variables and few objects, it is likely that purely by chance some variables are better at explaining outcome that the true interaction. In such case parsimony-based modelling will be more susceptible to overfitting than a brute-force approach. $\endgroup$ – user88 Jul 28 '15 at 19:36
  • $\begingroup$ @CagdasOzgenc For instance a large random subspace ensemble. $\endgroup$ – user88 Jul 28 '15 at 19:53
  • $\begingroup$ I feel like something like a Lasso approach could apply here. $\endgroup$ – theforestecologist Jan 13 '16 at 20:24
17
$\begingroup$

I think the previous answers do a good job of making important points:

  • Parsimonious models tend to have better generalization characteristics.
  • Parsimony is not truly a gold standard, but just a consideration.

I want to add a few comments that come out of my day to day job experience.

The generalization of predictive accuracy argument is, of course, strong, but is academically bias in its focus. In general, when producing a statistical model, the economies are not such that predictive performance is a completely dominant consideration. Very often there are large outside constraints on what a useful model looks like for a given application:

  • The model must be implementable within an existing framework or system.
  • The model must be understandable by a non-technical entity.
  • The model must be efficient computationally.
  • The model must be documentable.
  • The model must pass regulatory constraints.

In real application domains, many if not all of these considerations come before, not after, predictive performance - and the optimization of model form and parameters is constrained by these desires. Each of these constraints biases the scientist towards parsimony.

It may be true that in many domains these constraints are being gradually lifted. But it is the lucky scientist indeed that gets to ignore them are focus purely on minimizing generalization error.

This can be very frustrating for the first time scientist, fresh out of school (it definitely was for me, and continues to be when I feel that the constraints placed on my work are not justified). But in the end, working hard to produce an unacceptable product is a waste, and that feels worse than the sting to your scientific pride.

$\endgroup$
  • 2
    $\begingroup$ No parsimony is not a consideration. A sound inference procedure MUST rank a parsimonious model over a non-parsimonious model if they explain the data equally well. Otherwise total compressed codelength of the model and the data encoded by the model will not be the smallest. So yes it is a gold standard. $\endgroup$ – Cowboy Trader Jul 28 '15 at 16:47
  • 3
    $\begingroup$ Parsimony is NOT a "gold standard"! That statement is preposterous. If it were true, then why don't we always build models that fit nothing but the unconditional mean? We trade off bias and variance with reference to either a test set or, better still, completely new observations, and we do so within the constraints of our field, organization, and the law. Sometimes you only have enough information to make naive predictions. Sometimes you've got enough to add complexity. $\endgroup$ – Brash Equilibrium Jul 29 '15 at 4:45
  • 1
    $\begingroup$ @BrashEquilibrium I think what Cagdas is saying is, given the choice between equally predictive models, one should choose the most parsimonious one. $\endgroup$ – Matthew Drury Jul 29 '15 at 5:31
  • 1
    $\begingroup$ Ah. That's a different thing. Yes, in that case choose the most parsimonious model. I still don't think that amounts to parsimony being a "gold standard" though. $\endgroup$ – Brash Equilibrium Jul 29 '15 at 13:30
  • 1
    $\begingroup$ @MatthewDrury Brash, Cagdas. Interesting. Perhaps, parsimony is just one component of the gold standard; which is probably (or ought to be) better based around the notion of encompassing. A good exposition of this idea is provided in the following astrophysics lecture from Yale: oyc.yale.edu/astronomy/astr-160/lecture-11. 7:04 onwards. The idea also features in the econometric/forecasting literature by David Hendry and Grayham Mizon. They argue that encompassing is part of a progressive research strategy, of which parsimony is a single aspect. $\endgroup$ – Graeme Walsh Jan 14 '16 at 1:03
14
$\begingroup$

I think this is a very good question. In my opinion parsimony is overrated. Nature is rarely parsimonious, and so we shouldn't necessarily expect accurate predictive or descriptive models to be so either. Regarding the question of interpretability, if you choose a simpler model that only modestly conforms to reality merely because you can understand it, what exactly are you understanding? Assuming a more complex model had better predictive power, it would appear to be closer to the actual facts anyways.

$\endgroup$
  • 8
    $\begingroup$ Well said @dsaxton. There is a great misunderstanding of parsimony and a great under-appreciation of how volatile feature selection is. Parsimony is nice when it results from pre-specification. Most parsimony that results from data dredging is misleading and is only understood because it's wrong. $\endgroup$ – Frank Harrell Jul 28 '15 at 16:48
  • 2
    $\begingroup$ @FrankHarrell Would you elaborate on "only understood because it's wrong", or perhaps link to something you wrote previously about this? This is an interesting point that I would like to make sure I understand. $\endgroup$ – gui11aume Jul 28 '15 at 17:24
  • 8
    $\begingroup$ This is an extreme example but people who engage in racial profiling think they understand, with a single feature (e.g., skin color), what value someone has. To them the answer is simple. They only understand it because they are making a wrong judgment by oversimplifying. Parsimony is usually an illusion (except in Newtonian mechanics and a few other areas). $\endgroup$ – Frank Harrell Jul 28 '15 at 18:05
  • 1
    $\begingroup$ "Nature is rarely parsimonious": and one point where nature is particularly non-parsimonious is individuals (as opposed to our typical sample sizes!). Evolution uses a whole new population of new individuals each generation... IMHO the parsimony (Frank Harrell's pre-specified type - allowing any n of m available features into the model is in fact a very complex model - even if n << m, this is a non-so-small fraction of the original search space) is how we try to get at least something out of our far-too-small data sets. $\endgroup$ – cbeleites Dec 23 '17 at 13:54
2
$\begingroup$

Parsimony is not a golden start. It's an aspect in modeling. Modeling and especially forecasting can not be scripted, i.e. you can't just hand a script to a modeler to follow. You rather define principles upon which the modeling process must be based. So, the parsimony is one of these principles, application of which can not be scripted (again!). A modeler will consider the complexity when a selecting model.

Computational power has little to do with this. If you're in the industry your models will be consumed by business folks, product people, whoever you call them. You have to explain your model to them, it should make a sense to them. Having parsimonious models helps in this regard.

For instance, you're forecasting product sales. You should be able to describe what are the drivers of sales, and how they work. These must be related to concepts with which business operates, and the correlations must be understood and accepted by business. With complex models it could be very difficult to interpret the results of the model or attribute the differences with actuals. If you can't explain your models to business, you will not be valued by it.

One more thing that is particularly important for forecasting. Let's say your model is dependent on N exogenous variables. This means that you have to first obtain the forecasts of these variables in order to forecast your dependent variable. Having smaller N makes your life easier, so a simpler model is easier to use.

$\endgroup$
  • $\begingroup$ Although you mention forecasting, most of your answer seems to apply only to explanatory modeling. $\endgroup$ – rolando2 Jan 13 '16 at 22:44
  • $\begingroup$ @rolando2, it sounds like that because in my domain you can't simply hand the forecast to users. We have to explain the forecast, link it to drivers etc. When you get weather forecast you don't normally ask the forecaster to explain you why exactly they think it's going to rain with 50% chance. In my case I not only have to do it, but do it in a way that my consumers understand the results by linking it to business drivers that they deal with daily. That's why parsimony is valuable in its own right $\endgroup$ – Aksakal Jan 14 '16 at 0:32
1
$\begingroup$

Perhaps have a review of the Akaike Information Criterion, a concept that I only discovered by serendipity yesterday. The AIC seeks to identify which model and how many parameters are the best explanation for the observations at hand, rather than any basic Occam's Razor, or parsimony approach.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.