14
$\begingroup$

I have heard the following expression before:

"Optimization is the root of all evil in statistics".

For example, the top answer in this thread makes that statement in reference to the danger of optimizing too aggressively during model selection.

My first question is the following: Is this quote attributable to anyone in particular? (e.g. in the statistics literature)

From what I understand, the statement refers to the risks of overfitting. Traditional wisdom would say that proper cross validation already fights against this problem, but it looks like there is more to this problem than that.

Should statisticians & ML practitioners be wary of over-optimizing their models even when adhering to strict cross validation protocols (e.g. 100 nested 10-fold CV) ? If so, how do we know when to stop searching for "the best" model?

$\endgroup$
5
  • $\begingroup$ The second question definitely stands on its own though, does it not? $\endgroup$ Commented Jul 11, 2013 at 1:09
  • $\begingroup$ @Glen_b I left a reference in that thread. But just to clarify, Dikran already suggested to open follow up questions to his answer in separate threads, and I doubt this question should be addressed in a comment. $\endgroup$ Commented Jul 11, 2013 at 1:09
  • 3
    $\begingroup$ @RussellS.Pierce The edit that contained the question as it stands now - even though it was made before my comment - wasn't there when I loaded it and started researching the original question and possible answers, it only had what I'd describe as a rhetorical question there. The question as it stands now is fine. $\endgroup$
    – Glen_b
    Commented Jul 11, 2013 at 1:13
  • 2
    $\begingroup$ The cross-validation issue is covered in another question here: stats.stackexchange.com/questions/29354/… Cross-validation can help to avoid over-fitting, but it doesn't solve the problem completely. Knowing when to stop can be a decidedly tricky problem, and I don't think there can be a general solution. $\endgroup$ Commented Jul 11, 2013 at 11:32
  • 2
    $\begingroup$ "proper cross validation already fights against this problem, but it looks like there is more to this problem than that." Yes: the problem is still the variance of the cross validation estimates (combined with a multiple testing problematic). If I find the time, I'll write an answer for your related question. $\endgroup$
    – cbeleites
    Commented Jul 11, 2013 at 12:44

2 Answers 2

14
$\begingroup$

The quote is a paraphrase of a Donald Knuth quote, one which he has himself attributed to Hoare. Three extracts from the above page:

Premature optimization is the root of all evil (or at least most of it) in programming.

$ $

Premature optimization is the root of all evil.

$ $

Knuth refers to this as "Hoare's Dictum" 15 years later ...

I don't know that I agree with the statistics paraphrase*. There's plenty of 'evil' in statistics that doesn't relate to optimization.

Should statisticians & ML practitioners always be wary of over-optimizing their models even when adhering to strict cross validation protocols (e.g. 100 nested 10-fold CV) ? If so, how do we know when to stop searching for "the best" model?

I think the critical thing is to fully understand (or as fully as is feasible) the properties of what procedures you undertake.

$\,^\text{* I won't presume to comment on Knuth's use of it, since there's little I could}$ $\quad ^\text{say that he couldn't rightly claim to understand ten times as well as I do.}$

$\endgroup$
2
  • 2
    $\begingroup$ Thanks, this is helpful. I think there are some interesting connections between pre-mature optimization in programming and overfitting. I wonder if there are any similar quotes in our community, and if there is a rigorous way to fight this in statistics. $\endgroup$ Commented Jul 11, 2013 at 1:25
  • 5
    $\begingroup$ My use of the phrase was inspired by Knuth's, although the reason is different and from a Bayesian perspective pretty much all optimisation is a bad thing and marginalizing is better. $\endgroup$ Commented Jul 11, 2013 at 11:29
4
$\begingroup$

A couple of ways you could parse the quote (in statistics), assuming optimization refers to (data-driven) model selection:

  • If you care about prediction, you may be better off with model averaging instead of selecting a single model.
  • If you select a model on the same dataset used to fit the model, it will wreak havoc on the usual inference tools/procedures that assume you had chosen the model a priori. (Say you do stepwise regression, choosing the model size by cross-validation. For a Frequentist analysis, the usual p-values or CIs for the chosen model will be incorrect. I'm sure there are corresponding problems for Bayesian analyses that involve model selection.)
  • If your dataset is large enough compared to the family of models you consider, overfitting might not even be a problem and model selection may be unnecessary. (Say you're going to fit a linear regression using a dataset with few variables and very many observations. Any spurious variables should get coefficients estimated close to 0 anyway, so perhaps you needn't even bother selecting a smaller model.)
  • If your dataset is small enough, you might not have enough data to fit the "true" or "best" model for the problem. What does it even mean to do model-selection well, in that case? (Back to linear regression: Should you aim to select the "true" model with the right variables, even if you don't have enough data to measure them all adequately? Should you just pick the largest model for which you do have enough data?)
  • Finally, even when it's clear you can and should do model selection, cross-validation is not a panacea. It has many variants and even its own tuning parameter (number of folds, or train:test ratio) which impacts its properties. So don't trust it blindly.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.