Optimization: The root of all evil in statistics?

Question

I have heard the following expression before:

"Optimization is the root of all evil in statistics".

For example, the top answer in this thread makes that statement in reference to the danger of optimizing too aggressively during model selection.

My first question is the following: Is this quote attributable to anyone in particular? (e.g. in the statistics literature)

From what I understand, the statement refers to the risks of overfitting. Traditional wisdom would say that proper cross validation already fights against this problem, but it looks like there is more to this problem than that.

Should statisticians & ML practitioners be wary of over-optimizing their models even when adhering to strict cross validation protocols (e.g. 100 nested 10-fold CV) ? If so, how do we know when to stop searching for "the best" model?

Answer 1

The quote is a paraphrase of a Donald Knuth quote, one which he has himself attributed to Hoare. Three extracts from the above page:

Premature optimization is the root of all evil (or at least most of it) in programming.

$ $

Premature optimization is the root of all evil.

$ $

Knuth refers to this as "Hoare's Dictum" 15 years later ...

I don't know that I agree with the statistics paraphrase*. There's plenty of 'evil' in statistics that doesn't relate to optimization.

Should statisticians & ML practitioners always be wary of over-optimizing their models even when adhering to strict cross validation protocols (e.g. 100 nested 10-fold CV) ? If so, how do we know when to stop searching for "the best" model?

I think the critical thing is to fully understand (or as fully as is feasible) the properties of what procedures you undertake.

$\,^\text{* I won't presume to comment on Knuth's use of it, since there's little I could}$ $\quad ^\text{say that he couldn't rightly claim to understand ten times as well as I do.}$

Answer 2

A couple of ways you could parse the quote (in statistics), assuming optimization refers to (data-driven) model selection:

• If you care about prediction, you may be better off with model averaging instead of selecting a single model.
• If you select a model on the same dataset used to fit the model, it will wreak havoc on the usual inference tools/procedures that assume you had chosen the model a priori. (Say you do stepwise regression, choosing the model size by cross-validation. For a Frequentist analysis, the usual p-values or CIs for the chosen model will be incorrect. I'm sure there are corresponding problems for Bayesian analyses that involve model selection.)
• If your dataset is large enough compared to the family of models you consider, overfitting might not even be a problem and model selection may be unnecessary. (Say you're going to fit a linear regression using a dataset with few variables and very many observations. Any spurious variables should get coefficients estimated close to 0 anyway, so perhaps you needn't even bother selecting a smaller model.)
• If your dataset is small enough, you might not have enough data to fit the "true" or "best" model for the problem. What does it even mean to do model-selection well, in that case? (Back to linear regression: Should you aim to select the "true" model with the right variables, even if you don't have enough data to measure them all adequately? Should you just pick the largest model for which you do have enough data?)
• Finally, even when it's clear you can and should do model selection, cross-validation is not a panacea. It has many variants and even its own tuning parameter (number of folds, or train:test ratio) which impacts its properties. So don't trust it blindly.