I have heard the following expression before:
"Optimization is the root of all evil in statistics".
For example, the top answer in this thread makes that statement in reference to the danger of optimizing too aggressively during model selection.
My first question is the following: Is this quote attributable to anyone in particular? (e.g. in the statistics literature)
From what I understand, the statement refers to the risks of overfitting. Traditional wisdom would say that proper cross validation already fights against this problem, but it looks like there is more to this problem than that.
Should statisticians & ML practitioners be wary of over-optimizing their models even when adhering to strict cross validation protocols (e.g. 100 nested 10-fold CV) ? If so, how do we know when to stop searching for "the best" model?