For a stochastic data generating process (DGP) $$ Y=f(X)+\varepsilon $$ and a model producing a point prediction $$ \hat{Y}=\hat{f}(X), $$ the bias-variance decomposition is
\begin{align} \text{Err}(x_0) &=\mathbb E[(Y-\hat f(x_0))^2|X=x_0]\\ &=(\mathbb E[\hat f(x_0)−f(x_0)])^2+\mathbb E[(\hat f(x_0)−\mathbb E[\hat f(x_0)])^2]+\sigma^2_\varepsilon\\ &=\text{Bias}^2\ \ \ \quad\quad\quad\quad\quad\;\;+\text{Variance } \quad\quad\quad\quad\quad\quad+ \text{ Irreducible Error} \end{align}
(Hastie et al. "The Elements of Statistical Learning" (2nd edition, 2009) Section 7.3 p. 223; I use the notation $\text{Bias}^2$ instead of $\text{Bias}$). If there is a range models to choose from, the highly flexible ones will have low bias and high variance and will tend to overfit. The inflexible ones will have high bias and low variance and will tend to underfit. The model yielding the lowest expected squared error will be somewhere in between the two extremes.
For a deterministic DGP that lacks the additive random error, $$ Y=f(X), $$ the bias-variance decomposition tells us that variance and irreducible error are zero and only bias is left. If there is a range models to choose from, choosing the most flexible one will yield the lowest bias and hence the lowest expected squared error. This suggests it is impossible to overfit when the DGP is deterministic.
To me this sounds too good to be true. Perhaps the caveat is that the models here use the same set of regressors as the DGP, i.e. all the relevant variables are being considered and no irrelevant variables are included. This is unlikely to hold in practice. If the sets of regressors in the models vs. the DGP differ, there might be different story.
Questions:
- Does my reasoning for why it is impossible to overfit a deterministic DGP make sense? If not, why?
- Does the reasoning break down if the regressors used in the DGP and the model differ? If so, how?
Update: In practice, many DGPs could be considered entirely deterministic or almost deterministic with a negligible stochastic component, even though their mechanisms may be too complex for us to comprehend, let alone model accurately. If the answer to Q1 is that the reasoning is sound and the answer to Q2 is that the reasoning does not break down, as suggested by @markowitz, then overfitting should rarely be of concern in practice. This seems counterintuitive to me...