1
$\begingroup$

In machine learning with statistical approach, does the bias of the model solely depend on the selection of the model class without considering the training data? There is a claim regarding the bias-variance decomposition in a technical report (Settles Burr, Active Learning Literature Survey, 2009) saying:

The second term is the bias, which represents the error due to the model class itself, e.g., if a linear model is used to learn a function that is only approximately linear. This component of the overall error is invariant given a fixed model class. The third term is the model’s variance, which is the remaining component of the learner’s mean squared error with respect to the true regression function. Minimizing the variance, then, is guaranteed to minimize the future generalization error of the model (since the learner itself can do nothing about the noise or bias components)

I'm wondering why this is true or not. And why does the variance depend on the training data instead? I search online for a while but there is no answer for such doubt. I also consult with GPT, well, its answer resonate with the claim in the paper, and saying "no formal proof can be given as it is conceptual understanding of what bias represents in the bias-variance decomposition and how it relates to the choice of model class" (by GPT).

$\endgroup$
1
  • $\begingroup$ My conclusion to this raised question is simple, according to the the selected answer by @Richard, the bias is treated as invariant if and only if the model is assumed to be unbiased. I.e., the model has zero bias, thus remain invariant regarding the dataset. $\endgroup$ Feb 24 at 4:28

1 Answer 1

0
$\begingroup$

Consider a data generating process $$Y=f(X)+\varepsilon$$ where $\varepsilon$ is independent of $x$ with $\mathbb E(\varepsilon)=0$ and $\text{Var}(\varepsilon)=\sigma^2_\varepsilon$. According to Hastie et al. "The Elements of Statistical Learning" (2nd edition, 2009) Section 7.3 p. 223, we can derive an expression for the expected prediction error of a regression fit $\hat g(X)$ at an input point $X=x_0$, using squared-error loss:

\begin{align} \text{Err}(x_0) &=\mathbb E[(Y-\hat g(x_0))^2|X=x_0]\\ &=(\mathbb E[\hat g(x_0)−f(x_0)])^2+\mathbb E[(\hat g(x_0)−\mathbb E[\hat g(x_0)])^2]+\sigma^2_\varepsilon\\ &=\text{Bias}^2\ \ \ \quad\quad\quad\quad\quad\;\;+\text{Variance } \quad\quad\quad\quad\quad\quad+ \text{ Irreducible Error} \end{align}

(where I use the notation $\text{Bias}^2$ instead of $\text{Bias}$ and $\hat g$ instead of $\hat f$). As noted further on p. 224 (and referred to in this post), for linear models the averaged-over-$x_0$ bias can be further decomposed neatly into model bias and estimation bias, $$ \mathbb E_{x_0}\big[ (\mathbb E[\hat g(x_0)−f(x_0)])^2 \big] = \mathbb E_{x_0}\big[ (f(x_0)-x_0^{\top}\beta_*)^2 \big] + \mathbb E_{x_0}\big[ (x_0^{\top}\beta_* - \mathbb E_{x_0}[ x_0^{\top}\beta_\alpha ])^2 \big] $$ where $\beta_*=\arg\min_\beta \mathbb E[(f(X)-X^\top\beta)^2]$ are the parameters of the best-fitting linear approximation to $f$ and $\beta_\alpha$ are the estimated parameters. This shows that even if a model class is unbiased ($\text{Bias}^2=0$ for $\beta_*$), having a biased estimator $\beta_\alpha$ will result in $\text{Bias}^2\neq 0$. It is possible that the authors of your quote consider model class to define not only the model but also the estimator, and then they would be right, but without such a qualification I do not think they are.

$\endgroup$
2
  • $\begingroup$ Thanks for the detailed reply! Can we just make it more simple? Like, just assume our model is unbiased! Then the bias term just taken as Zero. I see most of the empirical results showing that the models of high complexity normally have Zero bias, but relatively higher variance. Do you think these empirical results can somehow support the Unbiased model assumption to hold for high complexity models, e.g., Neural Network? $\endgroup$ Feb 21 at 9:21
  • $\begingroup$ @sharp_flyingrain, is that a new question? If so, please ask it on a separate thread. The comments here can be used for clarifying your original question and my answer to it. $\endgroup$ Feb 21 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.