In ADALINE algorithm , with $y=x*w+b$, where $x$ is the feature vector of a sample and $w$ is the weight vector, the update rule (SGD) for the bias $b$ is: $b \leftarrow b + \eta(o - y)$.

With Gradient Descent, the final $b$ is actually: $$b \leftarrow b_{init} + \eta\sum_{i=1}^n(o_i - y_i)$$ where $b_{init}$ is a random initial value for $b$.

On the other hand, according to Bias–variance tradeoff, the mean square error decomposition is: $$Err(x)=Bias^2+Variance+Irreducible\ Error$$, where $Bias$ is: $$Bias=E[\hat{f}(x)]-f(x)$$

  • What is the relation between $b$ and $Bias$? Are they the same thing?
  • Regarding the equation calculation, is $E[\hat{f}(x)]-f(x)$ the sum of errors like $b$?

1 Answer 1


No, they are not the same thing.

In the first example, bias refers to the intercept of a linear model. It is simply a parameter just like $w$ that needs to be learned.

In the second example, bias is the difference between the ground truth and the expected prediction of the model. Here, the expectation is taken over different training sets from the same ground truth data distribution. It expresses how much is the specific model going to be wrong on average, thus the name bias.

  • $\begingroup$ So it seems they don't have any relation except they both are called bias. For the second $Bias$, do people actually calculate/analyze it for a certain purpose? Or it is simply an abstract concept for understanding the $Bias VS. Variance$? $\endgroup$ Jun 18, 2018 at 14:47
  • $\begingroup$ I have never seen it calculated in practice (I suppose taking expectation over generally unknown distribution is not feasible). $\endgroup$ Jun 18, 2018 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.