I understand that bias (in the context of bias-variance tradeoff) is "the expectation of the difference between the true, and estimated, parameters". However, I would like to understand who is biased and towards what. There are many articles which clearly explain what bias is, but this point is not mentioned.
$\begingroup$
$\endgroup$
2
-
2$\begingroup$ It's worth noting that the bias-variance tradeoff is a theoretical concept and, as such, is not subject to empirical test or verification. $\endgroup$– user78229Commented Dec 1, 2017 at 13:27
-
$\begingroup$ @DJohnson Thanks for adding that point to this post $\endgroup$– Rohit GavvalCommented Dec 4, 2017 at 9:30
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Your parameter estimates are biased. They are systematically either too large or too small.
Let's say the true parameter is $\theta$, and your estimate is $\hat \theta$. The bias is, as you say, the expected difference between the two, $\mathbb E[\hat \theta - \theta] = \mathbb E[\hat \theta] - \theta$ (true parameter treated as a constant). The bias is decided by the expected value of the estimated parameter. Downward bias means the parameter estimates are expected to be smaller than the true value they estimate; upward bias means the parameter estimates are expected to be larger than the true value they estimate.
It is not always possible to tell what the bias is as you need to know the true value $\theta$, which in many cases you can't.
$\begingroup$
$\endgroup$
Who is biased? the estimator $\hat\theta$ as an estimator of $\theta$
Towards what? It depends on the situation. All you can say generally is that an estimator is biased away from $\theta$ but in what direction depends on the situation.Tautologically speaking, it is biased in the direction $\frac{E(\hat\theta)-\theta}{\|E(\hat\theta)-\theta\|}$ or towards $E(\hat\theta)$.
Example: Ridge estimator for linear regression. If you regularize towards $\theta_0$, the estimator is biased towards $\theta_0$.
