0
$\begingroup$

I'm having some trouble relating unbiased estimators and bias error. By bias error, I mean the bias error we talk about when analyzing "bias-variance tradeoffs." Is this bias error and an unbiased estimator related in any way?

I know that an unbiased estimator means that the expected value of the estimator is equal to the true value of the parameter being estimated.

For example, for linear regression, we have the linear model

$$ y = X\beta + \epsilon $$ where $\beta$ is the true parameter that can't be observed. For least squares, we approximate this parameter with $$ \hat{\beta} = (X^TX)^{-1}X^Ty $$

Under the Gauss-Markov theorem, we can show that $E[\hat{\beta}] = \beta$, indicating that the least squares estimator is unbiased. Does an unbiased estimator mean that the bias error is zero?

If so, then I am confused by this because the two seems to be separate concepts to me albeit both using the term "bias." Wikipedia defines bias error as "The bias error is an error from erroneous assumptions in the learning algorithm. High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting)." Even though we have an unbiased estimator $\hat{\beta}$, we could still produce a model that doesn't accurately reflect the underlying data (for example using a linear model to approximate data that is non-linear).

$\endgroup$
5
  • $\begingroup$ The confusion comes from different usages of the term "bias" from the precise "expectation of the estimator is the estimated quantity" to an equivalent for "error" and "misspecified". $\endgroup$
    – Xi'an
    May 21 '20 at 6:44
  • $\begingroup$ @Xi'an Thanks. Just to clarify, the 2 usages of "bias" that I outlined in my OP are distinct from each other? $\endgroup$
    – roulette01
    May 21 '20 at 16:36
  • $\begingroup$ yes indeed..... $\endgroup$
    – Xi'an
    May 21 '20 at 17:17
  • $\begingroup$ related stats.stackexchange.com/questions/473380/… $\endgroup$
    – stats101
    Jun 22 '20 at 8:16
  • $\begingroup$ @Xi'an I asked a similar question in stats.stackexchange.com/questions/473710/…. When the squared error is decomposed into a bias, variance, and noise term, an unbiased estimator does not mean the bias term in the decomposition of the error is zero, right? $\endgroup$
    – David
    Jun 25 '20 at 18:24
1
$\begingroup$

Under the Gauss-Markov theorem, we can show that E[β^]=β, indicating that the least squares estimator is unbiased. Does an unbiased estimator mean that the bias error is zero?

By definition, bias error is the difference between the expected value of the estimator and the true value being estimated. Therefore yes, when an estimator is unbiased it does directly imply that the bias error is 0. Directly from the wiki An estimator or decision rule with zero bias is called unbiased.

... we could still produce a model that doesn't accurately reflect the underlying data (for example using a linear model to approximate data that is non-linear)

I think you're actually right here, but it doesn't necessarily contradict the definition of an unbiased estimator. You can have models where the estimator is unbiased and it paramaterises a model which the data wasn't actually drawn from. All an unbiased estimator means is that the expectation of the error term is zero for the parameters. So what I mean by this is, it is not hard to think of multiple different models which have the same expectation of the output for a given matrix of independent variables, but for values outside the support it could differ. I could be wrong about this though, if two models parameters expectations have the same value at every point on the curve, they might necessarily have to be the same model.

$\endgroup$
2
  • $\begingroup$ "All an unbiased estimator means is that the expectation of the error term is zero for the parameters." I have some other questions, but just want to clarify my understanding first. Individual errors can take on + or - values right? For example, if you fit a linear linear through a dataset, we could define all the points above the line as having positive error, and those below as having negative error. On average, we'd expect the errors to be zero? $\endgroup$
    – roulette01
    May 21 '20 at 4:41
  • $\begingroup$ Is this the answer to : stats.stackexchange.com/questions/473380/… $\endgroup$
    – stats101
    Jun 22 '20 at 8:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.