Definition from Wikipedia:

$\operatorname {Bias} _{\theta }[\,{\hat {\theta }}\,]=\operatorname {E} _{\theta }[\,{\hat {\theta }}\,]-\theta =\operatorname {E} _{\theta }[\,{\hat {\theta }}-\theta \,],$ where ${\displaystyle \operatorname {E} _{\theta }} $ denotes expected value over the distribution ${\displaystyle P(x\mid \theta )}$, i.e. averaging over all possible observations ${\displaystyle x}$ . The second equation follows since θ is measurable with respect to the conditional distribution ${\displaystyle P(x\mid \theta )} .$

Is this a general definition of bias? Because here, we are conditioning on $\theta$, so this implies that $\theta$ is a random variable, whereas in frequentist approach, we assume that the true parameter $\theta$ is fixed but unknown, so $P(x; \theta)$

Second definition I came across somewhere:

where the expectation of estimator is over the data (seen as samples from a random variable)

What does it mean in this context that expectation of estimator is over data? Could you provide exact formula for $\operatorname {E} _{\theta }[\,{\hat {\theta }}\,]$?


Definition from Wikipedia seems clear to me.

An estimator $\hat{\theta}$ is simply a function of observed data $X$, i.e. a statistic. We have $\hat{\theta} = g(X)$.

Bias is taken w.r.t. conditional distribution of $X$ on specific $\theta$, so

$$E_{\theta}[\hat{\theta}] = E_{\theta}[g(X)] = \int_{X} g(x) p(x|\theta) dx, \text{ for some } \theta \in \Theta $$

To elaborate a bit, we say that an estimator $\hat{\theta}$ of $\theta$ is unbiased if

$$E_{\theta}[\hat{\theta} - \theta] = 0,\ \text{for every } \theta \in \Theta$$

  • 1
    $\begingroup$ In frequentist statistics $\theta$ is not a random variable. You can condition only on random variables. $\endgroup$
    – mokebe
    Feb 11 '17 at 12:49
  • $\begingroup$ Also, why not $E_{θ}[\hat{θ}]=E_{\theta}[g(X)]$? And why in second definition given by me "expectation is over the data(seen as samples from a random variable)"? In your definition expectation is taken over different values of x and is based on some underlying "real" data generating distribution $p(x; \theta)$. So your definition of expected value is based on probabilty of "real" data distribution and "true" parameter $\theta$, not on the distribution of some sample of "real" data (estimated distribution based on sampled data) $\endgroup$
    – mokebe
    Feb 11 '17 at 13:19
  • $\begingroup$ data $X$ can be single observation or a whole i.i.d. sample $X_1, \dots, X_n$ $\endgroup$ Feb 11 '17 at 13:21
  • $\begingroup$ @mokebe I think you have a problem with understanding the underlying concept of statistical model, we have a family of ditributions $\{P_{\theta}; \theta \in \Theta\}$. We assume we know what family it is, and estimate only an unknown $\theta$. I can't be more precise sorry $\endgroup$ Feb 11 '17 at 13:24
  • 1
    $\begingroup$ But if $\theta$ is constant then $p(x|\theta)=p(x)$ so as a result it is better to write $p(x;\theta)$ to highlight that $\theta$ is not a random variable in this case, but a parameter. $\endgroup$
    – mokebe
    Feb 11 '17 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.