# Bias and expected value of estimator

Definition from Wikipedia:

$\operatorname {Bias} _{\theta }[\,{\hat {\theta }}\,]=\operatorname {E} _{\theta }[\,{\hat {\theta }}\,]-\theta =\operatorname {E} _{\theta }[\,{\hat {\theta }}-\theta \,],$ where ${\displaystyle \operatorname {E} _{\theta }}$ denotes expected value over the distribution ${\displaystyle P(x\mid \theta )}$, i.e. averaging over all possible observations ${\displaystyle x}$ . The second equation follows since θ is measurable with respect to the conditional distribution ${\displaystyle P(x\mid \theta )} .$

Is this a general definition of bias? Because here, we are conditioning on $\theta$, so this implies that $\theta$ is a random variable, whereas in frequentist approach, we assume that the true parameter $\theta$ is fixed but unknown, so $P(x; \theta)$

Second definition I came across somewhere:

where the expectation of estimator is over the data (seen as samples from a random variable)

What does it mean in this context that expectation of estimator is over data? Could you provide exact formula for $\operatorname {E} _{\theta }[\,{\hat {\theta }}\,]$?

Definition from Wikipedia seems clear to me.

An estimator $\hat{\theta}$ is simply a function of observed data $X$, i.e. a statistic. We have $\hat{\theta} = g(X)$.

Bias is taken w.r.t. conditional distribution of $X$ on specific $\theta$, so

$$E_{\theta}[\hat{\theta}] = E_{\theta}[g(X)] = \int_{X} g(x) p(x|\theta) dx, \text{ for some } \theta \in \Theta$$

To elaborate a bit, we say that an estimator $\hat{\theta}$ of $\theta$ is unbiased if

$$E_{\theta}[\hat{\theta} - \theta] = 0,\ \text{for every } \theta \in \Theta$$

• In frequentist statistics $\theta$ is not a random variable. You can condition only on random variables. Feb 11 '17 at 12:49
• Also, why not $E_{θ}[\hat{θ}]=E_{\theta}[g(X)]$? And why in second definition given by me "expectation is over the data(seen as samples from a random variable)"? In your definition expectation is taken over different values of x and is based on some underlying "real" data generating distribution $p(x; \theta)$. So your definition of expected value is based on probabilty of "real" data distribution and "true" parameter $\theta$, not on the distribution of some sample of "real" data (estimated distribution based on sampled data) Feb 11 '17 at 13:19
• data $X$ can be single observation or a whole i.i.d. sample $X_1, \dots, X_n$ Feb 11 '17 at 13:21
• @mokebe I think you have a problem with understanding the underlying concept of statistical model, we have a family of ditributions $\{P_{\theta}; \theta \in \Theta\}$. We assume we know what family it is, and estimate only an unknown $\theta$. I can't be more precise sorry Feb 11 '17 at 13:24
• But if $\theta$ is constant then $p(x|\theta)=p(x)$ so as a result it is better to write $p(x;\theta)$ to highlight that $\theta$ is not a random variable in this case, but a parameter. Feb 11 '17 at 13:44