Are all estimators biased? Is the unbiasedness only a theoretical or approximation case?

Question

The definition of unbiased estimator says that it's expected value has no difference comparing to a true value. So can we say that all estimators are biased (even slightly)? I thought that only in terms of infinite sampling we can converge to a true value, hence in this conditions estimators are not biased. Is the term unbiased estimator just a generalization or some real scenario issue? Why I see many times that people are calling estimators used in a statistical inference - unbiased. Is it just a approximation isue? What about convergence to a true value and infinite variance of population? Any help would highly appreciated!

Answer 1

As you said, unbiasedness is a theoretical property of an estimator because the expected value operation is theoretical, which means it doesn't depend on the sample size. So, an estimator is either biased or unbiased and doesn't change its state according to $n$.

I thought that only in terms of infinite sampling we can converge to a true value

I'd also like to comment on this one:

For example, let $\hat \theta_n$ be the n-sample estimator of parameter $\theta$. If $\mathbb E[\hat \theta_n]=\theta$, the estimator is unbiased and this means

• as $n$ increases, you get close to true $\theta$
• when averaged among all n-sample estimates, the estimator will converge to the true $\theta$ (this statement is the implication of the expected value)

Note that, the first one can also be satisfied by unbiased estimators, but the second cannot. Take for example, $\mathbb E[\hat\theta_n]=\frac{n}{n-1}\theta$. Then,

$$\lim_{n\rightarrow\infty}\mathbb E[\hat\theta_n]=\lim_{n\rightarrow\infty}\frac{n}{n-1}\theta=\theta$$

So, as the sample size, $n$, increases, the unbiased estimator converges to the true parameter.