Can a biased but consistent estimator have a non zero convergent bias? I understand that an estimator can be biased and yet consistent, and for me intuitivly in these cases the bias converge to zero as n goes to infinity, however can it be the case that the bias won't converge to zero in such an estimator? 
 A: This answer is adapted from an example on Wikipedia.
Let $\theta$ be a parameter of interest, let $\delta > 0$ be fixed, and let $(\widehat{\theta}_n)_{n=1}^\infty$ be a sequence of estimators of $\theta$ with the following discrete distribution under $\theta$:
$$
\begin{aligned}
P_\theta(\widehat{\theta}_n = \theta) &= 1 - 1/n, \\
P_\theta(\widehat{\theta}_n = n \delta + \theta) &= 1/n.
\end{aligned}
$$

Claim.
$\widehat{\theta}_n \overset{P_\theta}{\to} \theta$ as $n \to \infty$ (so the sequence $(\widehat{\theta}_n)_{n=1}^\infty$ is consistent), but the bias $E_\theta[\widehat{\theta}_n - \theta]$ does not converge to $0$ as $n \to \infty$.

Proof.
To show consistency, we must show that
$$
\lim_{n\to\infty} P_\theta(|\widehat{\theta}_n - \theta| > \varepsilon) = 0
$$
for all $\varepsilon > 0$.
Thus, let $\varepsilon > 0$ be given.
Note that for any $n$ we have
$$
\begin{aligned}
P_\theta(|\widehat{\theta}_n - \theta| > \varepsilon)
&= P_\theta(|\widehat{\theta}_n - \theta| > \varepsilon, \widehat{\theta}_n = \theta)
+ P_\theta(|\widehat{\theta}_n - \theta| > \varepsilon, \widehat{\theta}_n = n \delta + \theta) \\
&= P_\theta(|\theta - \theta| > \varepsilon, \widehat{\theta}_n = \theta)
+ P_\theta(|n \delta + \theta - \theta| > \varepsilon, \widehat{\theta}_n = n \delta + \theta) \\
&= P_\theta(0 > \varepsilon, \widehat{\theta}_n = \theta)
+ P_\theta(n \delta > \varepsilon, \widehat{\theta}_n = n \delta + \theta) \\
&= 0
+ P_\theta(n \delta > \varepsilon, \widehat{\theta}_n = n \delta + \theta) \\
&= \begin{cases}
0, & \text{if $n\delta \leq \varepsilon$,} \\
1/n, & \text{if $n\delta > \varepsilon$.}
\end{cases}
\end{aligned}
$$
In particular, we see that as $n \to \infty$, $P_\theta(|\widehat{\theta}_n - \theta| > \varepsilon) \to 0$, so $(\widehat{\theta}_n)_{n=1}^\infty$ is a consistent sequence of estimators.
However, we have
$$
\begin{aligned}
E_\theta[\widehat{\theta}_n - \theta]
&= (\theta - \theta) P_\theta(\widehat{\theta}_n = \theta)
+ (n \delta + \theta - \theta) P_\theta(\widehat{\theta}_n = n \delta + \theta) \\
&= n \delta P_\theta(\widehat{\theta}_n = n \delta + \theta) \\
&= \delta
\end{aligned}
$$
for all $n$, so the bias is constant and positive.
