13
$\begingroup$

Really stumped on this one. I would really like an example or situation where an estimator B would be both consistent and biased.

$\endgroup$
6
  • 3
    $\begingroup$ This is for a class? $\endgroup$
    – Glen_b
    Sep 25, 2015 at 13:44
  • 5
    $\begingroup$ I think that the late specification that you're looking for a time series example transforms this into a different question, as it would invalidate the excellent answers already provided. But this is fine -- You can ask a new question. $\endgroup$
    – Sycorax
    Sep 25, 2015 at 14:21
  • 6
    $\begingroup$ I see you have changed your question. Given that several answers already dealt with your previous question, I advise you to change it back and post a new question specifically for time series models. $\endgroup$
    – JohnK
    Sep 25, 2015 at 14:33
  • 3
    $\begingroup$ It is surprising that even though you ask for a time series related estimator, no one has mentioned OLS for an AR(1). The estimator is biased, but consistent, and it is fairly easy to show (and googling will give you plenty of material on this). Edit: it appears as the time series request was a late addition, which would explain the lack of such answers... $\endgroup$
    – hejseb
    Sep 25, 2015 at 18:35
  • 2
    $\begingroup$ Here's a pretty trivial example: $\bar{X}_n + \epsilon / n$, $\epsilon \neq 0$. $\endgroup$
    – dsaxton
    Jan 16, 2016 at 0:02

4 Answers 4

26
$\begingroup$

The simplest example I can think of is the sample variance that comes intuitively to most of us, namely the sum of squared deviations divided by $n$ instead of $n-1$:

$$S_n^2 = \frac{1}{n} \sum_{i=1}^n \left(X_i-\bar{X} \right)^2$$

It is easy to show that $E\left(S_n^2 \right)=\frac{n-1}{n} \sigma^2$ and so the estimator is biased. But assuming finite variance $\sigma^2$, observe that the bias goes to zero as $n \to \infty$ because

$$E\left(S_n^2 \right)-\sigma^2 = -\frac{1}{n}\sigma^2 $$

It can also be shown that the variance of the estimator tends to zero and so the estimator converges in mean-square. Hence, it is also convergent in probability.

$\endgroup$
6
  • 1
    $\begingroup$ This is a useful example, though it may apply a rather weak interpretation of "biased" here (which is used somewhat ambiguously in the question itself). One could also ask for something stronger, e.g., a sequence of estimator that is consistent, but with bias that does not vanish even asymptotically. $\endgroup$
    – cardinal
    Sep 25, 2015 at 14:22
  • $\begingroup$ @cardinal The bias must vanish asymptotically in order for the estimator to be consistent, no? $\endgroup$
    – JohnK
    Sep 25, 2015 at 14:24
  • 3
    $\begingroup$ Nope. (See comment stream for more details.) $\endgroup$
    – cardinal
    Sep 25, 2015 at 14:29
  • $\begingroup$ I would think it would be helpful call your estimator $\hat \sigma^2$ rather than $S^2$, as $S^2$ most typically refers to the unbiased estimator, while $\hat \sigma^2$ often refers to the MLE. $\endgroup$
    – Cliff AB
    Sep 25, 2015 at 15:20
  • $\begingroup$ @CliffAB Yes, this is what the index $n$ denotes, the sum of squared deviations is divided by $n$, instead of the conventional $n-1$. $\endgroup$
    – JohnK
    Sep 25, 2015 at 15:22
10
$\begingroup$

A simple example would be estimating the parameter $\theta > 0$ given $n$ i.i.d. observations $y_i \sim \text{Uniform}\left[0, \,\theta\right]$.

Let $\hat{\theta}_n = \max\left\{y_1, \ldots, y_n\right\}$. For any finite $n$ we have $\mathbb{E}\left[\theta_n\right] < \theta$ (so the estimator is biased), but in the limit it will equal $\theta$ with probability one (so it is consistent).

$\endgroup$
1
7
$\begingroup$

Consider any unbiased and consistent estimator $T_n$ and a sequence $\alpha_n$ converging to 1 ($\alpha_n$ need not to be random) and form $\alpha_nT_n$. It is biased, but consistent since $\alpha_n$ converges to 1.

From wikipedia:

Loosely speaking, an estimator $T_n$ of parameter $\theta$ is said to be consistent, if it converges in probability to the true value of the parameter: $$\underset{n\to\infty}{\operatorname{plim}}\;T_n = \theta.$$

Now recall that the bias of an estimator is defined as:

$$\operatorname{Bias}_\theta[\,\hat\theta\,] = \operatorname{E}_\theta[\,\hat{\theta}\,]-\theta $$

The bias is indeed non zero, and the convergence in probability remains true.

$\endgroup$
2
  • $\begingroup$ I appreciate the response and explanation. I have a better understanding now. Thanks $\endgroup$ Sep 25, 2015 at 13:05
  • $\begingroup$ This answer needs a minor fix-up at the beginning to make clear that not any unbiased $T_n$ will do. The original estimator sequence itself must be consistent. $\endgroup$
    – cardinal
    Sep 25, 2015 at 14:18
2
$\begingroup$

In a time series setting with a lagged dependent variable included as a regressor, the OLS estimator will be consistent but biased. The reason for this is that in order to show unbiasedness of the OLS estimator we need strict exogeneity, $E\left[\varepsilon_{t}\left|x_{1},\, x_{2,},\,\ldots,\, x_{T}\right.\right] $, i.e. that the error term, $\varepsilon_{t} $, in period $t $ is uncorrelated with all the regressors in all time periods. However, in order to show consistency of the OLS estimator we only need contemporanous exogeneity, $E\left[\varepsilon_{t}\left|x_{t}\right.\right] $, i.e. that the error term, $\varepsilon_{t} $, in period $t $ is uncorrelated with the regressors, $x_{t} $ in period $t $. Consider the AR(1) model: $y_{t}=\rho y_{t-1}+\varepsilon_{t},\;\varepsilon_{t}\sim N\left(0,\:\sigma_{\varepsilon}^{2}\right)$ with $x_{t}=y_{t-1} $ from now on.

First I show that strict exogeneity does not hold in a model with a lagged dependent variable included as a regressor. Let's look at the correlation between $\varepsilon_{t} $ and $x_{t+1}=y_{t} $ $$E\left[\varepsilon_{t}x_{t+1}\right]=E\left[\varepsilon_{t}y_{t}\right]=E\left[\varepsilon_{t}\left(\rho y_{t-1}+\varepsilon_{t}\right)\right] $$

$$=\rho E\left(\varepsilon_{t}y_{t-1}\right)+E\left(\varepsilon_{t}^{2}\right) $$

$$=E\left(\varepsilon_{t}^{2}\right)=\sigma_{\varepsilon}^{2}>0 \ (Eq. (1)).$$

If we assume sequential exogeneity, $E\left[\varepsilon_{t}\mid y_{1},\: y_{2},\:\ldots\ldots,y_{t-1}\right]=0 $, i.e. that the error term, $\varepsilon_{t} $, in period $t $ is uncorrelated with all the regressors in previous time periods and the current then the first term above, $\rho E\left(\varepsilon_{t}y_{t-1}\right) $, will dissapear. What is clear from above is that unless we have strict exogeneity the expectation $E\left[\varepsilon_{t}x_{t+1}\right]=E\left[\varepsilon_{t}y_{t}\right]\neq0 $. However, it should be clear that contemporaneous exogeneity, $E\left[\varepsilon_{t}\left|x_{t}\right.\right] $, does hold.

Now let's look at the bias of the OLS estimator when estimating the AR(1) model specified above. The OLS estimator of $\rho $, $\hat{\rho} $ is given as:

$$\hat{\rho}=\frac{\frac{1}{T}\sum_{t=1}^{T}y_{t}y_{t-1}}{\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}}=\frac{\frac{1}{T}\sum_{t=1}^{T}\left(\rho y_{t-1}+\varepsilon_{t}\right)y_{t-1}}{\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}}=\rho+\frac{\frac{1}{T}\sum_{t=1}^{T}\varepsilon_{t}y_{t-1}}{\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}} \ (Eq. (2))$$

Then take conditional expectation on all previous, contemporaneous and future values, $E\left[\varepsilon_{t}\left|y_{1},\, y_{2,},\,\ldots,\, y_{T-1}\right.\right] $, of $Eq. (2)$:

$$E\left[\hat{\rho}\left|y_{1},\, y_{2,},\,\ldots,\, y_{T-1}\right.\right]=\rho+\frac{\frac{1}{T}\sum_{t=1}^{T}\left[\varepsilon_{t}\left|y_{1},\, y_{2,},\,\ldots,\, y_{T-1}\right.\right]y_{t-1}}{\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}} $$

However, we know from $Eq. (1)$ that $E\left[\varepsilon_{t}y_{t}\right]=E\left(\varepsilon_{t}^{2}\right) $ such that $\left[\varepsilon_{t}\left|y_{1},\, y_{2,},\,\ldots,\, y_{T-1}\right.\right]\neq0 $ meaning that $\frac{\frac{1}{T}\sum_{t=1}^{T}\left[\varepsilon_{t}\left|y_{1},\, y_{2,},\,\ldots,\, y_{T-1}\right.\right]y_{t-1}}{\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}}\neq0 $ and hence $E\left[\hat{\rho}\left|y_{1},\, y_{2,},\,\ldots,\, y_{T-1}\right.\right]\neq\rho $ but is biased: $E\left[\hat{\rho}\left|y_{1},\, y_{2,},\,\ldots,\, y_{T-1}\right.\right]=\rho+\frac{\frac{1}{T}\sum_{t=1}^{T}\left[\varepsilon_{t}\left|y_{1},\, y_{2,},\,\ldots,\, y_{T-1}\right.\right]y_{t-1}}{\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}}=\rho+\frac{\frac{1}{T}\sum_{t=1}^{T}E\left(\varepsilon_{t}^{2}\right)y_{t-1}}{\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}}=$$\rho+\frac{\frac{1}{T}\sum_{t=1}^{T}\sigma_{\varepsilon}^{2}y_{t-1}}{\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}} $.

All I assume to show consistency of the OLS estimator in the AR(1) model is contemporanous exogeneity, $E\left[\varepsilon_{t}\left|x_{t}\right.\right]=E\left[\varepsilon_{t}\left|y_{t-1}\right.\right]=0 $ which leads to the moment condition, $E\left[\varepsilon_{t}x_{t}\right]=0 $ with $x_{t}=y_{t-1} $. As before, we have that the OLS estimator of $\rho $, $\hat{\rho} $ is given as: $$\hat{\rho}=\frac{\frac{1}{T}\sum_{t=1}^{T}y_{t}y_{t-1}}{\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}}=\frac{\frac{1}{T}\sum_{t=1}^{T}\left(\rho y_{t-1}+\varepsilon_{t}\right)y_{t-1}}{\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}}=\rho+\frac{\frac{1}{T}\sum_{t=1}^{T}\varepsilon_{t}y_{t-1}}{\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}} $$

Now assume that $plim\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}=\sigma_{y}^{2} $ and $\sigma_{y}^{2} $ is positive and finite, $0<\sigma_{y}^{2}<\infty $.

Then, as $T\rightarrow\infty $ and as long as a law of large numbers (LLN) applies we have that $p\lim\frac{1}{T}\sum_{t=1}^{T}\varepsilon_{t}y_{t-1}=E\left[\varepsilon_{t}y_{t-1}\right]=0 $. Using this result we have: $$\underset{T\rightarrow\infty}{p\lim\hat{\rho}}=\rho+\frac{p\lim\frac{1}{T}\sum_{t=1}^{T}\varepsilon_{t}y_{t-1}}{p\lim\frac{1}{T}\sum_{t=1}^{T}y_{t}^{2}}=\rho+\frac{0}{\sigma_{y}^{2}}=\rho $$

Thereby it has been shown that the OLS estimator of $p $, $\hat{\rho} $ in the AR(1) model is biased but consistent. Note that this result holds for all regressions where the lagged dependent variable is included as a regressor.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.