3
$\begingroup$

Suppose that I have two statistics that are known to be consistent , e.g : $ S_{n} ^2 $ (biased sample variance about sample mean) and $ S_{n-1}^2$ (bessel-corrected sample variance, that is unbiased).

I know that bias and consistency are different things but my question is:

  • does the bias matter in how (slow or fast) the sequence converges?
$\endgroup$
1
$\begingroup$

To measure the speed of convergence of a consistent estimator to the parameter it estimates, you need to specify some measure that will quantify the closeness of the estimator and the parameter, so that you can measure the accuracy of the estimator at each $n$, and for a given parameter value. An example of this would be to use the mean-square-error (MSE), given by:

$$\text{MSE}(\hat{\theta}_n) = \mathbb{E} \Big( (\hat{\theta}_n - \theta)^2 \Big| \theta \Big).$$

This gives you a measure of closeness of the estimator $\hat{\theta}_n$ to the parameter $\theta$. For this particular measure, it can be shown that:

$$\text{MSE}(\hat{\theta}_n) = \mathbb{V}(\hat{\theta}_n | \theta) + \text{Bias}(\hat{\theta}_n, \theta)^2.$$

You can see from this expression that the mean-squared-error is affected both by the bias of the estimator and also its variance. The overall MSE is an interaction of these two things.


Application to variance estimation: To preserve clarity of the value of $n$ used, I am going to vary your notation to denote the standard variance estimator (with Bessel's correction) as $S_n^2$ and the (uncorrected) sample variance estimator as $B_n^2$. Now, using moment results from O'Neill (2014), and taking $\kappa$ as the kurtosis parameter in the underlying distribution, we have:

$$\begin{equation} \begin{aligned} \text{MSE}(S_n^2) &= \mathbb{V}(S_n^2 | \sigma) + \text{Bias}(S_n^2, \sigma^2)^2 \\[6pt] &= \Bigg( \kappa - \frac{n-1}{n-3} \Bigg) \frac{\sigma^4}{n} + 0^2 \\[6pt] &= \Bigg( \kappa - \frac{n-1}{n-3} \Bigg) \frac{\sigma^4}{n}, \\[10pt] \text{MSE}(B_n^2) &= \mathbb{V}(B_n^2 | \sigma) + \text{Bias}(B_n^2, \sigma^2)^2 \\[6pt] &= \mathbb{V}\Big( \frac{n-1}{n} \cdot S_n^2 \Big| \sigma \Big) + \mathbb{E} \Big( \frac{n-1}{n} \cdot S_n^2 - \sigma^2 \Big| \sigma \Big)^2 \\[6pt] &= \Big( \frac{n-1}{n} \Big)^2 \Bigg( \kappa - \frac{n-1}{n-3} \Bigg) \frac{\sigma^4}{n} + \Big( \frac{n-1}{n} - 1 \Big)^2 \sigma^4 \\[6pt] &= \Big( \frac{n-1}{n} \Big)^2 \Bigg( \kappa - \frac{n-1}{n-3} \Bigg) \frac{\sigma^4}{n} + \frac{\sigma^4}{n^2} \\[6pt] &= \Bigg[ \Big( \frac{n-1}{n} \Big)^2 \Big( \kappa - \frac{n-1}{n-3} \Big) + \frac{1}{n} \Bigg] \frac{\sigma^4}{n}. \\[6pt] \end{aligned} \end{equation}$$

Hence, the difference of these error measures is:

$$\text{MSE}(S_n^2) - \text{MSE}(B_n^2) = \Bigg[ \Big( \kappa - \frac{n-1}{n-3} \Big) \cdot \frac{2n-1}{n^2} - \frac{1}{n} \Bigg] \frac{\sigma^2}{n}.$$

With a bit of algebra it can be shown that:

$$\text{MSE}(S_n^2) > \text{MSE}(B_n^2) \quad \quad \iff \quad \quad \kappa > \frac{n-3}{n-1} + \frac{n}{2n-1}.$$

The above formulae give you explicit expressions for the MSE of the two estimators for any value of $n$. As $n \rightarrow \infty$ we have $\text{MSE}(S_n^2) > \text{MSE}(B_n^2)$ if and only if $\kappa > \tfrac{3}{2}$, which is the case for distributions other than highly platykurtic distributions. So you can see that, other that in the case of highly platykurtic distributions, the biased sample variance estimator has lower MSE than the unbiased sample variance estimator.


Note: There are other measures you could potentially use to look at how close the estimator is to the parameter it is estimating. More generally, these measures are usually expected values of some "loss" function that depends on the distance of the estimator from the parameter. The MSE uses squared-error loss, which is a common loss function.


Correction: A previous version of this answer had an error in the MSE difference which flowed through to later results; this has now been corrected.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.