7
$\begingroup$

Given an IID normally distributed sample $X_1,...,X_n$ for $n$ small with mean $\mu$, standard deviation $\sigma$, sample mean $\overline{X}$ and sample standard deviation $s$ (the unbiased estimator form). I understand that

$$\frac{\overline{X} - \mu}{\frac{\sigma}{\sqrt{n}}} \sim N(0,1),$$

but I'm having trouble reconciling this with the fact that

$$\frac{\overline{X} - \mu}{\frac{s}{\sqrt{n}}} \sim t_{n-1}.$$

Since the $t$-distribution is like the standard normal distribution but with a higher variance (smaller peak and fatter tails), this would seem to suggest that the sample standard deviation $s$ systematically underestimates the population standard deviation, making it in fact a biased estimator.

$\endgroup$
  • $\begingroup$ $s^2$ is unbiased for $\sigma^2,$ so, yes, $s$ is biased for $\sigma.$ $\endgroup$ – soakley Jul 22 '14 at 1:51
5
$\begingroup$

Actually $s$ doesn't need to systematically underestimate $\sigma$; this could happen even if that weren't true.

As it is, $s$ is biased for $\sigma$ (the fact that $s^2$ is unbiased for $\sigma^2$ means that $s$ will be biased for $\sigma$, due to Jensen's inequality*, but that's not the central thing going on there.


* Jensen's inequality

If $g$ is a convex function, $g\left(\text{E}[X]\right) \leq \text{E}\left[g(X)\right]$ with equality only if $X$ is constant or $g$ is linear.

Now $g(X)=-\sqrt{X}$ is convex,

so $-\sqrt{\text{E}[X]} < \text{E}(-\sqrt{X})$, i.e. $\sqrt{\text{E}[X]} > \text{E}(\sqrt{X})\,$, implying $\sigma>E(s)$ if the random variable $s$ is not a fixed constant.


Edit: a simpler demonstration not invoking Jensen --

Assume that the distribution of the underlying variable has $\sigma>0$.

Note that $\text{Var}(s) = E(s^2)-E(s)^2$ this variance will always be positive for $\sigma>0$.

Hence $E(s)^2 = E(s^2)-\text{Var}(s) < \sigma^2$, so $E(s)<\sigma$.


So what is the main issue?

Let $Z=\frac{\overline{X} - \mu}{\frac{\sigma}{\sqrt{n}}}$

Note that you're dealing with $t=Z\cdot\frac{\sigma}{s}$.

That inversion of $s$ is important. So the effect on the variance it's not whether $s$ is smaller than $\sigma$ on average (though it is, very slightly), but whether $1/s$ is larger than $1/\sigma$ on average (and those two things are NOT the same thing).

And it is larger, to a greater extent than its inverse is smaller.

Which is to say $E(1/X)\neq 1/E(X)$; in fact, from Jensen's inequality:

$g(X) = 1/x$ is convex, so if $X$ is not constant,

$1/\left(\text{E}[X]\right) < \text{E}\left[1/X\right]$

So consider, for example, normal samples of size 10; $s$ is about 2.7% smaller than $\sigma$ on average, but $1/s$ is about 9.4% larger than $1/\sigma$ on average. So even if at n=10 we made our estimate of $\sigma$ 2.7-something percent larger** so that $E(\widehat\sigma)=\sigma$, the corresponding $t=Z\cdot\frac{\sigma}{\widehat\sigma}$ would not have unit variance - it would still be a fair bit larger than 1.

**(at other $n$ the adjustment would be different of course)

Since the t-distribution is like the standard normal distribution but with a higher variance (smaller peak and fatter tails)

If you adjust for the difference in spread, the peak is higher.


Why does the t-distribution become more normal as sample size increases?

The standard normal distribution vs the t-distribution

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.