question about MSE mean square error

Question

The following is taken literally from Wikipedia's mean squared error in the mean subheading:

"Suppose we have a random sample of size $n$ from a population, $X_1$, ... ,$X_n$. Suppose the sample units were chosen with replacement. That is, the $n$ units are selected one at a time, and previously selected units are still eligible for selection for all $n$ draws. The usual estimator for the $\mu$ is the sample average

$\bar{x}$=$\frac{1}{n}\sum_{i=1}^n X_i $

which has an expected value equal to the true mean $\mu$ (so it is unbiased) and a mean square error of

$MSE (\bar{X}) = E[(\bar{x}-\mu)^2] = \frac{\sigma^2}{n}$ where $\sigma^2$ is the population variance.

For a Gaussian distribution this is the best unbiased estimator (that is, it has the lowest MSE among all unbiased estimators), but not, say, for a Uniform distribution (continuous) uniform distribution."

My question is, I don't quite understand the very last sentence of the Mean subheading, "For a Gaussian distribution this is the best unbiased estimator (that is, it has the lowest MSE among all unbiased estimators), but not, say, for a Uniform distribution (continuous) uniform distribution."

Could someone please explain and give an example of the last paragraph?

Answer 1

I will try to give an intuitive example to understand why the arithmetic mean \begin{equation} \overline x_1 = \sum_{i=1}^{n} \frac{x_i}{n} \end{equation} is not as good as \begin{equation} \overline x_2 = \frac{a + b}{2} \end{equation} In the case where $X \sim \mathrm{unif}(\alpha,\beta)$

Imagine that you have 10 observations from $\mathrm{unif}(1,11)$ (There is a candy factory that puts a candy of 1cm, 2cm, ..., 11cm, 1cm, 2cm,..., 11cm,... in separate bags). We know from the above information that the real mean is 6 (For the factory example, the factory would have spend the same amount of sugar if each candy was 6cm)

Now, if you don't know any of the above and take a random sample to estimate that, then $\bar x_2$ would only require the smallest and the highest number to appear in your sample and that's it! It would always guess the correct answer with 0 error! $\bar x_1$ on the other hand, would be sensitive to every single value that you get and it will "fluctuate" around the real value. Furthermore, if the highest (or equivalently the lowest) value doesn't appear in your sample, then again $\bar x_2$ will almost always be closer to the real mean compared to $\bar x_1$. $\bar x_1$ will be better only if your sample is already centralized around 6 which is less likely to happen compared to the other possible scenarios.

For the candy factory example. If you try to predict the "average candy" that is in each bag, it's better to get the average between the smallest and the largest candy you had so far than averaging the candies in every single bag you open and change your prediction (and thus error) after every bag.

Answer 2

Note firstly that MSE, unlike the related measurement; variance (Var), is a biased estimator of a sample variability, which is one source of confusion in the text quoted. For a normal distribution (only) the relationship is $MSE(\bar{X})=\frac{n-1}{n}Var(\bar{X})$, with a more general relationship given through excess kurtosis correction. This confusion arises because as a sample MSE becomes a population MSE and a population Var, as $\lim_{n\to\infty}\frac{n-1}{n}=1$, that is, MSE is unbiased for a (very large) population, but is biased for $n$ small. The Wikipedia entry is admixing $\mu$ with $\bar{X}$ and $s^2$ with $\sigma^2$, and this is confusing, but is clearly referring to the population (very large $n$) case only. The entry attempts to gloss over this difficulty using the sentence "Suppose the sample units were chosen with replacement.", which is a curt description of bootstrap somewhat unconvincingly implying that we can make $n$ appear to be very large, using $n$ small. Also, note that there are limitations that apply to MSE usage.

More confusion arises not from MSE itself, but from a consideration of what to best use as the estimator of location for the individual distributions that MSE is applied to, as follows. For a normal distribution the mean value is a minimum variance unbiased estimator MVUE for location. However, for a uniform distribution (UD) the mean value is not MVUE for location. The UD mean value is unbiased, no problem there.

However, for a uniform distribution the midrange value, i.e., $\dfrac{\mathrm{max}-\mathrm{min}}{2}$, is also unbiased and is a better estimator of location than the mean value because the variance for the midrange value is less than the variance of the mean. To put it another way, for $n$ increasing, the UD midrange value converges faster to the central value than the mean value does.

Finally, Harter (1942) cites that Fisher (1922) "...has shown (p. 321) that the distribution of the mean of a sample of any size from a Cauchy distribution that same as that of a single observation and (pp. 348-351) that for a sample from a rectangular distribution the extreme observations contain the whole of the relevant information in the sample; as the sample size $n$ increases, the mean square error of the midrange decreases like $n^{-1}$ and that of the mean only like $n^{-\frac{1}{2}}$."

So, in other words, for a uniform distribution, MSE of midrange, $MSE\left(\dfrac{\mathrm{max}-\mathrm{min}}{2}\right)$ is preferred to MSE of the mean, $MSE(\bar{X})$ .