4
$\begingroup$

The following is taken literally from Wikipedia's mean squared error in the mean subheading:

"Suppose we have a random sample of size $n$ from a population, $X_1$, ... ,$X_n$. Suppose the sample units were chosen with replacement. That is, the $n$ units are selected one at a time, and previously selected units are still eligible for selection for all $n$ draws. The usual estimator for the $\mu$ is the sample average

$\bar{x}$=$\frac{1}{n}\sum_{i=1}^n X_i $

which has an expected value equal to the true mean $\mu$ (so it is unbiased) and a mean square error of

$MSE (\bar{X}) = E[(\bar{x}-\mu)^2] = \frac{\sigma^2}{n}$ where $\sigma^2$ is the population variance.

For a Gaussian distribution this is the best unbiased estimator (that is, it has the lowest MSE among all unbiased estimators), but not, say, for a Uniform distribution (continuous) uniform distribution."

My question is, I don't quite understand the very last sentence of the Mean subheading, "For a Gaussian distribution this is the best unbiased estimator (that is, it has the lowest MSE among all unbiased estimators), but not, say, for a Uniform distribution (continuous) uniform distribution."

Could someone please explain and give an example of the last paragraph?

$\endgroup$
  • $\begingroup$ Hello Cowboy, could you elaborate more about what you mean when you said it saturates the bound. I guess you mean it actually attains the lower bound? But how about for uniform? could you show why it is not? $\endgroup$ – john_w Nov 3 '18 at 15:43
3
$\begingroup$

I will try to give an intuitive example to understand why the arithmetic mean \begin{equation} \overline x_1 = \sum_{i=1}^{n} \frac{x_i}{n} \end{equation} is not as good as \begin{equation} \overline x_2 = \frac{a + b}{2} \end{equation} In the case where $X \sim \mathrm{unif}(\alpha,\beta)$

Imagine that you have 10 observations from $\mathrm{unif}(1,11)$ (There is a candy factory that puts a candy of 1cm, 2cm, ..., 11cm, 1cm, 2cm,..., 11cm,... in separate bags). We know from the above information that the real mean is 6 (For the factory example, the factory would have spend the same amount of sugar if each candy was 6cm)

Now, if you don't know any of the above and take a random sample to estimate that, then $\bar x_2$ would only require the smallest and the highest number to appear in your sample and that's it! It would always guess the correct answer with 0 error! $\bar x_1$ on the other hand, would be sensitive to every single value that you get and it will "fluctuate" around the real value. Furthermore, if the highest (or equivalently the lowest) value doesn't appear in your sample, then again $\bar x_2$ will almost always be closer to the real mean compared to $\bar x_1$. $\bar x_1$ will be better only if your sample is already centralized around 6 which is less likely to happen compared to the other possible scenarios.

For the candy factory example. If you try to predict the "average candy" that is in each bag, it's better to get the average between the smallest and the largest candy you had so far than averaging the candies in every single bag you open and change your prediction (and thus error) after every bag.

$\endgroup$
  • $\begingroup$ I'm assuming you meant $\bar x_2$ will again be better than $\bar x_1$ even if either the highest or lowest is missing. You wrote $\bar x_1$ twice. $\endgroup$ – Heteroskedastic Jim Nov 4 '18 at 15:32
  • $\begingroup$ Yes, all fixed by the user who edited my answer $\endgroup$ – Vasilis Vasileiou Nov 4 '18 at 16:58
  • 1
    $\begingroup$ I edited it. Keeping with the intuitive explanation which I +1, I think a comment about why the mid-range approach does not work similarly for the normal distribution might be helpful. Maybe something to do with how all values including a and b have equal densities with a continuous uniform distribution. $\endgroup$ – Heteroskedastic Jim Nov 4 '18 at 17:24
2
$\begingroup$

Note firstly that MSE, unlike the related measurement; variance (Var), is a biased estimator of a sample variability, which is one source of confusion in the text quoted. For a normal distribution (only) the relationship is $MSE(\bar{X})=\frac{n-1}{n}Var(\bar{X})$, with a more general relationship given through excess kurtosis correction. This confusion arises because as a sample MSE becomes a population MSE and a population Var, as $\lim_{n\to\infty}\frac{n-1}{n}=1$, that is, MSE is unbiased for a (very large) population, but is biased for $n$ small. The Wikipedia entry is admixing $\mu$ with $\bar{X}$ and $s^2$ with $\sigma^2$, and this is confusing, but is clearly referring to the population (very large $n$) case only. The entry attempts to gloss over this difficulty using the sentence "Suppose the sample units were chosen with replacement.", which is a curt description of bootstrap somewhat unconvincingly implying that we can make $n$ appear to be very large, using $n$ small. Also, note that there are limitations that apply to MSE usage.

More confusion arises not from MSE itself, but from a consideration of what to best use as the estimator of location for the individual distributions that MSE is applied to, as follows. For a normal distribution the mean value is a minimum variance unbiased estimator MVUE for location. However, for a uniform distribution (UD) the mean value is not MVUE for location. The UD mean value is unbiased, no problem there.

However, for a uniform distribution the midrange value, i.e., $\dfrac{\mathrm{max}-\mathrm{min}}{2}$, is also unbiased and is a better estimator of location than the mean value because the variance for the midrange value is less than the variance of the mean. To put it another way, for $n$ increasing, the UD midrange value converges faster to the central value than the mean value does.

Finally, Harter (1942) cites that Fisher (1922) "...has shown (p. 321) that the distribution of the mean of a sample of any size from a Cauchy distribution that same as that of a single observation and (pp. 348-351) that for a sample from a rectangular distribution the extreme observations contain the whole of the relevant information in the sample; as the sample size $n$ increases, the mean square error of the midrange decreases like $n^{-1}$ and that of the mean only like $n^{-\frac{1}{2}}$."

So, in other words, for a uniform distribution, MSE of midrange, $MSE\left(\dfrac{\mathrm{max}-\mathrm{min}}{2}\right)$ is preferred to MSE of the mean, $MSE(\bar{X})$ .

$\endgroup$
  • $\begingroup$ Hi Carl, thanks for your answer. However when you said "For a normal distribution (only), the relationship is $MSE(x\bar)=\frac{n-1}{n}Var(x\bar)$". But the definition of MSE of say an estimator A is the Expected Value of the deviation from truth, so it would be E(A - E(A)) , and this usually equal a "truth" value, and doesn't change from sample to sample right? But in your explanation in this particular sentence, does it mean the expected value will change according to the sample in this case? because normally I thought $\endgroup$ – john_w Nov 4 '18 at 22:07
  • $\begingroup$ the expected value is a unique "truth" value, and doesn't change from sample to sample. Is it because the "something" is biased, so that is why it changes from sample to sample? thanks $\endgroup$ – john_w Nov 4 '18 at 22:10
  • $\begingroup$ I linked to the bias proof: en.wikipedia.org/wiki/…. Take a look. The confusion is from using invariant population parameters and implying changing sample conditions. This admixture would confuse anyone. $\endgroup$ – Carl Nov 4 '18 at 22:15
  • $\begingroup$ I linked back to this Q/A on the Wikipedia talk page en.wikipedia.org/wiki/…, maybe someone will fix the source material. $\endgroup$ – Carl Nov 4 '18 at 23:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.