4
$\begingroup$

My textbook claims the following:

For a random sample $X_1, X_2, \ldots, X_n$ of normal random variables, one estimator of $\mu$ is $$\overline{X}_n = \dfrac{X_1 + X_2 + \cdots + X_n}{n}$$, where $X_i$ is a random variable and $\mu$ is the mean.

I'm wondering if $\overline{X}_n$ is an estimator of $\mu$ in general or whether it is only for random variables that have a normal distribution?

I would greatly appreciate it if someone could please take the time to clarify this.

$\endgroup$
3
  • $\begingroup$ Look up the law of large numbers (eg. Kolmogorov law of large numbers). $\endgroup$ Jun 10, 2017 at 1:59
  • $\begingroup$ @MatthewGunn Is this the same as the Central Limit Theorem? $\endgroup$ Jun 10, 2017 at 2:04
  • 3
    $\begingroup$ No, it is not. The Central Limit Theorem requires stronger assumptions than the Law of Large Numbers. $\endgroup$ Jun 10, 2017 at 2:06

3 Answers 3

13
$\begingroup$

Estimators are random variables. They exhibit properties that we use to assess their quality, advantages, and disadvantages. So it depends what you mean by "is an estimate of." I can say $\hat{\mu}_0 = 0$ is an estimate of $\mu$, but that doesn't mean it's useful or successful (it uses absolutely no information).

The three most popular properties of statistical estimators in my experience are the following:

  1. Bias
  2. Efficiency
  3. Consistency

The estimator I proposed $\hat{\mu}_0$ is a biased, inconsistent, and inefficient (i.e. it just sucks) estimator of $\mu$. When the $X_i$ are drawn iid from the population distribution and when the expectation of the population distribution exists (eg. Cauchy distribution does not satisfy this latter assumption) and is finite, the arithmetic mean estimator

$$\bar{X} := \frac{1}{n}\sum_i X_i$$

is an unbiased and consistent estimator of $\mu$. Efficiency is a matter of relativity, but Cramér and Rao have shown that when the population distribution is Gaussian or Bernoulli, the arithmetic mean is (asymptotically) the most statistically efficient estimator. (I think there are few more weak regulation conditions needed to establish that result, but that's the main idea.)

All in all, $\bar{X}$ is a great estimator. But when data is corrupted by outliers or when the population distribution has heavy tails (two things that aren't true of Gaussian or Bernoulli distributions), this thing can be somewhat unstable. That's when the tradeoff between the properties come to play: Can we design a new estimator that sacrifices a little bias for a huge advantage in efficiency? What is required of the outliers to ensure that $\bar{X}$ remains consistent and is this a realistic assumption? And so on, and so on.

$\endgroup$
4
  • $\begingroup$ This is a great answer that neatly explains thinking on estimators! (+1) However, the mean $\mu$ existing isn't a sufficient condition for WLLN (for consistency). $\endgroup$
    – user795305
    Jun 11, 2017 at 17:32
  • $\begingroup$ Are you referring to the finiteness of $\mu$ and/or Lesbegue-integrability of the RV? $\endgroup$ Jun 11, 2017 at 17:43
  • $\begingroup$ Lebesgue integrability is sufficient, but not necessary. I'm just saying that there's more to WLLN than the mean existing. For a counter example, see, for instance, math.stackexchange.com/questions/358848/… . See theorem 2.2.7 on page 70 of Durrett for the statement of a necessary and sufficient condition. $\endgroup$
    – user795305
    Jun 11, 2017 at 17:54
  • 1
    $\begingroup$ Thanks Ben, I added that the mean must also be finite. According to Terry Tao's article, that completes the set of assumptions. terrytao.wordpress.com/2008/06/18/… $\endgroup$ Jun 11, 2017 at 17:58
5
$\begingroup$

This is true most of the time.

By the Strong Law of Large Numbers $\frac{X_1 + X_2 + \ldots X_n}{n} \xrightarrow{a.s.} E[X]$

where $E[X]$ is the expected value (mean) of your distribution, and "$\xrightarrow{a.s.}$" denotes almost sure convergence. I'm guessing you aren't aware of almost sure convergence, but there's no harm in thinking about it like a limit from calculus.

There are some technical issues that can the Strong Law of Large Numbers fail to be true. One obvious exception is if you have independent, identically distributed random variables that come from a distribution that does not have an expected value (see Cauchy Distribution).

With that in mind, most random variable that people work with come from standard distributions where the Strong Law does hold. In these cases, the sample mean, $\bar{X_n}$, is a good estimate for $\mu$, because the Strong Law tells us that as the sample size, n, grows $\bar{X_n}$ will get closer and closer to $E[X] = \mu$

$\endgroup$
1
  • $\begingroup$ Maybe the most common exception in statistical practice is inverses $1/X$ (or ratios) where $X$ can be close to zero! $\endgroup$ Oct 12, 2021 at 22:55
4
$\begingroup$

Let $X_1, X_2, \dots$ be a sequence of independent and identically distributed random variables (and such that $\mathbb{E}[|X_i|] < \infty$). Any statistic $f(X_1, X_2, \dots, X_n)$ can be an estimator of the mean. The estimator $\bar X_n$ is an unbiaised and consistent estimator: $$ \mathbb{E} [\bar X_n] = \mathbb{E}[X_1], $$ and by the strong law of large numbers, almost surely $$ \bar X_n \xrightarrow{n \rightarrow \infty} \mathbb{E}[X_1]. $$


Other facts about $\bar X_n$.

You have probabilistic bounds on the error $|\bar X_n - \mathbb{E}[X_1]|$: $$ P(|\bar X_n - \mathbb{E}[X_1]| > \varepsilon) < \frac{Var(X_i)}{n \varepsilon^2}. $$ The bound can be refined for the case $Var(X_i) = \infty$ and it is even possible to characterize the almost sure convergence rate of $\bar X_n$ to $\mathbb{E}[X_i]$ using $\sup \{p \,|\, \mathbb{E}|X_i|^p < \infty\}$ (through refinements of the strong law of large numbers).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.