Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
My textbook claims the following:
For a random sample $X_1, X_2, \ldots, X_n$ of normal random variables, one estimator of $\mu$ is $$\overline{X}_n = \dfrac{X_1 + X_2 + \cdots + X_n}{n}$$, where $X_i$ is a random variable and $\mu$ is the mean.
I'm wondering if $\overline{X}_n$ is an estimator of $\mu$ in general or whether it is only for random variables that have a normal distribution?
I would greatly appreciate it if someone could please take the time to clarify this.
Estimators are random variables. They exhibit properties that we use to assess their quality, advantages, and disadvantages. So it depends what you mean by "is an estimate of." I can say $\hat{\mu}_0 = 0$ is an estimate of $\mu$, but that doesn't mean it's useful or successful (it uses absolutely no information).
The three most popular properties of statistical estimators in my experience are the following:
The estimator I proposed $\hat{\mu}_0$ is a biased, inconsistent, and inefficient (i.e. it just sucks) estimator of $\mu$. When the $X_i$ are drawn iid from the population distribution and when the expectation of the population distribution exists (eg. Cauchy distribution does not satisfy this latter assumption) and is finite, the arithmetic mean estimator
$$\bar{X} := \frac{1}{n}\sum_i X_i$$
is an unbiased and consistent estimator of $\mu$. Efficiency is a matter of relativity, but Cramér and Rao have shown that when the population distribution is Gaussian or Bernoulli, the arithmetic mean is (asymptotically) the most statistically efficient estimator. (I think there are few more weak regulation conditions needed to establish that result, but that's the main idea.)
All in all, $\bar{X}$ is a great estimator. But when data is corrupted by outliers or when the population distribution has heavy tails (two things that aren't true of Gaussian or Bernoulli distributions), this thing can be somewhat unstable. That's when the tradeoff between the properties come to play: Can we design a new estimator that sacrifices a little bias for a huge advantage in efficiency? What is required of the outliers to ensure that $\bar{X}$ remains consistent and is this a realistic assumption? And so on, and so on.
This is true most of the time.
By the Strong Law of Large Numbers $\frac{X_1 + X_2 + \ldots X_n}{n} \xrightarrow{a.s.} E[X]$
where $E[X]$ is the expected value (mean) of your distribution, and "$\xrightarrow{a.s.}$" denotes almost sure convergence. I'm guessing you aren't aware of almost sure convergence, but there's no harm in thinking about it like a limit from calculus.
There are some technical issues that can the Strong Law of Large Numbers fail to be true. One obvious exception is if you have independent, identically distributed random variables that come from a distribution that does not have an expected value (see Cauchy Distribution).
With that in mind, most random variable that people work with come from standard distributions where the Strong Law does hold. In these cases, the sample mean, $\bar{X_n}$, is a good estimate for $\mu$, because the Strong Law tells us that as the sample size, n, grows $\bar{X_n}$ will get closer and closer to $E[X] = \mu$
Let $X_1, X_2, \dots$ be a sequence of independent and identically distributed random variables (and such that $\mathbb{E}[|X_i|] < \infty$). Any statistic $f(X_1, X_2, \dots, X_n)$ can be an estimator of the mean. The estimator $\bar X_n$ is an unbiaised and consistent estimator: $$ \mathbb{E} [\bar X_n] = \mathbb{E}[X_1], $$ and by the strong law of large numbers, almost surely $$ \bar X_n \xrightarrow{n \rightarrow \infty} \mathbb{E}[X_1]. $$
Other facts about $\bar X_n$.
You have probabilistic bounds on the error $|\bar X_n - \mathbb{E}[X_1]|$: $$ P(|\bar X_n - \mathbb{E}[X_1]| > \varepsilon) < \frac{Var(X_i)}{n \varepsilon^2}. $$ The bound can be refined for the case $Var(X_i) = \infty$ and it is even possible to characterize the almost sure convergence rate of $\bar X_n$ to $\mathbb{E}[X_i]$ using $\sup \{p \,|\, \mathbb{E}|X_i|^p < \infty\}$ (through refinements of the strong law of large numbers).
