Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have come across a problem in textbook to estimate mean. The textbook problem is as follows:

Assume that $N$ data points, $x_1$, $x_2$, . . . , $x_N$ , have been generated by a one-dimensional Gaussian pdf of unknown mean, but of known variance. Derive the ML estimate of the mean.

My question is, Why do we need to estimate mean using MLE when we already know that mean is average of the data? The solution also says that MLE estimate is the average of the data. Do I need to do all the tiring maximizing MLE steps to find out that mean is nothing but average of the data i.e. $(x_1+x_2+\cdots+x_N)/N$ ?

share|improve this question
10  
You may be confused by two distinct meanings of the word "mean." In this question you use it to refer to (a) a parameter of a family of Gaussian distributions and (b) a statistic that can be computed from data. You might want to explore what this site has to say about MLE and parameters. – whuber Feb 9 at 3:15
1  
What about providing the reference for the textbook you quote? – Xi'an Feb 9 at 5:45
up vote 7 down vote accepted

Why do we need to estimate mean using MLE when we already know that mean is average of the data?

The text book problem states that $x_1,x_2,\dots,x_N$ is from $$x\sim\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}}$$ They tell you that $\sigma$ is known, but $\mu$ has to be estimated.

Is it really that obvious that a good estimate $\hat\mu=\bar x$?!

Here, $\bar x=\frac{1}{N}\sum_{i=1}^Nx_i$.

It wasn't obvious to me, and I was quite surprised to see that it is in fact MLE estimate.

Also, consider this: what if $\mu$ was known and $\sigma$ unknown? In this case MLE estimator is $$\hat\sigma^2=\frac{1}{N}\sum_{i=1}^N(x-\bar x)^2$$

Notice, how this estimator is not the same as a sample variance estimator! Don't "we already know" that the sample variance is given by the following equation? $$s^2=\frac{1}{N-1}\sum_{i}(x-\bar x)^2$$

share|improve this answer
    
nitpicky pet peeve: $s^2$ is not the sample variance, $\hat \sigma^2$ is. – Cliff AB Feb 9 at 5:33
1  
@CliffAB I think there's no lack of support for calling $s^2_{N-1}$ "the sample variance". Just as an example, the wikipedia page on Bessel's correction calls it that. Many books do as well. I'd prefer to lean toward your terminology myself but I think it's probably too strong to say $s^2_{N-1}$ is not the sample variance these days -- the terminology is very widespread, probably more widely used than calling $s^2_N$ by that name – Glen_b Feb 9 at 8:14
1  
@Glen_b I was brought up to call $s_N^2$ the "sample variance" (as in "variance of the sample, in its own right") and $s_{N-1}^2$ the "(estimated) population variance" (as in unbiased estimate, since as this post demonstrates, $s_N$ is also a useful estimator). But I took a (non-random) "poll" of textbooks and calculator manuals a few years back and found my usage to be strongly in the minority, though I did find plenty of examples of both. Don't know whether this is a trend. [Also, plain old $s$ and $\hat \sigma$ are irritatingly ambiguous at times... I appreciate the $N$ and $N-1$!] – Silverfish Feb 9 at 12:09
    
@CliffAB, I've seen $s^2$ used a lot in econometrics for a sample variance, and $\sigma^2$ for population parameter, e.g. in Greene "Econometric Analysis". – Aksakal Feb 9 at 14:26
1  
@CliffAB, I didn't make the terminology, but maybe the rationale in econometrics was that there's always more than one estimator of anything, including the variance. So, $\hat\sigma^2$ wouldn't be specific enough, it would appear to refer to any number of estimator one could come up with, while $s^2$ has a specific meaning of the average squared deviations. Now, in OLS context $s^2=\frac{e'e}{N-k}$, where $k$ is the num of parameters. As you see it's not always $N-1$, so even this notation is not absolutely specific, but it's assumed that we adjust for the num of parameters. – Aksakal Feb 9 at 14:34

In this case, the average of your sample happens to also be the maximum likelihood estimator. So doing all the work derive the MLE feels like an unnecessary exercise, as you get back to your intuitive estimate of the mean you would have used in the first place. Well, this wasn't "just by chance"; this was specifically chosen to show that MLE estimators often lead to intuitive estimators.

But what if there was no intuitive estimator? For example, suppose you had a sample of iid gamma random variables and you were interested in estimating the shape and the rate parameters. Perhaps you could try to reason out an estimator from the properties you know about Gamma distributions. But what would be the best way to do it? Using some combination of the estimated mean and variance? Why not use the estimated median instead of the mean? Or the log-mean? These all could be used to create some sort of estimator, but which will be a good one?

As it turns out, MLE theory gives us a great way of succinctly getting an answer to that question: take the values of the parameters that maximize the likelihood of the observed data (which seems pretty intuitive) and use that as your estimate. In fact, we have theory that states that under certain conditions, this will be approximately the best estimator. This is a lot better than trying to figure out a unique estimator for each type of data and then stepping lots of time worrying if it's really the best choice.

In short: while MLE doesn't provide new insight in the case of estimating the mean of normal data, it in general is a very, very useful tool.

share|improve this answer

It is a matter of confusing vocabulary, as illustrated by those quotes, straight from google:

average
noun: average; plural noun: averages

  1. a number expressing the central or typical value in a set of data, in particular the mode, median, or (most commonly) the mean, which is calculated by dividing the sum of the values in the set by their number. "the proportion of over-60s is above the EU average of 19 per cent" synonyms: mean, median, mode, midpoint, centre

Not the best definition, I agree! Especially when suggesting mean as a synonym. I would think average is most appropriate for datasets or samples as in $\bar{x}$ and should not be used for distributions, as $\mu$ in $\mathfrak{N}(\mu,\sigma²)$.

mean

In mathematics, mean has several different definitions depending on the context.

In probability and statistics, mean and expected value are used synonymously to refer to one measure of the central tendency either of a probability distribution or of the random variable characterized by that distribution. In the case of a discrete probability distribution of a random variable X, the mean is equal to the sum over every possible value weighted by the probability of that value; that is, it is computed by taking the product of each possible value x of X and its probability P(x), and then adding all these products together, giving $\mu = \sum x P(x)$.

For a data set, the terms arithmetic mean, mathematical expectation, and sometimes average are used synonymously to refer to a central value of a discrete set of numbers: specifically, the sum of the values divided by the number of values. The arithmetic mean of a set of numbers $x_1, x_2, ..., x_n$ is typically denoted by $\bar{x}$, pronounced "x bar". If the data set were based on a series of observations obtained by sampling from a statistical population, the arithmetic mean is termed the sample mean (denoted $\bar{x}$) to distinguish it from the population mean (denoted $\mu$ or $\mu_x$).

As suggested by this Wikipedia entry, mean applies to both distributions and samples or datasets. The mean of a dataset or sample is also the mean of the empirical distribution associated with this sample. The entry also exemplifies the possibility of a confusion between the terms since it gives average and expectation as synonyms.

expectation noun: expectation; plural noun: expectations

  1. Mathematics: another term for expected value.

I would restrict the use of expectation to an object obtained by an integral, as in $$\mathbb{E}[X]=\int_\mathcal{X} x\text{d}P(x)$$ but the average of a sample is once again the expectation associated with the empirical distribution derived from this sample.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.