5
$\begingroup$

Suppose we have $T_i,i=1..n$ i.i.d. with unknown distribution and we want to estimate $E[T]$. Note that in this setting we are not estimating E[T] as a parameter of a parameter-dependent family of distributions, therefore it is difficult to attach a meaning to a likelyhood function:

$L[E[T]]=P(T_i|E[T])$

as it would be done when for example estimating $\mu$ from maximum likelyhood, knowing that the underlying distribution is Gaussian.

Again what we would probably do is computing $\overline{T}=\frac{1}{n}\sum_i T_i$. And we would be sure that the estimator would be consistent and its variance would go to zero. For example we have trivially:

$E[\overline{T}]=E[T_i]$

and:

$\sigma^2[\overline{T}]=O(1/n)$

supposing that $T_i$ has finite variance.

Here is the question: can we prove that $\overline{T}$ is optimal in some way? To me conceptes from MLE estimators or sufficient statistics are a bit difficult to apply, since $E[T]$ is not a parameter of the distribution but maybe I am missing something? Can we "derive" the sample mean estimator to be optimal according to some criterion in the general case and without making assumptions on the underlying distributions ?

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Hi: This concept-issue is discussed in any good math stat class or book so I can't answer here but the article at the link below might help. Essentially, the MLE has attractive properties regardless of the underlying distribution. ocw.mit.edu/courses/mathematics/… $\endgroup$
    – mlofton
    Commented Apr 12, 2019 at 18:01
  • $\begingroup$ Thx a lot for the link I will read it carefully. But are you sure that it addresses my question? It seems to start from the assumption that we are trying to estimate a parameter from a distribution, like the first example I proposed for the mean of a gaussian distribution. But my question is about estimating the expectation value of the distribution, i.e. $E[X]$ without assumptions on the underlying distribution. How are the two things connected? To me they are a bit different? $\endgroup$
    – Thomas
    Commented Apr 12, 2019 at 18:35
  • $\begingroup$ Hi: I don't think it's possible to do what you're asking without an assumption about the underlying distribution. the mean is only "optimal" when the density has a certain form but maybe someone else can understand your question more deeply and have some insight. I also don't understand the line $L(E(T) = P(T_{i} | E(T))$ so maybe I'm totally not following. $\endgroup$
    – mlofton
    Commented Apr 13, 2019 at 15:02
  • $\begingroup$ Note that expectation of the distribution still involvsd the density so I'm pretty sure that the answer is no. $\endgroup$
    – mlofton
    Commented Apr 13, 2019 at 15:03
  • $\begingroup$ Thanks for your answer.The line indicates what, when estimating the mean $\mu$ from a gaussian distribution, would be called the likeleyhood function $L(\mu)$ en.wikipedia.org/wiki/Likelihood_function , i.e. the probability of the data observed, given the value of $\mu$. Yes the point is that these theories do not apply straightforwardly and I was looking for some other "theoretical" derivation of the sample mean estimator in the general case. $\endgroup$
    – Thomas
    Commented Apr 13, 2019 at 15:07

2 Answers 2

Reset to default
6
$\begingroup$

Though the sample mean is an unbiased estimator of the unknown population mean, it cannot be optimal in general. Take the case of the log-normal distribution. The maximum likelihood estimator of the mean on the original scale is a function of the sample mean and sample variance both computed on the log scale. This prevents outliers from ruining either the mean or SD. There is a relationship of this problem with that fact that an accurate nonparametric confidence interval for the population mean does not exist. When one wants to have a measure that 'works' on all continuous distributions, one has to use an estimator that aligns with nonparametrics, such as the sample median. BY doing so one pays a high efficiency price if the data are Gaussian, since the sample median in that case is an inefficient estimator of the population mean or median.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks a lot for your answer! You provided a nice example where the MLE of the mean is not equal to the sample mean. I have some questions on points that I do not understand. (1) you say that an accurate nonparametric estimation of the confidence interval for the mean does not exist. Why one cannot use the variance of the sample mean estimator that I showed in my question, estimating the population variance by the sample variance? (2) What is an estimator that "aligns" with nonparametric? $\endgroup$
    – Thomas
    Commented Apr 15, 2019 at 14:21
  • $\begingroup$ Overall, did I understand well that one cannot "derive" from "first principles", i.e. in a nonparametric setting and from some optimality procedure, the sample mean estimator and should regard it as simply "being given"? $\endgroup$
    – Thomas
    Commented Apr 15, 2019 at 14:40
  • $\begingroup$ @Frank Harrell can say more ( if he pleases. it's always appreciated ) but note that the sample mean is not the only measure of centrality. There's the median and various other robust measures of the median ( or mean ) that can have better-more attractive properties than the sample mean or the MLE. There is much discussion of this concept in the robust-statistics field. Again, it's not my field but terms like efficiency and breakdown point are important there. Point is that the mean-MLE are not necessarily the best or special. It depends on the criteria one uses. $\endgroup$
    – mlofton
    Commented Apr 15, 2019 at 16:00
  • $\begingroup$ I re-read what Frank and note that he is making a "deeper" point than I'm trying to make. He's saying that, for certain distribuitions, the standard parameteric approach for the CI is suboptimal even for squared loss ( I think ). I'm saying that, there are other criteria, besides squared loss where the standard parameteric approach for CI does not "win". That's well known already but I'm saying it anyway just to not confuse with what Frank is saying. (which is trickier ). $\endgroup$
    – mlofton
    Commented Apr 15, 2019 at 16:07
  • $\begingroup$ Mmm... Ok but for the moment I would like to focus on the population mean,i.e. E[X] as a measure of centrality. As you said, MLE are not necessarily the best and this depends on the criterion. MLE are optimal according to the maximum likelyhood criterion by construction. I would like to know if there is a criterion that makes the sample mean optimal by construction in a non-parametric framework, where there is little information to play with. $\endgroup$
    – Thomas
    Commented Apr 15, 2019 at 16:22
0
$\begingroup$

What you stated in your original post is that the sample mean converges to the true mean in probability (in fact, it does with probability one) if expectations and variances are finite, i.e., the Law of Large Numbers. Together with unbiasedness, I think this makes the sample mean "optimal enough". No distributional assumptions are needed except finite first and second moments. Notice that this works equally well to estimate probabilities of events, since $P(E) = E[I_E]$. Yes, if you want to claim that the sample mean is the MLE then you need distributional assumptions.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.