1
$\begingroup$

I have samples $\{x_1, x_2, x_3, \dots , x_n\}$ of a random variable $X$. I compute Maximum Likelihood Estimator $\hat{\theta}_n$ using the sample data.

Now, if I collect one more sample $x_{n+1}$ and compute the MLE $\hat{\theta}_{n+1}$ again with $n+1$ samples. Would it be possible to prove that $\hat{\theta}_{n+1}$ is a better estimate of true parameter than $\hat{\theta}_{n}$?

Please note that I am only adding one more sample $(x_{n+1})$ to the existing sample set $\{x_1, x_2, \dots , x_n\}$. I am not sampling n+1 data points again.

$\endgroup$
  • $\begingroup$ No, not for all possible realisations of the $n+1$ sample. $\endgroup$ – Xi'an Feb 20 '19 at 19:26
  • $\begingroup$ @Xi'an Can you please elaborate on that? I am not sampling (n+1) data points again. I am only sampling one more data point and adding to the set. Are there any conditions on the (n+1)th sample to make sure that the MLE gets improved. $\endgroup$ – neo89 Feb 20 '19 at 20:16
  • 1
    $\begingroup$ With 'better estimate' you mean lower expected mean squared error? $\endgroup$ – Sextus Empiricus Feb 20 '19 at 20:18
  • 1
    $\begingroup$ Do you mean whether $\hat\theta_{n+1}$ is better than $\hat\theta_{n}$, unconditional on $\hat\theta_{n}$? When $\hat\theta_{n}$ happens to be close to, or equal to $\theta$, then $\hat\theta_{n+1}$ won't necessarily be better. So it is not true for every specific case/realization. $\endgroup$ – Sextus Empiricus Feb 20 '19 at 20:21
  • 1
    $\begingroup$ While Xi'ans example is for an unbiased estimator, the same works for the MLE. There will always be a non-zero number of possible results for $x_{n+1}$ that will lead to a worse estimator, or at least not better (the uniform distribution $\mathcal{U}(0,\theta)$ is actually an, not so common, interesting case where an extra sample point can't make the MLE worse). An example is the sample mean for a sample from a normal distributed population, which can get further away from the population mean when you increase the sample size. $\endgroup$ – Sextus Empiricus Feb 20 '19 at 21:17
1
$\begingroup$

If you add an extra point to the data then the estimate may get worse, or at least is not neccesarily becoming better.

There will always be a non-empty set of possible results for $x_{n+1}$ that will lead to a worse estimator, or at least not better.

  • An example where an extra point can't make the MLE worse (but there are still points that won't make it better) is the uniform distribution $\mathcal{U}(0,\theta)$ where an extra sample point can't make the MLE worse (since the MLE is the sample maximum which won't decrease upon adding an extra point). However, as Xi'an showed in his comment, the the least variance unbiased estimator for this case can get worse when you add an extra point to the sample.

  • An example where the MLE can get worse is the sample mean which is the MLE in the case of estimating the mean for a normal distributed population. The sample mean can get further away from the population mean, get worse, when you increase the sample size. 

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.