# MLE estimation: sensitive to data?

I am trying to estimate a MLE for an exponential distribution using fmincon in Maltab. I am having problem to estimate my parameter. For instance, I simulate an exponential distribution with a chosen parameter and then use the simulated data in my MLE. I therefore hope to find the same parameter that I generated my simulated data with. The problem that I have is that I can't get fmincon to evaluate the correct parameter...it always give me my upper bound (bound set prior to estimation) of my parameter. Why? I was told that a MLE can be very sensitive to the data...the structure of the data.

Any insights for me?

-
 Why are you using fmincon - a numerical minimization procedure - in the first place? The MLE for the exponential distribution can be derived analytically (it is the sample mean mean). That being said, this question would be much easier to answer if you gave us the full code for the procedure that you're using (i.e. show us how you use fmincon). – MånsT Aug 19 '12 at 13:01 My bet is that you are having numerical issues because you are working on the "raw" scale - for numerical ML you should always work on the log scale. Maximise the log density, not the density itself. Even though this has an analytic solution, you should still get close using numerical optimisation - else there is something suspect with the numerical optimisation (either the matlab routine or the way you've applied it). – probabilityislogic Aug 20 '12 at 0:39

The density of the exponential model is

$$f(x) = \frac{1}{\theta} e^{-x/\theta}.$$

The log-likelihood $\ell$ of a sample of size $n$ is thus

$$\ell(\theta) = - n \log(\theta) -\frac{1}{\theta} \sum_{i=1}^n x_i.$$

The maximum likelihood estimate of the parameter $\theta$, say $\hat{\theta}$ is such that the derivative is 0. In other words

$$\frac{d\ell}{d\theta} \big\vert _{\hat{\theta}} = 0 = - \frac{n}{\hat{\theta}} + \frac{1}{\hat{\theta}^2}\sum_{i=1}^n x_i,$$

which solves out to $\hat{\theta} = \bar{x}$. In the case of the exponential distribution, the MLE is the sample mean. With this preamble you can:

1. Test whether the fmincon function works as you expect.
2. See that, just like the sample mean, MLEs cannot be exact estimators.
3. See that MLE is not intrinsically different from other estimators. Whatever you mean by "sensitive to the data", it is probably not a property of MLEs, but more of the particular model you try to fit.
-
There is a small slip-up in the algebra here. You've parametrized the exponential via a rate parameter, so the MLE is $1/\bar x$, which is consistent with your equations. – cardinal Aug 19 '12 at 17:52
@cardinal thanks (+1), I'll fix that. – gui11aume Aug 19 '12 at 22:02
just to add to point 3 - mle can be sensitive but only when the sample size is not much larger than the number of parameters – probabilityislogic Aug 20 '12 at 1:02

If you expect the estimate from simulated data to be equal to the value of the parameter form the simulated distribution you are naive. That will never happen. This is not a question of sensitivity to data. It is the natural variability of estimates. The estimate have a distribution with a mean and a variance. So although I don't know the specifics of the program you are using, it would not be unusual to see the program provide a confidence interval along with the estimate to provide an idea of the uncertainty in the estimate.

Now maximum likelihood is based on a parametric form of the distribution ( the negative exponential in your case). So if the data appear to come from a distribution that is not exponential the estimate could be sensitive to the characteristics of the data set. As an illustrative example, the sample mean is the mle for the population mean of a normal distribution. But if the data contains a very large positive or negative observation the result will be very sensitive to that outlier.

-
Just wanted to say that "...you are very naive" sounds unnecessarily harsh, at least to my ears. It just doesn't seem in line with the friendly spirit that we see in most answers here at CV... – MånsT Aug 19 '12 at 13:06
I am not trying to be unfriendly. I just don't see a better way to say it. I am not even sure that the OP meant this the way I assume. But if you think that a data set will return a parameter value equal to the one used to generate the data than you are naive about statistics. It is not meant to be insulting. I am just telling it like it is. – Michael Chernick Aug 19 '12 at 13:31
I agree with @MånsT's comment - as you know, many of the questioners on this site are not very experienced with statistics and the concept of the difference between a parameter value and an estimator (and the idea that an estimator is a random variable with a distribution) can be a subtle issue for a beginner. Also, the quote from the OP "The problem that I have is that I can't get fmincon to evaluate the correct parameter...it always give me my upper bound (bound set prior to estimation) of my parameter." makes me think this may actually be a problem using the software. – Macro Aug 19 '12 at 13:35
Okay guys. I toned it down from very naive to naive. I understand that we have users asking questions who are beginners that are trying to learn how to use statistics. I have been participating on the site for almost 4 months. I am not trying to insult anyone. But if we teach we should be firm. I am reading into the OPs remark. I may not be interpeting it right. – Michael Chernick Aug 19 '12 at 13:48
All I said was that IF you think that every time you take a sample from an exponential distribution with a parameter having value lambda that the estimate from the sample will be the same value, lambda,THEN you are naive about statistics. We can be polite and respectful but that does not mean that we should water down what we say. – Michael Chernick Aug 19 '12 at 13:50