3
$\begingroup$

I want to find the maximum likelihood estimator of the "rate parameter theta of the Exponential Distribution".

So i followed the following commands in R:

 x=rexp(500,rate=2)
 f <- function(x,theta){
        sum(-dexp(x,rate=theta,log=T))
 }
 optimize(f=f,x=x,interval=c(0,5))

I do not understand the logic - why do we consider maximum=FALSE(default) in the optimize function?

Why haven't we used maximum=TRUE to find the MLE?

$\endgroup$
  • $\begingroup$ I must be missing your point, but the maximum likelihood estimator of the rate parameter of an exponential is the reciprocal of the mean. Invoking an optimizer should get you the right answer but is like going from Paris to Versailles via Vladivostok. $\endgroup$ – Nick Cox Nov 30 '13 at 1:16
  • $\begingroup$ @NickCox Would you please tell me the proper way to find MLE? $\endgroup$ – ABC Nov 30 '13 at 1:23
  • $\begingroup$ Maximize the likelihood, naturally. Not trying to outsmart you, but what answer are you seeking? The particular method depends on whether there is a closed form solution that gets you there in one (unusual, but true in this case) or you have to estimate it numerically. $\endgroup$ – Nick Cox Nov 30 '13 at 1:25
  • 1
    $\begingroup$ I agree with @NickCox - the only purpose I can see to this exercise would be if it were part of an introduction to finding MLEs numerically by beginning with an example you can also easily do by hand. Which implies you're doing this for some subject. Is that the case? $\endgroup$ – Glen_b Nov 30 '13 at 1:26
  • $\begingroup$ @Glen_b yes. I want to find MLE by using R. It is one of my course. $\endgroup$ – ABC Nov 30 '13 at 1:28
5
$\begingroup$

The term inside your definition of f :- sum(-dexp(x,rate=theta,log=T)) is NOT the likelihood, but something else.

What is it that is being calculated?

When you consider what it is that is being optimized there, you will also understand why you're minimizing that function in order to maximize the likelihood.

To quote your own algebra, here's the likelihood:

$\cal{L}(\theta)=\prod_{i=1}^{n}\theta e^{-\theta x_i}=\theta^n e^{-\theta \sum_{i=1}^{n}x_i}$

dexp with log=TRUE doesn't return the density. Here's what the help says: log, log.p logical; if TRUE, probabilities p are given as log(p). ... that is when you say log=TRUE you get the log of the density.

The likelihood at $\theta$ will be the product of the densities, taken at each data point.

The log-likelihood is the sum of the log-densities, over the data points, evaluated at a given $\theta$.

That is, sum(dexp(x,rate=theta,log=T)) would be the log-likelihood function. We'd want to maximize that.

But we have sum(-dexp(x,rate=theta,log=T)) (don't ask me why they didn't write the obviously equivalent but presumably faster -sum(dexp(x,rate=theta,log=T))).

That is, the program is minimizing the negative log-likelihood, which is equivalent to maximizing the log-likelihood. Here's the result on calling f on theta values between 1 and 3:

enter image description here

By contrast, this is what the likelihood function looks like:

enter image description here

sum(dexp(x,rate=theta,log=T)) is calculating $θ^ne^{−θ∑^n_{i=1}x_i}$?

It's calculating the log of that quantity.

But here I see I have the minus sign in every program related to MLE in my lecture sheet.

Minimizing rather than maximizing is a convention. There's no particular need for it.

The R documentation is saying that in optim function par Initial values for the parameters to be optimized over. How can I select the initial values?

Would you please tell me how can I relate this program
fexp = function(theta, x){ prod(dexp(x,rate=(1/theta))) }
res3<-optimize(f=fexp,interval=c(0,50), maximum=T, x=x)
res3

with my above program that I have posted in the question?

Why here is the prod function being called?

Because the likelihood is a product.

And why here we have mentioned maximum=T?

Because it's computing the likelihood, which we want to maximize.

Edit: I notice another issue with the above code: it says rate = 1/theta. That implies that the theta there is not the rate parameter of your earlier mathematics and code, but is in fact a scale parameter. Watch out for that! Another thing to watch out for is that likelihood calculations often have underflow problems (and sometimes, overflow problems).

$\endgroup$
  • $\begingroup$ Actually i have missed the class for political unrest in our country. So it would be very kind if you explain me what the above commands are trying to proof ? $\endgroup$ – ABC Nov 30 '13 at 1:26
  • 1
    $\begingroup$ This comment: "i have missed the class for political unrest in our country" is such a non sequitur as to require explanation. Are you actually saying that you're learning to compute MLEs in R during a class on political unrest? If you don't understand what your code is doing, it's hardly surprising that it makes no sense to you. Do you know what the likelihood function for the exponential rate parameter is? Could you write it down? $\endgroup$ – Glen_b Nov 30 '13 at 1:30
  • 2
    $\begingroup$ @Glen_b My second reading was that Harry missed a class because of political unrest in his country. $\endgroup$ – Nick Cox Nov 30 '13 at 1:32
  • 1
    $\begingroup$ Sorry, but I don't understand your question there. Take another look at my updated answer. I will say your replies come so quickly that it suggests you're not spending enough time thinking about the replies you are getting. Stop. Slow down. Investigate. Contemplate. $\endgroup$ – Glen_b Nov 30 '13 at 1:59
  • 1
    $\begingroup$ Oh, it's now clear . prod(dexp(x,rate=theta)) is calculating $\prod_{i=1}^{n}\theta e^{-\theta x_i}$ . You are really a nice teacher , also a wise person i have ever seen before. Thank you very very very much. $\endgroup$ – ABC Nov 30 '13 at 3:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.