# exponential density & bernoulli distribution

I had asked a question on Maximum Likelihood earlier. Now I have two questions that are related to this question which had:

Let $x$ have an exponential density $p(x|\theta) = \theta e^{-\theta x} \text{ if }x \ge 0;\quad 0 \text{ otherwise.}$

Now suppose that n samples $x_1, ... , x_n$ are drawn independently according to $p(x|\theta)$ Show that the maximum-likelihood estimate for $\theta$ is given by

$$\hat{\theta} = \frac{1}{\frac{1}{n} \,\, \sum^n_{k = 1} x_k}$$

The second question is similar

Let $x$ be a $d$-dimensional binary (0 or 1) vector with a multivariate Bernoulli distribution

$$p(x|\theta) = \prod^d_{i=1} \theta_i^{x_i}(1-\theta _i)^{1-x_i}$$

where $\theta = ( \theta_i, ..., \theta_d)^t$ is an unknown parameter vector, $\theta_i$ being the probability that $x_i = 1$.

Show that the maximum-likelihood estimate for $\theta$ is

$$\hat{\theta} = \frac{1}{n} \sum^n_{k = 1} x_k$$

how would I show that the maximum-likelihood estimate is given for either of them. I've been reading tutorials and stuff and I still don't quite understand it.

• Hi Jonas, I have edited your mathematics a little. Please check that it still says what you want. If you want $\hat{\theta}$ instead of $\theta_h$, you get that by typing $\hat{\theta}$. Mar 25, 2013 at 6:21
• thanks I think there are a few changes that I need to make but it is mostly correct Mar 25, 2013 at 7:17

Considered as a function of its parameters $\theta$ for fixed data $x$, the likelihood evaluates the probability that that you would observe the data at some values for $\theta$ under the assumption that they are drawn from this particular model.

The likelihood function $L(\theta|x)$ represents the joint probability of observing all of these data under the model $f(x|\theta)$. When you find a value of $\theta$ for which this joint probability is the global maximum, we reason that this is the parameter value that most likely gave rise to the data. For some simple models, we can define the MLE estimate in closed form using calculus.

We start by writing the likelihood function $L(\theta|x)=f(\theta|x)$. For the exponential case, $L(\theta|x)=\prod_{i=1}^n \theta\exp(-\theta x_i)$. However, the product operator makes evaluation very messy and difficult. Most people deal with the log-likelihood because it transforms the product into a sum and because the logarithm attains its maximum at the same place: $\ln(L(\theta|x))=\sum_{i=1}^n (\ln(\theta)-\theta x_i).$ Now comes the fun part: taking the derivative of the log-likelihood with respect to $\theta$. This helpful because when the derivative is zero, we know we have found either a maximum or a minimum. $\frac{d \ln(L)}{d \theta}=\sum_{i=1}^n (\frac {1}{\theta}- x_i)$. Manipulating the sigma operator, we find $\frac{d \ln(L)}{d \theta}=\frac{n}{\theta}+ \sum_{i=1}^n x_i$. Setting the derivative equal to zero and solving for $\theta$: $0=\frac {n}{\theta}-\sum_{i=1}^n x_i$, which can be manipulated to show $\frac{1}{\hat \theta}=\frac {1}{n}\sum_{i=1}^nx_i$.

To confirm that this is a maximum, we can check the second derivatives using standard results from calculus, or look at a plot of various values of $(\theta, \ln(L))$ in the neighborhood of $\hat \theta$. For this problem, I believe there is only one solution for $\hat \theta$, so we don't have to worry about proving that this is the global maximum.

I hope this helps. The same procedures should help you with the Bernoulli problem, but I am less familiar with that process so I would not want to speak out of turn. This is only a cursory treatment of the reasoning process of MLE. I would highly recommend Gary King's book Unifying Political Methodology, which contains a very thorough, very accessible explanation of the MLE procedure.

@Jonas, is there anything that could be made more clear in my answer? I notice you marked it as "correct" and then reversed that.

• Is it really true that "The likelihood function $L(\theta\mid x)$ represents the joint probability of observing all of these data under the model $f(\theta\mid x)$." I thought the model was $f(x\mid \theta)$. Mar 25, 2013 at 14:43
• I aspire to one day write an answer that Dilip approves of. (:
– Sycorax
Mar 25, 2013 at 14:48
• When you are writing an answer intended to help beginners, you should be particularly careful in your exposition, and in this case, the difference between $f(x\mid\theta)$ and $f(\theta\mid x)$ has been the subject of so much controversy among statisticians that your error was especially egregious. Incidentally, I don't think that it is all that "messy and difficult" to find the maximum value of $\theta^n\exp(-y\theta)$ (where $y$ is the constant $\sum_i x_i$) for $\theta \in (0,\infty)$ without resorting to logarithms. Mar 25, 2013 at 19:48
• Yep. I realized my mistake immediately after hitting "comment." And I wasn't even lucky enough to delete it before you replied. All around, not my best day.
– Sycorax
Mar 25, 2013 at 20:08