Parameter estimation problem: maximum likelihood [duplicate]

Question

Suppose I have some observations $x_{1}, x_{2}, \dots, x_{n}$. I also have a probability density function with one unknown parameter $\theta$. I would like to find such $\theta$, which would give the highest probability of observing the data points I have. How do I solve such problem?

Answer 1

This is known as a maximum likelihood problem. For simplicity let's say that the observations $x_i$ are independent and the probability density function is $g(x\,|\,\theta)$. Then the likelihood function, the probability of observing the data points you have, has the following formula:

$$f(\theta)=P(x_1,x_2,\ldots,x_n\,|\,\theta) = \prod_{i=1}^n g(x_i\,|\,\theta) = g(x_1\,|\,\theta)g(x_2\,|\,\theta)\cdots g(x_n\,|\,\theta) $$

To solve your problem, you just need to find the value of $\theta$ that will get you the highest value of $f(\theta)$. Since you only have one parameter $\theta$, you could just graph $f(\theta)$ over some interval where you think the maximum is, and find the highest point on the graph. Of course you run the risk that the true global maximum was not in the interval you chose, but in most statistics problems $f(\theta)$ has a nice, upside-down bowl shape with a single well-defined global maximum that is easy to find.

For a more automated answer that scales to harder problems, you can use a numerical optimization algorithm. When performing the optimization, people typically prefer to solve an equivalent problem, which is to find the $\theta$ that minimizes $- \log f(\theta)$:

$$- \log f(\theta) = - \sum_{i=1}^n \log g(x_i\,|\,\theta)$$

This can be done using many methods, including gradient descent, Newton's method, and golden section search.

You might wonder why people prefer to work with $- \log f(\theta)$ instead of $f(\theta)$ itself. There are two reasons. First, taking the log is convenient because it turns the product into a sum; this makes calculating derivatives simpler, and most optimization algorithms like to use derivatives of the function to solve the optimization faster. Second, taking the negative is just a trick to make the problem fit the standard mold for optimization. To keep things concise, almost all textbooks and algorithms for optimization focus on the task of minimizing a function.