14
$\begingroup$

I am wondering if maximum likelihood estimation ever used in statistics. We learn the concept of it but I wonder when it is actually used. If we assume the distribution of the data, we find two parameters, one for the mean and one for the variance, but do you actually use it in real situations?

Can somebody tell me a simple case in which it is used for?

$\endgroup$
  • 33
    $\begingroup$ It is $\textit{the}$ most prevalent estimation method. $\endgroup$ – JohnK Mar 14 '17 at 15:06
  • 3
    $\begingroup$ I was trying to ask when are we not using MLE $\endgroup$ – Haitao Du Mar 14 '17 at 16:10
  • 4
    $\begingroup$ logistic regression. Poisson regression. OLS is the MLE when the errors are assumed normal. Gamma regression. Any classical GLM really. Estimating the mean from a normal population. Estimating the success probability from a series of binomial trials. Estimating the poisson event rate. I could go on... $\endgroup$ – GoF_Logistic Mar 14 '17 at 16:56
  • 4
    $\begingroup$ This very narrow search of our site produces hundreds of examples of actual uses. $\endgroup$ – whuber Mar 14 '17 at 16:56
  • 6
    $\begingroup$ @hxd1011 You are not using MLE when you are not using its methods to obtain variances of estimates or to compute confidence, prediction, or tolerance intervals, for instance, even when the estimator you are using might happen to agree with the MLE estimator. For instance, you are not using MLE when you run any t-test. You're often not using it when you are applying unbiased estimators. Philosophically, you wouldn't be using MLE when you care about a loss function or you have a prior distribution. $\endgroup$ – whuber Mar 14 '17 at 21:16
25
$\begingroup$

I am wondering if maximum likelihood estimation ever used in statistics.

Certainly! Actually quite a lot -- but not always.

We learn the concept of it but I wonder when it is actually used.

When people have a parametric distributional model, they quite often choose to use maximum likelihood estimation. When the model is correct, there are a number of handy properties of maximum likelihood estimators.

For one example -- the use of generalized linear models is quite widespread and in that case the parameters describing the mean are estimated by maximum likelihood.

It can happen that some parameters are estimated by maximum likelihood and others are not. For example, consider an overdispersed Poisson GLM -- the dispersion parameter won't be estimated by maximum likelihood, because the MLE is not useful in that case.

If we assume the distribution of the data, we find two parameters

Well, sometimes you might have two, but sometimes you have one parameter, sometimes three or four or more.

one for the mean and one for the variance,

Are you thinking of a particular model perhaps? This is not always the case. Consider estimating the parameter of an exponential distribution or a Poisson distribution, or a binomial distribution. In each of those cases, there's one parameter and the variance is a function of the parameter that describes the mean.

Or consider a generalized gamma distribution, which has three parameters. Or a four-parameter beta distribution, which has (perhaps unsurprisingly) four parameters. Note also that (depending on the particular parameterization) the mean or the variance or both might not be represented by a single parameter but by functions of several of them.

For example, the gamma distribution, for which there are three parameterizations that see fairly common use -- the two most common of which have both the mean and the variance being functions of two parameters.

Typically in a regression model or a GLM, or a survival model (among many other model types), the model may depend on multiple predictors, in which case the distribution associated with each observation under the model may have one of its own parameter (or even several parameters) that are related to many predictor variables ("independent variables").

$\endgroup$
  • 5
    $\begingroup$ "When people have a parametric distributional model." Don't forget non-parametric maximum likelihood estimation, to include empirical likelihood. $\endgroup$ – Mark L. Stone Mar 14 '17 at 15:14
  • 3
    $\begingroup$ @Mark Relatively more rare, though. I'll add a word to my answer. $\endgroup$ – Glen_b -Reinstate Monica Mar 14 '17 at 15:15
  • $\begingroup$ Can we use maximum likelihood estimation even though we assume the distribution, for example, is normal? I think we don't need to but we can still use it, am I right? $\endgroup$ – user122358 Mar 14 '17 at 15:28
  • $\begingroup$ @user122358 Glen and Mark answered you already. You can either assume a distribution or not. Most of the time you assume a distribution and thus likelihood function. $\endgroup$ – SmallChess Mar 14 '17 at 15:33
  • 3
    $\begingroup$ "When people have a parametric distributional model." Don't forget semi-parametric maximum likelihood estimation, to include partial likelihood. ;) $\endgroup$ – Scortchi - Reinstate Monica Mar 14 '17 at 15:44
8
$\begingroup$

While maximize likelihood estimators can look suspicious given the assumptions on the data distribution, Quasi Maximum Likelihood Estimators are often used. The idea is to start by assuming a distribution and solve for the MLE, then remove the explicit distributional assumption and instead look at how your estimator performs under more general conditions. So the Quasi MLE just becomes a smart way of getting an estimator, and the bulk of the work is then deriving the properties of the estimator. Since the distributional assumptions are dropped, the quasi MLE usually doesn't have the nice efficiency properties though.

As a toy example, suppose you have an iid sample $x_1, x_2, ..., x_n$, and you want an estimator for the variance of $X$. You could start by assuming $X \sim N (\mu, \sigma^2)$, write the likelihood using the normal pdf, and solve for the argmax to get $\hat\sigma^2 = n^{-1}\sum (x_i - \bar x)^2$. We can then ask questions like under what conditions is $\hat\sigma^2$ a consistent estimator, is it unbiased (it is not), is it root n consistent, what is it's asypmtotic distribution, etc.

$\endgroup$
  • $\begingroup$ Also, you may check this thread on the intuition behind the Quasi MLE. $\endgroup$ – Richard Hardy Mar 15 '17 at 11:43
5
$\begingroup$

Maximum likelihood estimation is often used in machine learning to train:

Note that in some cases one prefers to add some regularization, which is sometimes equivalent to Maximum a posteriori estimation, e.g. Why is Lasso penalty equivalent to the double exponential (Laplace) prior?.

$\endgroup$
3
$\begingroup$

Can somebody tell me a simple case in which it is used for?

A very typical case is in logistic regression. Logistic regression is a technique used often in machine learning to classify data points. For example, logistic regression can be used to classify whether an email is spam or is not spam or classify whether a person has or does not have a disease.

Specifically, the logistic regression model says that the probability a data point $x_i$ is in class 1 is as follows: $h_\theta(x_i) = P[y_i = 1] = \frac{1}{1+e^{-\theta^T x_i}}$

The parameter vector $\theta$ is typically estimated using MLE.

Specifically, using optimization methods, we find the estimator $\hat\theta$ such that the expression $-\sum_{i=1}^n y_i\log(h_\hat\theta(x_i)) + (1-y_i)\log(1-h_{\hat\theta}(x_i))$ is minimized. This expression is the negative log likelihood, so minimizing this is equivalent to maximizing the likelihood.

$\endgroup$
1
$\begingroup$

We are using MLE all the time, but we may not feel it. I will give two simple examples to show.

Example 1

If we observe coin flip result, with $8$ head out of $10$ flips (assuming iid. from Bernoulli), how to guess the parameter $\theta$ (prob of head) of the coin? We may say $\theta=0.8$, using "counting".

Why use counting? this is actually implicitly using MLE! Where the problem is

$$ \underset \theta {\text{Maximize}}~~~\theta^{8}(1-\theta)^{2} $$

To solve the equation, we will need some calculus, but the conclusion is counting.

Example 2

How would we estimate a Gaussian distribution parameters from data? We use empirical mean as estimated mean and empirical variance as estimated variance, which is also coming from MLE!.

$\endgroup$
  • 6
    $\begingroup$ Example 1 is also a Bayes solution and a method of moments (MM) solution (and probably it's the solution using other procedures as well). Example 2 is the MM solution. It would be much more convincing to exhibit procedures that are exclusively MLE--for otherwise one wouldn't ever need MLE. $\endgroup$ – whuber Mar 14 '17 at 16:55
  • $\begingroup$ Why example 1 becomes a Bayes solution and Example 2 becomes the MM solution? What is MM, by the way? $\endgroup$ – user122358 Mar 14 '17 at 17:06
  • $\begingroup$ @user122358 MM is the method of moments. See here, for instance: en.wikipedia.org/wiki/Method_of_moments_(statistics) $\endgroup$ – jld Mar 14 '17 at 19:55
0
$\begingroup$

Some maximum likelihood uses in wireless communication:

  • Decoding of digital data from noisy received signals, with or without redundant codes.
  • Estimation of time-, phase-, and frequency-offsets in receivers.
  • Estimation of the (parameters of the) propagation channel.
  • Estimation of delay, angle of arrival, and Doppler shift (e.g., radar).
  • Estimation of a mobile position (e.g., GPS).
  • Estimation of clock offsets for synchronization of all kinds of distributed settings.
  • A multitude of calibration procedures.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.