Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Could anyone explain to me in detail about maximum likelihood estimation (MLE) in layman's terms? I would like to know the underlying concept before going into mathematical derivation or equation.

share|improve this question
7  
It's unclear what kind of answer you're after. Do you know what likelihood is, for example? If not, better to find out that first. –  Glen_b Aug 19 '14 at 12:56
2  
In addition, I think any answer that doesn't involve math at some level will be inadequate. –  gregmacfarlane Aug 19 '14 at 13:17

11 Answers 11

up vote 13 down vote accepted

Say you have some data. Say you're willing to assume that the data comes from some distribution -- perhaps Gaussian. There are an infinite number of different Gaussians that the data could have come from: different means, different variances. MLE will pick the Gaussian that is the most likely one that your data came from.

So say you've got a data set of y = -1,3, and 7. The most likely Gaussian from which that data could have come has a mean of 3 and a variance of 16. It could have been sampled from some other Gaussian. But one with a mean of 3 and variance of 16 is more likely to have generated your data than any other.

Moving to regression: instead of the mean being a constant, the mean is a linear function of the data, as specified by the regression equation. So, say you've got data like x = 2,4,10 along with y from before. The mean of that Gaussian is now the fitted regression model $X'\hat\beta$, where $\hat\beta =[-1.9,.9]$

Moving to GLMs: replace Gaussian with some other distribution (from the exponential family). The mean is now a linear function of the data, as specified by the regression equation, transformed by the link function. So, it's $g(X'\beta)$, where $g(x) = e^x/(1+e^x)$ for logit (with binomial data).

share|improve this answer
10  
"MLE will pick the Gaussian that is the most likely one, given your data." Hmmm, isn't it actually: MLE will pick the Gaussian under which your data are most likely? Which is slightly different from picking the "most likely Gaussian"... wouldn't picking the most likely Gaussian require a consideration of prior beliefs? –  Jake Westfall Aug 20 '14 at 1:01
2  
You'd need to clearly specify what you mean by "the most likely one" or this is likely to be misunderstood. –  Glen_b Aug 20 '14 at 2:33
    
Sure, my answer is def incomplete and hand-waivey. The intention is to provide intuition before op goes to learn it formally. –  generic_user Aug 20 '14 at 4:11
6  
@ACD I don't think this is just incomplete but provides the correct intuition. For example, I don't see any problem with not discussing special cases like the likelihood function have more than one maximum. But the difference between the distribution most likely to produce the observed data and the most likely distribution given the data is the very fundamental difference between frequentist and bayesian inference. So if you explain it like that, you are just creating a stumbling block for the future. –  Erik Aug 25 '14 at 11:30
3  
Sure okay, but is the more correct conceptual explanation any harder to understand than the one you've written? I don't think so. I think most of your answer is just fine, but I would just urge you, for the sake of posterity, to just slightly edit some of the phrasing to avoid discussing "the most likely Gaussian" and instead point out that the thing that we want to be "likely" (in colloquial terms) under ML is not the hypothesis but the data. I think this can be a minor but important edit to your otherwise nice answer. –  Jake Westfall Aug 25 '14 at 16:16

Maximum Likelihood Estimation (MLE) is a technique to find the most likely function that explains observed data. I think math is necessary, but don't let it scare you!

Let's say that we have a set of points in the $x,y$ plane, and we want to know the function parameters $\beta$ and $\sigma$ that most likely fit the data (in this case we know the function because I specified it to create this example, but bear with me).

data   <- data.frame(x = runif(200, 1, 10))
data$y <- 0 + beta*data$x + rnorm(200, 0, sigma2)
plot(data$x, data$y)

data points

In order to do a MLE, we need to make assumptions about the form of the function. In a linear model, we assume that the points follow a normal (Gaussian) probability distribution, with mean $x\beta$ and standard deviation $\sigma^2$:
$y = \mathcal{N}(x\beta, \sigma^2)$. The equation of this probability density function is: $$\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left(-\frac{(y_i-x_i\beta)^2}{2\sigma^2}\right)}$$

What we want to find is the parameters $\beta$ and $\sigma$ that maximize this probability for all points $(x_i, y_i)$. This is the "likelihood" function, $\mathcal{L}$

$$\mathcal{L} = \prod_{i=1}^n y_i = \prod_{i=1}^n \dfrac{1}{\sqrt{2\pi\sigma^2}} \exp\Big({-\dfrac{(y_i - x_i\beta)^2}{2\sigma^2}}\Big)$$ For various reasons, it's easier to use the log of the likelihood function: $$\log(\mathcal{L}) = \sum_{i = 1}^n-\frac{n}{2}\log(2\pi) -\frac{n}{2}\log(\sigma^2) - \frac{1}{2\sigma^2}(y_i - x_i\beta)^2$$

We can code this as a function in R with $\theta = (\beta,\sigma^2)$.

linear.lik <- function(theta, y, X){
  n      <- nrow(X)
  k      <- ncol(X)
  beta   <- theta[1:k]
  sigma2 <- theta[k+1]
  e      <- y - X%*%beta
  logl   <- -.5*n*log(2*pi)-.5*n*log(sigma2) - ( (t(e) %*% e)/ (2*sigma2) )
  return(-logl)
}

This function, at different values of $\beta$ and $\sigma^2$, creates a surface.

surface <- list()
k <- 0
for(beta in seq(0, 5, 0.1)){
  for(sigma2 in seq(0.1, 5, 0.1)){
    k <- k + 1
    logL <- linear.lik(theta = c(0, beta, sigma2), y = data$y, X = cbind(1, data$x))
    surface[[k]] <- data.frame(beta = beta, sigma2 = sigma2, logL = -logL)
  }
}
surface <- do.call(rbind, surface)
library(lattice)
wireframe(logL ~ beta*sigma2, surface, shade = TRUE)

likelihood surface

As you can see, there is a maximum point somewhere on this surface. We can find parameters that specify this point with R's built-in optimization commands. This comes reasonably close to uncovering the true parameters $0, \beta = 2.7, \sigma^2 = 1.3$

linear.MLE <- nlm(f=linear.lik, p=c(1,1,1), hessian=TRUE, y=data$y, X=cbind(1, data$x))
linear.MLE$estimate

## [1] -0.04481  2.70841  1.71870

Ordinary least squares is the maximum likelihood for a linear model, so it makes sense that lm would give us the same answers. (Note that $\sigma^2$ is used in determining the standard errors).

summary(lm(y ~ x, data))

## 
## Call:
## lm(formula = y ~ x, data = data)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -3.106 -0.893  0.032  0.880  3.281 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  -0.0449     0.2334   -0.19     0.85    
## x             2.7084     0.0387   69.98   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.32 on 198 degrees of freedom
## Multiple R-squared:  0.961,  Adjusted R-squared:  0.961 
## F-statistic: 4.9e+03 on 1 and 198 DF,  p-value: <2e-16
share|improve this answer

The MLE is the value of the parameter of interest that maximizes the probability of observing the data that you observed. In other words, it is the value of the parameter that makes the observed data most likely to have been observed.

share|improve this answer
1  
And what if the likelihood function that is thus maximized, is, on its flip-side, the probability density function from a continuous random variable? Does the MLE still maximizes a probability? And if not, what does it do? –  Alecos Papadopoulos Aug 19 '14 at 13:29
    
@AlecosPapadopoulos It is my understanding that the likelihood function can be considered a probability function of the parameter, and the MLE is parameter value that maximizes that probability function. However your question suggests that there is more nuances? –  Heisenberg Aug 19 '14 at 15:27
3  
@Heisenberg The answer treated the likelihood function as the joint probability function of the sample (for which the ML provides the max w.r.t the parameters, and so maximizes probability for any sample). And this is correct when the r.v's are discrete, but not when they are continuous, since the joint density, by construction is not a joint probability. I wouldn't characterize this as a "nuance", it is a fundamental difference between the discrete and the continuous worlds. –  Alecos Papadopoulos Aug 19 '14 at 15:46
    
@AlecosPapadopoulos I see. So you take issue with the use of the word "probability function" vs "density function." That is correct. –  Heisenberg Aug 19 '14 at 15:54

If your data come from a probability distribution with an unknown parameter $\theta$, the maximum likelihood estimate of $\theta$ is that which makes the data you actually observed most probable.

In the case where your data are independent samples from that probability distribution, the likelihood (for a given value of $\theta$) is calculated by multiplying together the probabilities of all observations (for that given value of $\theta$) - it's just the joint probability of the whole sample. And the value of $\theta$ for which it's a maximum is the maximum likelihood estimate.

(If the data are continuous read 'probability density' for 'probability'. So if they're measured in inches the density would be measured in probability per inch.)

share|improve this answer
2  
One quibble. I don't think you can think of them as probabilities when $y$ is continuous. –  Dimitriy V. Masterov Dec 4 '12 at 1:07
    
@DimitriyV.Masterov Indeed, they're not. Even when you can, if I remember right, likelihood was only defined (by Fisher, I think) 'up to a multiplicative constant'. –  Glen_b Dec 4 '12 at 3:03
    
@Dimitriy, good point; I've added it. –  Scortchi Dec 4 '12 at 9:11
    
@Glen, For most purposes - likelihood ratio tests, maximum likelihood estimation - you can drop the constant. For comparing AIC between non-nested models you can't. Don't think it need enter into a layman's definition anyway. –  Scortchi Dec 4 '12 at 9:14
1  
As long as you drop the same constant, you still can. –  Glen_b Dec 4 '12 at 9:32

The maximum likelihood (ML) estimate of a parameter is the value of that parameter under which your actual observed data are most likely, relative to any other possible values of the parameter.

The idea is that there are any number of "true" parameter values that could have led to your actually observed data with some non-zero (albeit perhaps small) probability. But the ML estimate gives the parameter value that would have led to your observed data with the highest probability.

This must not be confused with the value of the parameter that is most likely to have actually produced your data!

I like the following passage from Sober (2008, pp. 9-10) on this distinction. In this passage, we have some observed data denoted $O$ and a hypothesis denoted $H$.

You need to remember that "likelihood" is a technical term. The likelihood of H, Pr(O|H), and the posterior probability of H, Pr(H|O), are different quantities and they can have different values. The likelihood of H is the probability that H confers on O, not the probability that O confers on H. Suppose you hear a noise coming from the attic of your house. You consider the hypothesis that there are gremlins up there bowling. The likelihood of this hypothesis is very high, since if there are gremlins bowling in the attic, there probably will be noise. But surely you don’t think that the noise makes it very probable that there are gremlins up there bowling. In this example, Pr(O|H) is high and Pr(H|O) is low. The gremlin hypothesis has a high likelihood (in the technical sense) but a low probability.

In terms of the example above, ML would favor the gremlin hypothesis. In this particular comical example, that is clearly a bad choice. But in a lot of other more realistic cases, the ML estimate might be a very reasonable one.

Reference

Sober, E. (2008). Evidence and Evolution: the Logic Behind the Science. Cambridge University Press.

share|improve this answer
4  
This seems to me to be the first answer that makes this crucial point clearly and simply. Note however, that it only "would have led to your observed data with the highest probability" if your data are discrete (like binomial data), but 'would have led to your observed data with the highest joint density' if your data are continuous (like normal data). –  gung Aug 20 '14 at 2:43
4  
Thanks @gung. I am aware of the technicality that you mention but I was slightly concerned that any discussion of "joint densities" would be a bit of a stretch for "layman's terms"... –  Jake Westfall Aug 20 '14 at 2:46
    
I agree w/ you, & I figured you knew about this. I just thought I'd mention it since it came up elsewhere on this thread. –  gung Aug 20 '14 at 15:16

It is possible to say something without using (much) math, but for actual statistical applications of maximum likelihood you need mathematics.

Maximum likelihood estimation is related to what philosophers call inference to the best explanation, or abduction. We use this all the time! Note, I do not say that maximum likelihood is abduction, that term is much wider, and some cases of Bayesian estimation (with an empirical prior) can probably also be seen as abduction. Some examples taken form http://plato.stanford.edu/entries/abduction/#Aca See also https://en.wikipedia.org/wiki/Abductive_reasoning (In computer science "abduction" is also used in the context of non-probabilistic models.)

  1. "You happen to know that Tim and Harry have recently had a terrible row that ended their friendship. Now someone tells you that she just saw Tim and Harry jogging together. The best explanation for this that you can think of is that they made up. You conclude that they are friends again." This because that conclusion makes the observation you try to explain more probable than under the alternative, that they are still not talking.

Another example: You work in a kindergarten, and one day a child starts to walk in a strange way, and saying he broke his legs. You examine and find nothing wrong. Then you can reasonably infer that one of his parents broke their legs, since children then often actuate as described, so that is an "inference to the best explanation" and an instance of (informal) maximum likelihood.

Abduction is about fonding pattern in data, and then searching for possible theories that can possibly make those patterns probable. Then choosing the possible explanation, which makes the observed pattern maximally probable, is just maximum likelihood!

The prime example of abduction in science is evolution. There is no one single observation that implies evolution, but evolution makes observed patterns more probable than other explanations.

Another typical example is medical diagnosos? Which possible medical condition makes the observed pattern of symptoms the most probable? Again, this is also maximum likelihood! (Or, in this case, maybe bayesian estimation is a better fit, we must take into account the prior probability of the various possible explanations). But that is a technicality, in this case we can have empirical priors which can be seen as a natural part of the statistical model, and what we call model, what we call prior is some arbitrary statistical convention.

To get back to the original question about layman term explanation of MLE, here is one simple example: When my daughters where 6 and 7 years old, I asked them this. We made two urns (two shoe-boxes), in one we put 2 black balls, 8 red, in the other the numbers where switched. Then we mixed the urns, and we draw one urn randomly. Then we took at random one ball from that urn. It was red.

Then I asked : From which urn do you think that red ball was drawn? After about one seconds thinking, they answered (in choir): From the one with 8 red ball!

Then I asked: Why do you think so? And anew, after about one second: "Because then it is easier to draw a red ball!". That is, easier=more probable. That was maximum likelihood (it is an easy exercise to write up the probability model), and it is "inference to the best explanation", that is, abduction.

share|improve this answer
    
googling for "abductive reasoning and maximum likelihood estimation" gives a lot of relevant hits. –  kjetil b halvorsen Aug 19 '14 at 15:29

Say you have some data $X$ that comes from Normal distribution with unknown mean $\mu$. You want to find what is the value of $\mu$, however you have no idea how to achieve it. One thing you could do is to try several values of $\mu$ and check which of them is the best one. To do this you need however some method for checking which of the values is "better" then others. The likelihood function, $L$, lets you to check which values of $\mu$ are most likely given the data you have. For this purpose it uses probabilities of your data-points estimated under a probability function $f$ with a given value of $\mu$:

$$ L(\mu|X) = \prod^N_{i=1} f(x_i, \mu) $$

or log-likelihood:

$$ \ln L(\mu|X) = \sum^N_{i=1} \ln f(x_i, \mu) $$

You use this function to check which value of $\mu$ maximizes the likelihood, i.e. which is the most likely given the data you have. As you can see, this can be achieved with product of probabilities or with sum of log-probabilities (log-likelihood). In our example $f$ would be probability density function for normal distribution, but the approach can be extended into much more complicated problems.

In practice you do not plug-in some guessed values of $\mu$ into the likelihood function but rather use different statistical approaches that are known to provide maximum likelihood estimates of the parameters of interest. There are lots of such approaches that are problem-specific - some are simple, some complicated (check Wikipedia for more information). Below I provide a simple example of how ML works in practice.

Example

First lets generate some fake data:

set.seed(123)
x <- rnorm(1000, 1.78)

and define a likelihood function that we want to maximize (the likelihood of Normal distribution with different values of $\mu$ given the data $X$):

llik <- function(mu) sum(log(dnorm(x, mu)))

next, what we do is we check different values of $\mu$ using our function:

ll <- vapply(seq(-6, 6, by=0.001), llik, numeric(1))

plot(seq(-6, 6, by=0.001), ll, type="l", ylab="Log-Likelihood", xlab=expression(mu))
abline(v=mean(x), col="red")

The same could be achieved faster with an optimization algorithm that looks for the maximum value of a function in a more clever way that going brute force. There are multiple such examples, e.g. one of the most basic in R is optimize:

optimize(llik, interval=c(-6, 6), maximum=TRUE)$maximum

enter image description here

The black line shows estimates of log-likelihood function under different values of $\mu$. The red line on the plot marks the $1.78$ value that is exactly the same as the arithmetic average (that actually is maximum likelihood estimator of $\mu$), the highest point of log-likelihood function estimated with brute force search and with optimize algorithm.

This example shows how you can use multiple approaches to find the value that maximizes the likelihood function to find the "best" value of your parameter.

share|improve this answer

As you wanted, I will use very naive terms. Suppose you have collected some data $\{y_1, y_2,\ldots,y_n\}$ and have reasonable assumption that they follow some probability distribution. But you don't usually know the parameter(s) of that distribution from such samples. Parameters are the "population characteristics" of the probability distribution that you have assumed for the data. Say, your plotting or prior knowledge suggests you to consider the data as being Normally distributed. Mean and variance are the two parameters that represent a Normal distribution. Let, $\theta=\{\mu,\sigma^2\}$ be the set of parameters. So the joint probability of observing the data $\{y_1, y_2,\ldots,y_n\}$ given the set of parameters $\theta=\{\mu,\sigma^2\}$ is given by, $p(y_1, y_2,\ldots,y_n|\theta)$.

Likelihood is "the probability of observing the data" so equivalent to the joint pdf (for discrete distribution joint pmf). But it is expressed as a function of the parameters or $L(\theta|y_1, y_2,\ldots,y_n)$. So that for this particular data set you may find the value of $\theta$ for which $L(\theta)$ is maximum. In words, you find $\theta$ for which the probability of observing this particular set of data is maximum. Thus comes the term "Maximum Likelihood". Now you find the set of $\{\mu,\sigma^2\}$ for which $L$ is maximized. That set of $\{\mu,\sigma^2\}$ for which $L(\theta)$ is maximum is called the Maximum Likelihood Estimate.

share|improve this answer

Suppose you have a coin. Tossing it can give either heads or tails. But you don't know if it's a fair coin. So you toss it 1000 times. It comes up as heads 1000 times, and never as tails.

Now, it's possible that this is actually a fair coin with a 50/50 chance for heads/tails, but it doesn't seem likely, does it? The chance of tossing a fair coin 1000 times and heads never coming up is $0.5^{2000}$, very small indeed.

The MLE tries to help you find the best explanation in a situation like this -- when you have some result, and you want to figure out what the value of the parameter is that is most likely to give that result. Here, we have 2000 heads out of 2000 tosses -- so we would use an MLE to find out what probability of getting a head best explains getting 2000 heads out of 2000 tosses.

It's the Maximum Likelihood Estimator. It estimates the parameter (here, it's a probability distribution function) that is most likely to have produced the result you are currently looking at.

To finish up our example, taking the MLE would return that the probability of getting a head that best explains getting 2000 heads out of 2000 tosses is $1$.

share|improve this answer

Try this link. It has pretty crisp explanation about MLE, MAP, EM. I think it covers basic idea of MLE in simple terms.

share|improve this answer
    
Welcome to our site! –  kjetil b halvorsen Nov 7 '14 at 8:46
    
@kjetilbhalvorsen Thanks :) –  Nimish Kulkarni Nov 9 '14 at 1:38

The way I understand MLE is this: You only get to see what the nature wants you to see. Things you see are facts. These facts have an underlying process that generated it. These process are hidden, unknown, needs to be discovered. Then the question is: Given the observed fact, what is the likelihood that process P1 generated it? What is the likelihood that process P2 generated it? And so on... One of these likelihoods is going to be max of all. MLE is a function that extracts that max likelihood.

Think of a coin toss; coin is biased. No one knows the degree of bias. It could range from o(all tails) to 1 (all heads). A fair coin will be 0.5 (head/tail equally likely). When you do 10 tosses, and you observe 7 Heads, then the MLE is that degree of bias which is more likely to produce the observed fact of 7 heads in 10 tosses.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.