Maximum Likelihood Estimation (MLE) in layman terms

Question

Could anyone explain to me in detail about maximum likelihood estimation (MLE) in layman's terms? I would like to know the underlying concept before going into mathematical derivation or equation.

Answer 1

Say you have some data. Say you're willing to assume that the data comes from some distribution -- perhaps Gaussian. There are an infinite number of different Gaussians that the data could have come from (which correspond to the combination of the infinite number of means and variances that a Gaussian distribution can have). MLE will pick the Gaussian (i.e., the mean and variance) that is "most consistent" with your data (the precise meaning of consistent is explained below).

So, say you've got a data set of $y = \{-1, 3, 7\}$. The most consistent Gaussian from which that data could have come has a mean of 3 and a variance of 16. It could have been sampled from some other Gaussian. But one with a mean of 3 and variance of 16 is most consistent with the data in the following sense: the probability of getting the particular $y$ values you observed is greater with this choice of mean and variance, than it is with any other choice.

Moving to regression: instead of the mean being a constant, the mean is a linear function of the data, as specified by the regression equation. So, say you've got data like $x = \{ 2,4,10 \}$ along with $y$ from before. The mean of that Gaussian is now the fitted regression model $X'\hat\beta$, where $\hat\beta =[-1.9,.9]$

Moving to GLMs: replace Gaussian with some other distribution (from the exponential family). The mean is now a linear function of the data, as specified by the regression equation, transformed by the link function. So, it's $g(X'\beta)$, where $g(x) = e^x/(1+e^x)$ for logit (with binomial data).

Answer 2

Maximum Likelihood Estimation (MLE) is a technique to find the most likely function that explains observed data. I think math is necessary, but don't let it scare you!

Let's say that we have a set of points in the $x,y$ plane, and we want to know the function parameters $\beta$ and $\sigma$ that most likely fit the data (in this case we know the function because I specified it to create this example, but bear with me).

data   <- data.frame(x = runif(200, 1, 10))
data$y <- 0 + beta*data$x + rnorm(200, 0, sigma)
plot(data$x, data$y)

In order to do a MLE, we need to make assumptions about the form of the function. In a linear model, we assume that the points follow a normal (Gaussian) probability distribution, with mean $x\beta$ and variance $\sigma^2$: $y = \mathcal{N}(x\beta, \sigma^2)$. The equation of this probability density function is: $$\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left(-\frac{(y_i-x_i\beta)^2}{2\sigma^2}\right)}$$

What we want to find is the parameters $\beta$ and $\sigma$ that maximize this probability for all points $(x_i, y_i)$. This is the "likelihood" function, $\mathcal{L}$

$$\mathcal{L} = \prod_{i=1}^n y_i = \prod_{i=1}^n \dfrac{1}{\sqrt{2\pi\sigma^2}} \exp\Big({-\dfrac{(y_i - x_i\beta)^2}{2\sigma^2}}\Big)$$ For various reasons, it's easier to use the log of the likelihood function: $$\log(\mathcal{L}) = \sum_{i = 1}^n-\frac{n}{2}\log(2\pi) -\frac{n}{2}\log(\sigma^2) - \frac{1}{2\sigma^2}(y_i - x_i\beta)^2$$

We can code this as a function in R with $\theta = (\beta,\sigma)$.

linear.lik <- function(theta, y, X){
  n      <- nrow(X)
  k      <- ncol(X)
  beta   <- theta[1:k]
  sigma2 <- theta[k+1]^2
  e      <- y - X%*%beta
  logl   <- -.5*n*log(2*pi)-.5*n*log(sigma2) - ( (t(e) %*% e)/ (2*sigma2) )
  return(-logl)
}

This function, at different values of $\beta$ and $\sigma$, creates a surface.

surface <- list()
k <- 0
for(beta in seq(0, 5, 0.1)){
  for(sigma in seq(0.1, 5, 0.1)){
    k <- k + 1
    logL <- linear.lik(theta = c(0, beta, sigma), y = data$y, X = cbind(1, data$x))
    surface[[k]] <- data.frame(beta = beta, sigma = sigma, logL = -logL)
  }
}
surface <- do.call(rbind, surface)
library(lattice)
wireframe(logL ~ beta*sigma, surface, shade = TRUE)

As you can see, there is a maximum point somewhere on this surface. We can find parameters that specify this point with R's built-in optimization commands. This comes reasonably close to uncovering the true parameters $0, \beta = 2.7, \sigma = 1.3$

linear.MLE <- optim(fn=linear.lik, par=c(1,1,1), lower = c(-Inf, -Inf, 1e-8), 
                    upper = c(Inf, Inf, Inf), hessian=TRUE, 
                    y=data$y, X=cbind(1, data$x), method = "L-BFGS-B")
linear.MLE$par


## [1] -0.1303868  2.7286616  1.3446534

Ordinary least squares is the maximum likelihood for a linear model, so it makes sense that lm would give us the same answers. (Note that $\sigma^2$ is used in determining the standard errors).

summary(lm(y ~ x, data))

## 
## Call:
## lm(formula = y ~ x, data = data)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.3616 -0.9898  0.1345  0.9967  3.8364 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) -0.13038    0.21298  -0.612    0.541    
## x            2.72866    0.03621  75.363   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.351 on 198 degrees of freedom
## Multiple R-squared:  0.9663, Adjusted R-squared:  0.9661 
## F-statistic:  5680 on 1 and 198 DF,  p-value: < 2.2e-16

Answer 3

The maximum likelihood (ML) estimate of a parameter is the value of that parameter under which your actual observed data are most likely, relative to any other possible values of the parameter.

The idea is that there are any number of "true" parameter values that could have led to your actually observed data with some non-zero (albeit perhaps small) probability. But the ML estimate gives the parameter value that would have led to your observed data with the highest probability.

This must not be confused with the value of the parameter that is most likely to have actually produced your data!

I like the following passage from Sober (2008, pp. 9-10) on this distinction. In this passage, we have some observed data denoted $O$ and a hypothesis denoted $H$.

You need to remember that "likelihood" is a technical term. The likelihood of H, Pr(O|H), and the posterior probability of H, Pr(H|O), are different quantities and they can have different values. The likelihood of H is the probability that H confers on O, not the probability that O confers on H. Suppose you hear a noise coming from the attic of your house. You consider the hypothesis that there are gremlins up there bowling. The likelihood of this hypothesis is very high, since if there are gremlins bowling in the attic, there probably will be noise. But surely you don’t think that the noise makes it very probable that there are gremlins up there bowling. In this example, Pr(O|H) is high and Pr(H|O) is low. The gremlin hypothesis has a high likelihood (in the technical sense) but a low probability.

In terms of the example above, ML would favor the gremlin hypothesis. In this particular comical example, that is clearly a bad choice. But in a lot of other more realistic cases, the ML estimate might be a very reasonable one.

Reference

Sober, E. (2008). Evidence and Evolution: the Logic Behind the Science. Cambridge University Press.

Answer 4

It is possible to say something without using (much) math, but for actual statistical applications of maximum likelihood you need mathematics.

Maximum likelihood estimation is related to what philosophers call inference to the best explanation, or abduction. We use this all the time! Note, I do not say that maximum likelihood is abduction, that term is much wider, and some cases of Bayesian estimation (with an empirical prior) can probably also be seen as abduction.

Some examples (the first taken from http://plato.stanford.edu/entries/abduction/#Aca). See also https://en.wikipedia.org/wiki/Abductive_reasoning (In computer science "abduction" is also used in the context of non-probabilistic models.)

1. "You happen to know that Tim and Harry have recently had a terrible row that ended their friendship. Now someone tells you that she just saw Tim and Harry jogging together. The best explanation for this that you can think of is that they made up. You conclude that they are friends again." This because that conclusion makes the observation you try to explain more probable than under the alternative, that they are still not talking.

2. You work in a kindergarten, and one day a child starts to walk in a strange way, and saying he broke his legs. You examine and find nothing wrong. Then you can reasonably infer that one of his parents broke their legs, since children then often actuate as described, so that is an "inference to the best explanation" and an instance of (informal) maximum likelihood. (and, of course, that explanation might be wrong, it is only probable, not sure. Abduction/maximum likelihood cannot give sure conclusions).

Abduction is about finding pattern in data, and then searching for possible theories that can possibly make those patterns probable. Then choosing the possible explanation, which makes the observed pattern maximally probable, is just maximum likelihood!

The prime example of abduction in science is evolution. There is no one single observation that implies evolution, but evolution makes observed patterns more probable than other explanations.

Another typical example is medical diagnosis? Which possible medical condition makes the observed pattern of symptoms the most probable? Again, this is also maximum likelihood! (Or, in this case, maybe bayesian estimation is a better fit, we must take into account the prior probability of the various possible explanations). But that is a technicality, in this case we can have empirical priors which can be seen as a natural part of the statistical model, and what we call model, what we call prior is some arbitrary(*) statistical convention.

To get back to the original question about layman term explanation of MLE, here is one simple example: When my daughters where 6 and 7 years old, I asked them this. We made two urns (two shoe-boxes), in one we put 2 black balls, 8 red, in the other the numbers where switched. Then we mixed the urns, and we draw one urn randomly. Then we took at random one ball from that urn. It was red.

Then I asked : From which urn do you think that red ball was drawn? After about one seconds thinking, they answered (in choir): From the one with 8 red balls!

Then I asked: Why do you think so? And anew, after about one second (in choir again): "Because then it is easier to draw a red ball!". That is, easier=more probable. That was maximum likelihood (it is an easy exercise to write up the probability model), and it is "inference to the best explanation", that is, abduction.

(*) Why do I say "arbitrary?" To continue the medical diagnosis problem, say the patient is a man with some difficult to diagnose condition the physician didn't see earlier. Then, say, in the talk with the patient it arises that he visited someplace in tropical Africa short time ago. That is a new piece of data, but its effect in the typical models (used in this kind of situation, be it formal or informal) will be to change the prior of the difficult possible explanations, as tropical diseases like malaria now will get higher prior probability. So the new data enters the analysis in the prior.

Answer 5

The MLE is the value of the parameter of interest that maximizes the probability of observing the data that you observed. In other words, it is the value of the parameter that makes the observed data most likely to have been observed.

Answer 6

If your data come from a probability distribution with an unknown parameter $\theta$, the maximum likelihood estimate of $\theta$ is that which makes the data you actually observed most probable.

In the case where your data are independent samples from that probability distribution, the likelihood (for a given value of $\theta$) is calculated by multiplying together the probabilities of all observations (for that given value of $\theta$) - it's just the joint probability of the whole sample. And the value of $\theta$ for which it's a maximum is the maximum likelihood estimate.

(If the data are continuous read 'probability density' for 'probability'. So if they're measured in inches the density would be measured in probability per inch.)

Answer 7

Let's play a game: I am in a dark room, no one can see what I do but you know that either (a) I throw a dice and count the number of '1's as 'success' or (b) I toss a coin and I count the number of heads as 'success'.

As I said, you can not see which of the two I do but I give you just one single piece of information: I tell you that I have thrown a dice 100 times or I have tossed the coin 100 times and that I had 17 successes.

The question is to guess whether I have thrown a dice or tossed a coin.

You will probably answer that I tossed a dice.

If you do, then you probably have 'made a guess by maximizing the likelihood' because if I observe 17 successes out 100 experiments, it is more likely that I have thrown a dice than that I have tossed a coin.

So what you have done is taking that value of the 'probability of success' (1/6 for a dice and 1/2 for a coin) that makes it most likely to observe 17 successes in 100. 'More likely' meaning that the chance that you have 17 times a '1' in 100 throws of a dice is higher than the chance of having 17 heads out of 100 coin tosses.

Answer 8

Say you have some data $X$ that comes from Normal distribution with unknown mean $\mu$. You want to find what is the value of $\mu$, however you have no idea how to achieve it. One thing you could do is to try several values of $\mu$ and check which of them is the best one. To do this you need however some method for checking which of the values is "better" then others. The likelihood function, $L$, lets you to check which values of $\mu$ are most likely given the data you have. For this purpose it uses probabilities of your data-points estimated under a probability function $f$ with a given value of $\mu$:

$$ L(\mu|X) = \prod^N_{i=1} f(x_i, \mu) $$

or log-likelihood:

$$ \ln L(\mu|X) = \sum^N_{i=1} \ln f(x_i, \mu) $$

You use this function to check which value of $\mu$ maximizes the likelihood, i.e. which is the most likely given the data you have. As you can see, this can be achieved with product of probabilities or with sum of log-probabilities (log-likelihood). In our example $f$ would be probability density function for normal distribution, but the approach can be extended into much more complicated problems.

In practice you do not plug-in some guessed values of $\mu$ into the likelihood function but rather use different statistical approaches that are known to provide maximum likelihood estimates of the parameters of interest. There are lots of such approaches that are problem-specific - some are simple, some complicated (check Wikipedia for more information). Below I provide a simple example of how ML works in practice.

Example

First lets generate some fake data:

    set.seed(123)
    x <- rnorm(1000, 1.78)

and define a likelihood function that we want to maximize (the likelihood of Normal distribution with different values of $\mu$ given the data $X$):

    llik <- function(mu) sum(log(dnorm(x, mu)))

next, what we do is we check different values of $\mu$ using our function:

    ll <- vapply(seq(-6, 6, by=0.001), llik, numeric(1))

    plot(seq(-6, 6, by=0.001), ll, type="l", ylab="Log-Likelihood", 
              xlab=expression(mu))
    abline(v=mean(x), col="red")

The same could be achieved faster with an optimization algorithm that looks for the maximum value of a function in a more clever way that going brute force. There are multiple such examples, e.g. one of the most basic in R is optimize:

    optimize(llik, interval=c(-6, 6), maximum=TRUE)$maximum

The black line shows estimates of log-likelihood function under different values of $\mu$. The red line on the plot marks the $1.78$ value that is exactly the same as the arithmetic average (that actually is maximum likelihood estimator of $\mu$), the highest point of log-likelihood function estimated with brute force search and with optimize algorithm.

This example shows how you can use multiple approaches to find the value that maximizes the likelihood function to find the "best" value of your parameter.

Answer 9

One task in statistics is to fit a distribution function to a set of data points to generalize what's intrinsic about the data. When one is fitting a distribution a)choose an appropriate distribution b)set the movable parts (parameters), for example mean, variance, etc. When doing all this one also needs an objective, aka objective function/error function. This is required to define the meaning of "best" or "best in what sense". MLE is the procedure where this objective function is set as the maximum of the probability mass/density function of the chosen distribution. Other techniques differ how they choose this objective function. For example ordinary least squares (OLS) takes the minimum sum of squared errors. For the Gaussian case OLS and MLE are equivalent because the Gaussian distribution has that (x-m)^2 term in the density function that makes the objectives of OLS and MLE coincide. You can see that it is a squared difference term like OLS.

Of course one can choose any objective function. However the intuitive meaning will not be always clear. MLE assumes that we know the distribution to start with. In other techniques, this assumption is relaxed. Especially in those cases it is more common to have a custom objective function.

Answer 10

As you wanted, I will use very naive terms. Suppose you have collected some data $\{y_1, y_2,\ldots,y_n\}$ and have reasonable assumption that they follow some probability distribution. But you don't usually know the parameter(s) of that distribution from such samples. Parameters are the "population characteristics" of the probability distribution that you have assumed for the data. Say, your plotting or prior knowledge suggests you to consider the data as being Normally distributed. Mean and variance are the two parameters that represent a Normal distribution. Let, $\theta=\{\mu,\sigma^2\}$ be the set of parameters. So the joint probability of observing the data $\{y_1, y_2,\ldots,y_n\}$ given the set of parameters $\theta=\{\mu,\sigma^2\}$ is given by, $p(y_1, y_2,\ldots,y_n|\theta)$.

Likelihood is "the probability of observing the data" so equivalent to the joint pdf (for discrete distribution joint pmf). But it is expressed as a function of the parameters or $L(\theta|y_1, y_2,\ldots,y_n)$. So that for this particular data set you may find the value of $\theta$ for which $L(\theta)$ is maximum. In words, you find $\theta$ for which the probability of observing this particular set of data is maximum. Thus comes the term "Maximum Likelihood". Now you find the set of $\{\mu,\sigma^2\}$ for which $L$ is maximized. That set of $\{\mu,\sigma^2\}$ for which $L(\theta)$ is maximum is called the Maximum Likelihood Estimate.

Answer 11

Suppose you have a coin. Tossing it can give either heads or tails. But you don't know if it's a fair coin. So you toss it 1000 times. It comes up as heads 1000 times, and never as tails.

Now, it's possible that this is actually a fair coin with a 50/50 chance for heads/tails, but it doesn't seem likely, does it? The chance of tossing a fair coin 1000 times and heads never coming up is $0.5^{2000}$, very small indeed.

The MLE tries to help you find the best explanation in a situation like this -- when you have some result, and you want to figure out what the value of the parameter is that is most likely to give that result. Here, we have 2000 heads out of 2000 tosses -- so we would use an MLE to find out what probability of getting a head best explains getting 2000 heads out of 2000 tosses.

It's the Maximum Likelihood Estimator. It estimates the parameter (here, it's a probability distribution function) that is most likely to have produced the result you are currently looking at.

To finish up our example, taking the MLE would return that the probability of getting a head that best explains getting 2000 heads out of 2000 tosses is $1$.

Answer 12

Just to show very simple graphics and R code for MLEs in binomial and Poisson models.

Binomial. Suppose you know there are $n = 50$ trials of which $x=19$ are Successes. Then for what value of $p$ is the binomial PDF maximized? This PDF considered as a function of $p$ and (possibly) without its norming constant) is called a likelihood function. Then the MLE of $p$ is $\hat p = x/n = 19/50 = 0.32.$

p = seq(0,1, by=.01)
like = dbinom(19, 59, p)
mle = mean(p[like==max(like)]);  mle
      # 'mean' in case two values of 'like' at max
[1] 0.38

hdr = "MLE of Binomial Success, Probability"
plot(p, like, type="l", lwd=2, main=hdr)
 abline(h=0, col="green2")
 abline(v = mle, lwd=2, lty="dotted", col="red")

Poisson. Suppose the model is Poisson with mean $\lambda$ in a domain of time, area, or volume. We observe $x = 13$ Poisson events in the domain. Then the MLE of $\lambda$ is $\hat \lambda = 13.$

lam = seq(0, 50, by = .01)
like = dpois(13, lam)
mle = mean(lam[like==max(like)]);  mle
[1] 13
hdr="MLE of Poisson Mean"
plot(lam, like, type="l", lwd=2, main=hdr)
 abline(h=0, col="green2")
 abline(v = mle, lwd=2, lty="dotted", col="red")

Answer 13

You have a model, which you impose that the data comes from. In a way, you wanna reconcile between the model and reality. To do so, you wanna minimize the discrepancy between the two. How would you do that? You have $\Theta$ vector of parameters that you can tune in order to achieve so. Minimizing the discrepancy between reality and the model is similar to choosing the parameters that make it most likely that the true data came from the model. Hence, the idea of maximum likelihood. Hope it makes sense!

Answer 14

The way I understand MLE is this: You only get to see what the nature wants you to see. Things you see are facts. These facts have an underlying process that generated it. These process are hidden, unknown, needs to be discovered. Then the question is: Given the observed fact, what is the likelihood that process P1 generated it? What is the likelihood that process P2 generated it? And so on... One of these likelihoods is going to be max of all. MLE is a function that extracts that max likelihood.

Think of a coin toss; coin is biased. No one knows the degree of bias. It could range from o(all tails) to 1 (all heads). A fair coin will be 0.5 (head/tail equally likely). When you do 10 tosses, and you observe 7 Heads, then the MLE is that degree of bias which is more likely to produce the observed fact of 7 heads in 10 tosses.