# Maximum Likelihood Estimation (MLE) in layman terms

Could anyone explain to me in detail about maximum likelihood estimation (MLE) in layman's terms? I would like to know the underlying concept before going into mathematical derivation or equation.

-
It's unclear what kind of answer you're after. Do you know what likelihood is, for example? If not, better to find out that first. – Glen_b Aug 19 '14 at 12:56
In addition, I think any answer that doesn't involve math at some level will be inadequate. – gregmacfarlane Aug 19 '14 at 13:17

Say you have some data. Say you're willing to assume that the data comes from some distribution -- perhaps Gaussian. There are an infinite number of different Gaussians that the data could have come from: different means, different variances. MLE will pick the Gaussian that is "most consistent" with your data (the precise meaning of consistent is explained below).

So say you've got a data set of y = -1,3, and 7. The most consistent Gaussian from which that data could have come has a mean of 3 and a variance of 16. It could have been sampled from some other Gaussian. But one with a mean of 3 and variance of 16 is most consistent with the data in the following sense: the probability of getting the particular $y$ values you observed is greater with this choice of mean and variance, than it is with any other choice.

Moving to regression: instead of the mean being a constant, the mean is a linear function of the data, as specified by the regression equation. So, say you've got data like x = 2,4,10 along with y from before. The mean of that Gaussian is now the fitted regression model $X'\hat\beta$, where $\hat\beta =[-1.9,.9]$

Moving to GLMs: replace Gaussian with some other distribution (from the exponential family). The mean is now a linear function of the data, as specified by the regression equation, transformed by the link function. So, it's $g(X'\beta)$, where $g(x) = e^x/(1+e^x)$ for logit (with binomial data).

-
"MLE will pick the Gaussian that is the most likely one, given your data." Hmmm, isn't it actually: MLE will pick the Gaussian under which your data are most likely? Which is slightly different from picking the "most likely Gaussian"... wouldn't picking the most likely Gaussian require a consideration of prior beliefs? – Jake Westfall Aug 20 '14 at 1:01
You'd need to clearly specify what you mean by "the most likely one" or this is likely to be misunderstood. – Glen_b Aug 20 '14 at 2:33
@ACD I don't think this is just incomplete but provides the correct intuition. For example, I don't see any problem with not discussing special cases like the likelihood function have more than one maximum. But the difference between the distribution most likely to produce the observed data and the most likely distribution given the data is the very fundamental difference between frequentist and bayesian inference. So if you explain it like that, you are just creating a stumbling block for the future. – Erik Aug 25 '14 at 11:30
Sure okay, but is the more correct conceptual explanation any harder to understand than the one you've written? I don't think so. I think most of your answer is just fine, but I would just urge you, for the sake of posterity, to just slightly edit some of the phrasing to avoid discussing "the most likely Gaussian" and instead point out that the thing that we want to be "likely" (in colloquial terms) under ML is not the hypothesis but the data. I think this can be a minor but important edit to your otherwise nice answer. – Jake Westfall Aug 25 '14 at 16:16
@Max: Thanks a lot for finally going ahead and fixing this answer! I think it might make sense to explicitly write here for the future readers: the critique expressed in the above upvoted comments by Erik and Jake does not apply anymore, after the answer has been edited. – amoeba Oct 7 '15 at 22:38

Maximum Likelihood Estimation (MLE) is a technique to find the most likely function that explains observed data. I think math is necessary, but don't let it scare you!

Let's say that we have a set of points in the $x,y$ plane, and we want to know the function parameters $\beta$ and $\sigma$ that most likely fit the data (in this case we know the function because I specified it to create this example, but bear with me).

data   <- data.frame(x = runif(200, 1, 10))
data$y <- 0 + beta*data$x + rnorm(200, 0, sigma2)
plot(data$x, data$y)


In order to do a MLE, we need to make assumptions about the form of the function. In a linear model, we assume that the points follow a normal (Gaussian) probability distribution, with mean $x\beta$ and standard deviation $\sigma^2$:
$y = \mathcal{N}(x\beta, \sigma^2)$. The equation of this probability density function is: $$\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left(-\frac{(y_i-x_i\beta)^2}{2\sigma^2}\right)}$$

What we want to find is the parameters $\beta$ and $\sigma$ that maximize this probability for all points $(x_i, y_i)$. This is the "likelihood" function, $\mathcal{L}$

$$\mathcal{L} = \prod_{i=1}^n y_i = \prod_{i=1}^n \dfrac{1}{\sqrt{2\pi\sigma^2}} \exp\Big({-\dfrac{(y_i - x_i\beta)^2}{2\sigma^2}}\Big)$$ For various reasons, it's easier to use the log of the likelihood function: $$\log(\mathcal{L}) = \sum_{i = 1}^n-\frac{n}{2}\log(2\pi) -\frac{n}{2}\log(\sigma^2) - \frac{1}{2\sigma^2}(y_i - x_i\beta)^2$$

We can code this as a function in R with $\theta = (\beta,\sigma^2)$.

linear.lik <- function(theta, y, X){
n      <- nrow(X)
k      <- ncol(X)
beta   <- theta[1:k]
sigma2 <- theta[k+1]
e      <- y - X%*%beta
logl   <- -.5*n*log(2*pi)-.5*n*log(sigma2) - ( (t(e) %*% e)/ (2*sigma2) )
return(-logl)
}


This function, at different values of $\beta$ and $\sigma^2$, creates a surface.

surface <- list()
k <- 0
for(beta in seq(0, 5, 0.1)){
for(sigma2 in seq(0.1, 5, 0.1)){
k <- k + 1
logL <- linear.lik(theta = c(0, beta, sigma2), y = data$y, X = cbind(1, data$x))
surface[[k]] <- data.frame(beta = beta, sigma2 = sigma2, logL = -logL)
}
}
surface <- do.call(rbind, surface)
library(lattice)
wireframe(logL ~ beta*sigma2, surface, shade = TRUE)


As you can see, there is a maximum point somewhere on this surface. We can find parameters that specify this point with R's built-in optimization commands. This comes reasonably close to uncovering the true parameters $0, \beta = 2.7, \sigma^2 = 1.3$

linear.MLE <- nlm(f=linear.lik, p=c(1,1,1), hessian=TRUE, y=data$y, X=cbind(1, data$x))
linear.MLE$estimate ## [1] -0.04481 2.70841 1.71870  Ordinary least squares is the maximum likelihood for a linear model, so it makes sense that lm would give us the same answers. (Note that$\sigma^2$is used in determining the standard errors). summary(lm(y ~ x, data)) ## ## Call: ## lm(formula = y ~ x, data = data) ## ## Residuals: ## Min 1Q Median 3Q Max ## -3.106 -0.893 0.032 0.880 3.281 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) -0.0449 0.2334 -0.19 0.85 ## x 2.7084 0.0387 69.98 <2e-16 *** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 1.32 on 198 degrees of freedom ## Multiple R-squared: 0.961, Adjusted R-squared: 0.961 ## F-statistic: 4.9e+03 on 1 and 198 DF, p-value: <2e-16  - Thanks @gregmacfarlane for this nice and useful answer. A minor point: should we not first define beta and sigma2 in the line R code data$y <- 0 + beta*data$x + rnorm(200, 0, sigma2) ? And is 0 + useful? – emeryville Dec 10 '15 at 6:07 Yes beta and sigma2 would need to be defined for this code to run. I hid them so that we could "discover" the parameters, which are almost always unknown when you run MLE. – gregmacfarlane Dec 10 '15 at 17:58 You're also right that 0 + anything doesn't actually do anything; I merely included it because regression models usually have an intercept. And if the MLE were trying to optimize beta, sigma2 and alpha, I couldn't show the pretty surface plot (unless you know of an R package that will plot in four dimensions!) – gregmacfarlane Dec 10 '15 at 17:59 @gregmacfarlane Based on your great answer, is my understanding correct: the MLE in regression finds the means and variance (which is assumed to be constant) of many probability distribution under which our data (y given x) is most likely to occur? Additionally, the variance is constant due to homoscedasticity assumption? – quirik May 23 at 12:39 I'm not sure exactly what you mean. Given an IID assumption (non-heteroskedastic) the variance-covariance matrix of the model coefficients is$\sigma^2(X'X)^{-1}$, and the MLE estimate of sigma2 is an unbiased estimator of the scale on that VCOV. If IID does not hold, then this estimator is invalid. But you also can't guarantee that the variance is in every likelihood function. Sometimes you have to estimate the variance separately (through simulation, etc). – gregmacfarlane May 23 at 17:37 The MLE is the value of the parameter of interest that maximizes the probability of observing the data that you observed. In other words, it is the value of the parameter that makes the observed data most likely to have been observed. - And what if the likelihood function that is thus maximized, is, on its flip-side, the probability density function from a continuous random variable? Does the MLE still maximizes a probability? And if not, what does it do? – Alecos Papadopoulos Aug 19 '14 at 13:29 @AlecosPapadopoulos It is my understanding that the likelihood function can be considered a probability function of the parameter, and the MLE is parameter value that maximizes that probability function. However your question suggests that there is more nuances? – Heisenberg Aug 19 '14 at 15:27 @Heisenberg The answer treated the likelihood function as the joint probability function of the sample (for which the ML provides the max w.r.t the parameters, and so maximizes probability for any sample). And this is correct when the r.v's are discrete, but not when they are continuous, since the joint density, by construction is not a joint probability. I wouldn't characterize this as a "nuance", it is a fundamental difference between the discrete and the continuous worlds. – Alecos Papadopoulos Aug 19 '14 at 15:46 @AlecosPapadopoulos I see. So you take issue with the use of the word "probability function" vs "density function." That is correct. – Heisenberg Aug 19 '14 at 15:54 @ Alecos Papadopoulos: The real<point here is, maybe, that to justify MLE in the discrete case is direct, while the continuous case needs an extra argument, we can find the joint probability for some$\epsilon$-cube around the data point, and then observe that for small enough$\epsilon$, the concrete value of$\epsilon$do not matter. – kjetil b halvorsen Sep 8 '15 at 12:02 If your data come from a probability distribution with an unknown parameter$\theta$, the maximum likelihood estimate of$\theta$is that which makes the data you actually observed most probable. In the case where your data are independent samples from that probability distribution, the likelihood (for a given value of$\theta$) is calculated by multiplying together the probabilities of all observations (for that given value of$\theta$) - it's just the joint probability of the whole sample. And the value of$\theta$for which it's a maximum is the maximum likelihood estimate. (If the data are continuous read 'probability density' for 'probability'. So if they're measured in inches the density would be measured in probability per inch.) - One quibble. I don't think you can think of them as probabilities when$y$is continuous. – Dimitriy V. Masterov Dec 4 '12 at 1:07 @DimitriyV.Masterov Indeed, they're not. Even when you can, if I remember right, likelihood was only defined (by Fisher, I think) 'up to a multiplicative constant'. – Glen_b Dec 4 '12 at 3:03 @Dimitriy, good point; I've added it. – Scortchi Dec 4 '12 at 9:11 @Glen, For most purposes - likelihood ratio tests, maximum likelihood estimation - you can drop the constant. For comparing AIC between non-nested models you can't. Don't think it need enter into a layman's definition anyway. – Scortchi Dec 4 '12 at 9:14 As long as you drop the same constant, you still can. – Glen_b Dec 4 '12 at 9:32 The maximum likelihood (ML) estimate of a parameter is the value of that parameter under which your actual observed data are most likely, relative to any other possible values of the parameter. The idea is that there are any number of "true" parameter values that could have led to your actually observed data with some non-zero (albeit perhaps small) probability. But the ML estimate gives the parameter value that would have led to your observed data with the highest probability. This must not be confused with the value of the parameter that is most likely to have actually produced your data! I like the following passage from Sober (2008, pp. 9-10) on this distinction. In this passage, we have some observed data denoted$O$and a hypothesis denoted$H$. You need to remember that "likelihood" is a technical term. The likelihood of H, Pr(O|H), and the posterior probability of H, Pr(H|O), are different quantities and they can have different values. The likelihood of H is the probability that H confers on O, not the probability that O confers on H. Suppose you hear a noise coming from the attic of your house. You consider the hypothesis that there are gremlins up there bowling. The likelihood of this hypothesis is very high, since if there are gremlins bowling in the attic, there probably will be noise. But surely you don’t think that the noise makes it very probable that there are gremlins up there bowling. In this example, Pr(O|H) is high and Pr(H|O) is low. The gremlin hypothesis has a high likelihood (in the technical sense) but a low probability. In terms of the example above, ML would favor the gremlin hypothesis. In this particular comical example, that is clearly a bad choice. But in a lot of other more realistic cases, the ML estimate might be a very reasonable one. Reference Sober, E. (2008). Evidence and Evolution: the Logic Behind the Science. Cambridge University Press. - This seems to me to be the first answer that makes this crucial point clearly and simply. Note however, that it only "would have led to your observed data with the highest probability" if your data are discrete (like binomial data), but 'would have led to your observed data with the highest joint density' if your data are continuous (like normal data). – gung Aug 20 '14 at 2:43 Thanks @gung. I am aware of the technicality that you mention but I was slightly concerned that any discussion of "joint densities" would be a bit of a stretch for "layman's terms"... – Jake Westfall Aug 20 '14 at 2:46 I agree w/ you, & I figured you knew about this. I just thought I'd mention it since it came up elsewhere on this thread. – gung Aug 20 '14 at 15:16 It is possible to say something without using (much) math, but for actual statistical applications of maximum likelihood you need mathematics. Maximum likelihood estimation is related to what philosophers call inference to the best explanation, or abduction. We use this all the time! Note, I do not say that maximum likelihood is abduction, that term is much wider, and some cases of Bayesian estimation (with an empirical prior) can probably also be seen as abduction. Some examples taken from http://plato.stanford.edu/entries/abduction/#Aca See also https://en.wikipedia.org/wiki/Abductive_reasoning (In computer science "abduction" is also used in the context of non-probabilistic models.) 1. "You happen to know that Tim and Harry have recently had a terrible row that ended their friendship. Now someone tells you that she just saw Tim and Harry jogging together. The best explanation for this that you can think of is that they made up. You conclude that they are friends again." This because that conclusion makes the observation you try to explain more probable than under the alternative, that they are still not talking. Another example: You work in a kindergarten, and one day a child starts to walk in a strange way, and saying he broke his legs. You examine and find nothing wrong. Then you can reasonably infer that one of his parents broke their legs, since children then often actuate as described, so that is an "inference to the best explanation" and an instance of (informal) maximum likelihood. (and, of course, that explanation might be wrong, it is only probable, not sure. Abduction/maximum likelihood cannot give sure conclusions). Abduction is about finding pattern in data, and then searching for possible theories that can possibly make those patterns probable. Then choosing the possible explanation, which makes the observed pattern maximally probable, is just maximum likelihood! The prime example of abduction in science is evolution. There is no one single observation that implies evolution, but evolution makes observed patterns more probable than other explanations. Another typical example is medical diagnosis? Which possible medical condition makes the observed pattern of symptoms the most probable? Again, this is also maximum likelihood! (Or, in this case, maybe bayesian estimation is a better fit, we must take into account the prior probability of the various possible explanations). But that is a technicality, in this case we can have empirical priors which can be seen as a natural part of the statistical model, and what we call model, what we call prior is some arbitrary(*) statistical convention. To get back to the original question about layman term explanation of MLE, here is one simple example: When my daughters where 6 and 7 years old, I asked them this. We made two urns (two shoe-boxes), in one we put 2 black balls, 8 red, in the other the numbers where switched. Then we mixed the urns, and we draw one urn randomly. Then we took at random one ball from that urn. It was red. Then I asked : From which urn do you think that red ball was drawn? After about one seconds thinking, they answered (in choir): From the one with 8 red balls! Then I asked: Why do you think so? And anew, after about one second (inchoir again): "Because then it is easier to draw a red ball!". That is, easier=more probable. That was maximum likelihood (it is an easy exercise to write up the probability model), and it is "inference to the best explanation", that is, abduction. (*) Why do I say "arbitrary?" To continue the medical diagnosis problem, say the patient is a man with some difficlt to diagnose condition the physician didnt see earlier. Then, say, in the talk with the patient it arises that he visited someplace in tropical Africa short time ago. That is a new piece of data, but its effect in the typical models (used in this kind of situation, be it formal or informal) will be to change the prior of the difficult possible explanations, as tropical diseases like malaria now will get higher prior probability. So the new data enters the analysis in the prior. - googling for "abductive reasoning and maximum likelihood estimation" gives a lot of relevant hits. – kjetil b halvorsen Aug 19 '14 at 15:29 Let's play a game: I am in a dark room, no one can see what I do but you know that either (a) I throw a dice and count the number of '1's as 'success' or (b) I toss a coin and I count the number of heads as 'success'. As I said, you can not see which of the two I do but I give you just one single piece of information: I tell you that I have thrown a dice 100 times or I have tossed the coin 100 times and that I had 17 successes. The question is to guess whether I have thrown a dice or tossed a coin. You will probably answer that I tossed a dice. If you do, then you probably have 'made a guess by maximizing the likelihood' because if I observe 17 successes out 100 experiments, it is more likely that I have thrown a dice than that I have tossed a coin. So what you have done is taking that value of the 'probability of success' (1/6 for a dice and 1/2 for a coin) that makes it most likely to observe 17 successes in 100. 'More likely' meaning that the chance that you have 17 times a '1' in 100 throws of a dice is higher than the chance of having 17 heads out of 100 coin tosses. - As I said in my answer, 'abduction' or 'inference to the best explanation'. – kjetil b halvorsen Sep 8 '15 at 19:53 @kjetil b halvorsen: I don't understand what you want to say? – fcop Sep 8 '15 at 20:05 I only try to compare to my answer above. Those terms are used in other fields (philosoph, CS) for more or less the same idea: Choose the explanation which overall fits best to the facts, with a probabilistic model that leads to maximum likelihood. – kjetil b halvorsen Sep 8 '15 at 20:14 @kjetil b halvorsen: can I then conclude that my example is ok ? To say the same in layman's terms :-) ? – fcop Sep 8 '15 at 20:22 Say you have some data$X$that comes from Normal distribution with unknown mean$\mu$. You want to find what is the value of$\mu$, however you have no idea how to achieve it. One thing you could do is to try several values of$\mu$and check which of them is the best one. To do this you need however some method for checking which of the values is "better" then others. The likelihood function,$L$, lets you to check which values of$\mu$are most likely given the data you have. For this purpose it uses probabilities of your data-points estimated under a probability function$f$with a given value of$\mu$: $$L(\mu|X) = \prod^N_{i=1} f(x_i, \mu)$$ or log-likelihood: $$\ln L(\mu|X) = \sum^N_{i=1} \ln f(x_i, \mu)$$ You use this function to check which value of$\mu$maximizes the likelihood, i.e. which is the most likely given the data you have. As you can see, this can be achieved with product of probabilities or with sum of log-probabilities (log-likelihood). In our example$f$would be probability density function for normal distribution, but the approach can be extended into much more complicated problems. In practice you do not plug-in some guessed values of$\mu$into the likelihood function but rather use different statistical approaches that are known to provide maximum likelihood estimates of the parameters of interest. There are lots of such approaches that are problem-specific - some are simple, some complicated (check Wikipedia for more information). Below I provide a simple example of how ML works in practice. Example First lets generate some fake data: set.seed(123) x <- rnorm(1000, 1.78)  and define a likelihood function that we want to maximize (the likelihood of Normal distribution with different values of$\mu$given the data$X$): llik <- function(mu) sum(log(dnorm(x, mu)))  next, what we do is we check different values of$\mu$using our function: ll <- vapply(seq(-6, 6, by=0.001), llik, numeric(1)) plot(seq(-6, 6, by=0.001), ll, type="l", ylab="Log-Likelihood", xlab=expression(mu)) abline(v=mean(x), col="red")  The same could be achieved faster with an optimization algorithm that looks for the maximum value of a function in a more clever way that going brute force. There are multiple such examples, e.g. one of the most basic in R is optimize: optimize(llik, interval=c(-6, 6), maximum=TRUE)$maximum


The black line shows estimates of log-likelihood function under different values of $\mu$. The red line on the plot marks the $1.78$ value that is exactly the same as the arithmetic average (that actually is maximum likelihood estimator of $\mu$), the highest point of log-likelihood function estimated with brute force search and with optimize algorithm.

This example shows how you can use multiple approaches to find the value that maximizes the likelihood function to find the "best" value of your parameter.

-

As you wanted, I will use very naive terms. Suppose you have collected some data $\{y_1, y_2,\ldots,y_n\}$ and have reasonable assumption that they follow some probability distribution. But you don't usually know the parameter(s) of that distribution from such samples. Parameters are the "population characteristics" of the probability distribution that you have assumed for the data. Say, your plotting or prior knowledge suggests you to consider the data as being Normally distributed. Mean and variance are the two parameters that represent a Normal distribution. Let, $\theta=\{\mu,\sigma^2\}$ be the set of parameters. So the joint probability of observing the data $\{y_1, y_2,\ldots,y_n\}$ given the set of parameters $\theta=\{\mu,\sigma^2\}$ is given by, $p(y_1, y_2,\ldots,y_n|\theta)$.

Likelihood is "the probability of observing the data" so equivalent to the joint pdf (for discrete distribution joint pmf). But it is expressed as a function of the parameters or $L(\theta|y_1, y_2,\ldots,y_n)$. So that for this particular data set you may find the value of $\theta$ for which $L(\theta)$ is maximum. In words, you find $\theta$ for which the probability of observing this particular set of data is maximum. Thus comes the term "Maximum Likelihood". Now you find the set of $\{\mu,\sigma^2\}$ for which $L$ is maximized. That set of $\{\mu,\sigma^2\}$ for which $L(\theta)$ is maximum is called the Maximum Likelihood Estimate.

-

Suppose you have a coin. Tossing it can give either heads or tails. But you don't know if it's a fair coin. So you toss it 1000 times. It comes up as heads 1000 times, and never as tails.

Now, it's possible that this is actually a fair coin with a 50/50 chance for heads/tails, but it doesn't seem likely, does it? The chance of tossing a fair coin 1000 times and heads never coming up is $0.5^{2000}$, very small indeed.

The MLE tries to help you find the best explanation in a situation like this -- when you have some result, and you want to figure out what the value of the parameter is that is most likely to give that result. Here, we have 2000 heads out of 2000 tosses -- so we would use an MLE to find out what probability of getting a head best explains getting 2000 heads out of 2000 tosses.

It's the Maximum Likelihood Estimator. It estimates the parameter (here, it's a probability distribution function) that is most likely to have produced the result you are currently looking at.

To finish up our example, taking the MLE would return that the probability of getting a head that best explains getting 2000 heads out of 2000 tosses is $1$.

-

Try this link. It has pretty crisp explanation about MLE, MAP, EM. I think it covers basic idea of MLE in simple terms.

-
Welcome to our site! – kjetil b halvorsen Nov 7 '14 at 8:46
@kjetilbhalvorsen Thanks :) – Nimish Kulkarni Nov 9 '14 at 1:38

The way I understand MLE is this: You only get to see what the nature wants you to see. Things you see are facts. These facts have an underlying process that generated it. These process are hidden, unknown, needs to be discovered. Then the question is: Given the observed fact, what is the likelihood that process P1 generated it? What is the likelihood that process P2 generated it? And so on... One of these likelihoods is going to be max of all. MLE is a function that extracts that max likelihood.

Think of a coin toss; coin is biased. No one knows the degree of bias. It could range from o(all tails) to 1 (all heads). A fair coin will be 0.5 (head/tail equally likely). When you do 10 tosses, and you observe 7 Heads, then the MLE is that degree of bias which is more likely to produce the observed fact of 7 heads in 10 tosses.

-