# MLE in layman terms

Could anyone explain to me in detail about maximum likelihood estimation (MLE) in layman's terms? I would like to know the underlying concept before going into mathematical derivation or equation.

-
It's unclear what kind of answer you're after. Do you know what likelihood is, for example? If not, better to find out that first. –  Glen_b Aug 19 at 12:56
In addition, I think any answer that doesn't involve math at some level will be inadequate. –  gregmacfarlane Aug 19 at 13:17

Say you have some data. Say you're willing to assume that the data comes from some distribution -- perhaps Gaussian. There are an infinite number of different Gaussians that the data could have come from: different means, different variances. MLE will pick the Gaussian that is the most likely one that your data came from.

So say you've got a data set of y = -1,3, and 7. The most likely Gaussian from which that data could have come has a mean of 3 and a variance of 16. It could have been sampled from some other Gaussian. But one with a mean of 3 and variance of 16 is more likely to have generated your data than any other.

Moving to regression: instead of the mean being a constant, the mean is a linear function of the data, as specified by the regression equation. So, say you've got data like x = 2,4,10 along with y from before. The mean of that Gaussian is now the fitted regression model $X'\hat\beta$, where $\hat\beta =[-1.9,.9]$

Moving to GLMs: replace Gaussian with some other distribution (from the exponential family). The mean is now a linear function of the data, as specified by the regression equation, transformed by the link function. So, it's $g(X'\beta)$, where $g(x) = e^x/(1+e^x)$ for logit (with binomial data).

-
"MLE will pick the Gaussian that is the most likely one, given your data." Hmmm, isn't it actually: MLE will pick the Gaussian under which your data are most likely? Which is slightly different from picking the "most likely Gaussian"... wouldn't picking the most likely Gaussian require a consideration of prior beliefs? –  Jake Westfall Aug 20 at 1:01
You'd need to clearly specify what you mean by "the most likely one" or this is likely to be misunderstood. –  Glen_b Aug 20 at 2:33
Sure, my answer is def incomplete and hand-waivey. The intention is to provide intuition before op goes to learn it formally. –  ACD Aug 20 at 4:11
@ACD I don't think this is just incomplete but provides the correct intuition. For example, I don't see any problem with not discussing special cases like the likelihood function have more than one maximum. But the difference between the distribution most likely to produce the observed data and the most likely distribution given the data is the very fundamental difference between frequentist and bayesian inference. So if you explain it like that, you are just creating a stumbling block for the future. –  Erik Aug 25 at 11:30
Sure okay, but is the more correct conceptual explanation any harder to understand than the one you've written? I don't think so. I think most of your answer is just fine, but I would just urge you, for the sake of posterity, to just slightly edit some of the phrasing to avoid discussing "the most likely Gaussian" and instead point out that the thing that we want to be "likely" (in colloquial terms) under ML is not the hypothesis but the data. I think this can be a minor but important edit to your otherwise nice answer. –  Jake Westfall Aug 25 at 16:16

Maximum Likelihood Estimation (MLE) is a technique to find the most likely function that explains observed data. I think math is necessary, but don't let it scare you!

Let's say that we have a set of points in the $x,y$ plane, and we want to know the function parameters $\beta$ and $\sigma$ that most likely fit the data (in this case we know the function because I specified it to create this example, but bear with me).

data   <- data.frame(x = runif(200, 1, 10))
data$y <- 0 + beta*data$x + rnorm(200, 0, sigma2)
plot(data$x, data$y)


In order to do a MLE, we need to make assumptions about the form of the function. In a linear model, we assume that the points follow a normal (Gaussian) probability distribution, with mean $x\beta$ and standard deviation $\sigma^2$:
$y = \mathcal{N}(x\beta, \sigma^2)$. The equation of this probability density function is: $$\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left(-\frac{(y_i-x_i\beta)^2}{2\sigma^2}\right)}$$

What we want to find is the parameters $\beta$ and $\sigma$ that maximize this probability for all points $(x_i, y_i)$. This is the "likelihood" function, $\mathcal{L}$

$$\mathcal{L} = \prod_{i=1}^n y_i = \prod_{i=1}^n \dfrac{1}{\sqrt{2\pi\sigma^2}} \exp\Big({-\dfrac{(y_i - x_i\beta)^2}{2\sigma^2}}\Big)$$ For various reasons, it's easier to use the log of the likelihood function: $$\log(\mathcal{L}) = \sum_{i = 1}^n-\frac{n}{2}\log(2\pi) -\frac{n}{2}\log(\sigma^2) - \frac{1}{2\sigma^2}(y_i - x_i\beta)^2$$

We can code this as a function in R with $\theta = (\beta,\sigma^2)$.

linear.lik <- function(theta, y, X){
n      <- nrow(X)
k      <- ncol(X)
beta   <- theta[1:k]
sigma2 <- theta[k+1]
e      <- y - X%*%beta
logl   <- -.5*n*log(2*pi)-.5*n*log(sigma2) - ( (t(e) %*% e)/ (2*sigma2) )
return(-logl)
}


This function, at different values of $\beta$ and $\sigma^2$, creates a surface.

surface <- list()
k <- 0
for(beta in seq(0, 5, 0.1)){
for(sigma2 in seq(0.1, 5, 0.1)){
k <- k + 1
logL <- linear.lik(theta = c(0, beta, sigma2), y = data$y, X = cbind(1, data$x))
surface[[k]] <- data.frame(beta = beta, sigma2 = sigma2, logL = -logL)
}
}
surface <- do.call(rbind, surface)
library(lattice)
wireframe(logL ~ beta*sigma2, surface, shade = TRUE)


As you can see, there is a maximum point somewhere on this surface. We can find parameters that specify this point with R's built-in optimization commands. This comes reasonably close to uncovering the true parameters $0, \beta = 2.7, \sigma^2 = 1.3$

linear.MLE <- nlm(f=linear.lik, p=c(1,1,1), hessian=TRUE, y=data$y, X=cbind(1, data$x))
linear.MLE$estimate ## [1] -0.04481 2.70841 1.71870  Ordinary least squares is the maximum likelihood for a linear model, so it makes sense that lm would give us the same answers. (Note that$\sigma^2$is used in determining the standard errors). summary(lm(y ~ x, data)) ## ## Call: ## lm(formula = y ~ x, data = data) ## ## Residuals: ## Min 1Q Median 3Q Max ## -3.106 -0.893 0.032 0.880 3.281 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) -0.0449 0.2334 -0.19 0.85 ## x 2.7084 0.0387 69.98 <2e-16 *** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 1.32 on 198 degrees of freedom ## Multiple R-squared: 0.961, Adjusted R-squared: 0.961 ## F-statistic: 4.9e+03 on 1 and 198 DF, p-value: <2e-16  - The MLE is the value of the parameter of interest that maximizes the probability of observing the data that you observed. In other words, it is the value of the parameter that makes the observed data most likely to have been observed. - And what if the likelihood function that is thus maximized, is, on its flip-side, the probability density function from a continuous random variable? Does the MLE still maximizes a probability? And if not, what does it do? – Alecos Papadopoulos Aug 19 at 13:29 @AlecosPapadopoulos It is my understanding that the likelihood function can be considered a probability function of the parameter, and the MLE is parameter value that maximizes that probability function. However your question suggests that there is more nuances? – Heisenberg Aug 19 at 15:27 @Heisenberg The answer treated the likelihood function as the joint probability function of the sample (for which the ML provides the max w.r.t the parameters, and so maximizes probability for any sample). And this is correct when the r.v's are discrete, but not when they are continuous, since the joint density, by construction is not a joint probability. I wouldn't characterize this as a "nuance", it is a fundamental difference between the discrete and the continuous worlds. – Alecos Papadopoulos Aug 19 at 15:46 @AlecosPapadopoulos I see. So you take issue with the use of the word "probability function" vs "density function." That is correct. – Heisenberg Aug 19 at 15:54 It is possible to say something without using (much) math, but for actual statistical applications of maximum likelihood you need mathematics. Maximum likelihood estimation is related to what philosophers call inference to the best explanation, or abduction. We use this all the time! Note, I do not say that maximum likelihood is abduction, that term is much wider, and some cases of Bayesian estimation (with an empirical prior) can probably also be seen as abduction. Some examples taken form http://plato.stanford.edu/entries/abduction/#Aca See also https://en.wikipedia.org/wiki/Abductive_reasoning 1. "You happen to know that Tim and Harry have recently had a terrible row that ended their friendship. Now someone tells you that she just saw Tim and Harry jogging together. The best explanation for this that you can think of is that they made up. You conclude that they are friends again." This because that conclusion makes the observation you try to explain more probable than under the alternative, that they are still not talking. Another example: You work in a kindergarten, and one day a child starts to walk in a strange way, and saying he broke his legs. You examine and find nothing wrong. Then you can reasonably infer that one of his parents broke their legs, since children then often actuate as described, so that is an "inference to the best explanation" and an instance of (informal) maximum likelihood. Abduction is about fonding pattern in data, and then searching for possible theories that can possibly make those patterns probable. Then choosing the possible explanation, which makes the observed pattern maximally probable, is just maximum likelihood! The prime example of abduction in science is evolution. There is no one single observation that implies evolution, but evolution makes observed patterns more probable than other explanations. Another typical example is medical diagnosos? Which possible medical condition makes the observed pattern of symptoms the most probable? Again, this is also maximum likelihood! (Or, in this case, maybe bayesian estimation is a better fit, we must take into account the prior probability of the various possible explanations). But that is a technicality, in this case we can have empirical priors which can be seen as a natural part of the statistical model, and what we call model, what we call prior is some arbitrary statistical convention. To get back to the original question about layman term explanation of MLE, here is one simple example: When my daughters where 6 and 7 years old, I asked them this. We made two urns (two shoe-boxes), in one we put 2 black balls, 8 red, in the other the numbers where switched. Then we mixed the urns, and we draw one urn randomly. Then we took at random one ball from that urn. It was red. Then I asked : From which urn do you think that red ball was drawn? After about one seconds thinking, they answered (in choir): From the one with 8 red ball! Then I asked: Why do you think so? And anes, after about one second: "Because then it is easier to draw a red ball!. That is, easier=more probable. That was maximum likelihood (it is an easy exercise to write up the probability model), and it is "inference to the best explanation", that is, abduction. - googling for "abductive reasoning and maximum likelihood estimation" gives a lot of relevant hits. – kjetil b halvorsen Aug 19 at 15:29 The maximum likelihood (ML) estimate of a parameter is the value of that parameter under which your actual observed data are most likely, relative to any other possible values of the parameter. The idea is that there are any number of "true" parameter values that could have led to your actually observed data with some non-zero (albeit perhaps small) probability. But the ML estimate gives the parameter value that would have led to your observed data with the highest probability. This must not be confused with the value of the parameter that is most likely to have actually produced your data! I like the following passage from Sober (2008, pp. 9-10) on this distinction. In this passage, we have some observed data denoted$O$and a hypothesis denoted$H$. You need to remember that "likelihood" is a technical term. The likelihood of H, Pr(O|H), and the posterior probability of H, Pr(H|O), are different quantities and they can have different values. The likelihood of H is the probability that H confers on O, not the probability that O confers on H. Suppose you hear a noise coming from the attic of your house. You consider the hypothesis that there are gremlins up there bowling. The likelihood of this hypothesis is very high, since if there are gremlins bowling in the attic, there probably will be noise. But surely you don’t think that the noise makes it very probable that there are gremlins up there bowling. In this example, Pr(O|H) is high and Pr(H|O) is low. The gremlin hypothesis has a high likelihood (in the technical sense) but a low probability. In terms of the example above, ML would favor the gremlin hypothesis. In this particular comical example, that is clearly a bad choice. But in a lot of other more realistic cases, the ML estimate might be a very reasonable one. Reference Sober, E. (2008). Evidence and Evolution: the Logic Behind the Science. Cambridge University Press. - This seems to me to be the first answer that makes this crucial point clearly and simply. Note however, that it only "would have led to your observed data with the highest probability" if your data are discrete (like binomial data), but 'would have led to your observed data with the highest joint density' if your data are continuous (like normal data). – gung Aug 20 at 2:43 Thanks @gung. I am aware of the technicality that you mention but I was slightly concerned that any discussion of "joint densities" would be a bit of a stretch for "layman's terms"... – Jake Westfall Aug 20 at 2:46 I agree w/ you, & I figured you knew about this. I just thought I'd mention it since it came up elsewhere on this thread. – gung Aug 20 at 15:16 One task in statistics is to fit a distribution function to a set of data points to generalize what's intrinsic about the data. When one is fitting a distribution a)choose an appropriate distribution b)set the movable parts (parameters), for example mean, variance, etc. When doing all this one also needs an objective, aka objective function/error function. This is required to define the meaning of "best" or "best in what sense". MLE is the procedure where this objective function is set as the maximum of the probability mass/density function of the chosen distribution. Other techniques differ how they choose this objective function. For example ordinary least squares (OLS) takes the minimum sum of squared errors. For the Gaussian case OLS and MLE are equivalent because the Gaussian distribution has that (x-m)^2 term in the density function that makes the objectives of OLS and MLE coincide. You can see that it is a squared difference term like OLS. Of course one can choose any objective function. However the intuitive meaning will not be always clear. MLE assumes that we know the distribution to start with. In other techniques, this assumption is relaxed. Especially in those cases it is more common to have a custom objective function. - Suppose you have a coin. Tossing it can give either heads or tails. But you don't know if it's a fair coin. So you toss it 1000 times. It comes up as heads 1000 times, and never as tails. Now, it's possible that this is actually a fair coin with a 50/50 chance for heads/tails, but it doesn't seem likely, does it? The chance of tossing a fair coin 1000 times and heads never coming up is$0.5^{2000}$, very small indeed. The MLE tries to help you find the best explanation in a situation like this -- when you have some result, and you want to figure out what the value of the parameter is that is most likely to give that result. Here, we have 2000 heads out of 2000 tosses -- so we would use an MLE to find out what probability of getting a head best explains getting 2000 heads out of 2000 tosses. It's the Maximum Likelihood Estimator. It estimates the parameter (here, it's a probability distribution function) that is most likely to have produced the result you are currently looking at. To finish up our example, taking the MLE would return that the probability of getting a head that best explains getting 2000 heads out of 2000 tosses is$1\$.

-

Try this link. It has pretty crisp explanation about MLE, MAP, EM. I think it covers basic idea of MLE in simple terms.

-
Welcome to our site! –  kjetil b halvorsen Nov 7 at 8:46
@kjetilbhalvorsen Thanks :) –  Nimish Kulkarni Nov 9 at 1:38