# What is the difference between “likelihood” and “probability”?

In non-technical parlance, "likelihood" is usually a synonym for "probability," but in statistical usage there is a clear distinction in perspective: the number that is the probability of some observed outcomes given a set of parameter values is regarded as the likelihood of the set of parameter values given the observed outcomes.

Can someone give a more down-to-earth description of what this means? In addition, some examples of how "probability" and "likelihood" disagree would be nice.

-
Great question. I would add "odds" and "chance" in there too :) –  Neil McGuigan Sep 14 '10 at 5:28
I think you should take a look at this question stats.stackexchange.com/questions/665/… because Likelihood is for statistic purpose and probability for probability. –  robin girard Sep 14 '10 at 6:04
Wow, these are some really good answers. So a big thanks for that! Some point soon, I'll pick one I particularly like as the "accepted" answer (although there are several that I think are equally deserved). –  Douglas S. Stones Sep 15 '10 at 1:13
Also note that the "likelihood ratio" is actually a "probability ratio" since is a function of the observations. –  JohnRos Nov 2 '11 at 10:29

The answer depends on whether you are dealing with discrete or continuous random variables. So, I will split my answer accordingly. I will assume that you want some technical details and not necessarily an explanation in plain english. If my assumption is not correct please let me know and I will revise my answer.

Discrete Random Variables

Suppose that you have a stochastic process that takes discrete values (e.g., outcomes of tossing a coin 10 times, number of customers who arrive at a store in 10 minutes etc). In such cases, we can calculate the probability of observing a particular set of outcomes by making suitable assumptions about the underlying stochastic process (e.g., probability of coin landing heads is $p$ and that coin tosses are independent).

Denote the observed outcomes by $O$ and the set of parameters that describe the stochastic process as $\theta$. Thus, when we speak of probability we want to calculate $P(O|\theta)$. In other words, given specific values for $\theta$, $P(O|\theta)$ is the probability that we would observe the outcomes represented by $O$.

However, when we model a real life stochastic process, we often do not know $\theta$. We simply observe $O$ and the goal then is to arrive at an estimate for $\theta$ that would be a plausible choice given the observed outcomes $O$. We know that given a value of $\theta$ the probability of observing $O$ is $P(O|\theta)$. Thus, a 'natural' estimation process is to choose that value of $\theta$ that would maximize the probability that we would actually observe $O$. In other words, we find the parameter values $\theta$ that maximize the following function:

$L(\theta|O) = P(O|\theta)$

$L(\theta|O)$ is called as the likelihood function. Notice that by definition the likelihood function is conditioned on the observed $O$ and that it is a function of the unknown parameters $\theta$.

Continuous Random Variables

In the continuous case the situation is similar with one important difference. We can no longer talk about the probability that we observed $O$ given $\theta$ as in the continuous case $P(O|\theta) = 0$. Without getting into technicalities, the basic idea is as follows:

Denote the probability density function (pdf) associated with the outcomes $O$ as: $f(O|\theta)$. Thus, in the continuous case we estimate $\theta$ given observed outcomes $O$ by maximizing the following function:

$L(\theta|O) = f(O|\theta)$

In this situation, we cannot technically assert that we are finding the parameter value that maximizes the probability that we observe $O$ as we maximize the pdf associated with the observed outcomes $O$.

-
The distinction between discrete and continuous variables disappears from the point of view of measure theory. –  whuber Sep 14 '10 at 15:48
@whuber yes but an answer using measure theory is not that accessible to everyone. –  user28 Sep 14 '10 at 20:09
@Srikant: Agreed. The comment was for the benefit of the OP, who is a mathematician (but perhaps not a statistician) to avoid being misled into thinking there is something fundamental about the distinction. –  whuber Sep 14 '10 at 20:36
You can interpret a continuous density the same as the discrete case if $O$ is replaced by $dO$, in the sense that if we ask for $Pr(O\in(O',O'+dO') |\theta)$ (i.e. probability that the data $O$ is contained in an infinintesimal region about $O'$) and the answer is $f(O'|\theta)dO'$ (the $dO'$ makes this clear that we are calculating the area of an infinintesimaly thin "bin" of a histogram). –  probabilityislogic Jan 28 '11 at 13:40

I'll give you the perspective from the view of Likelihood Theory which originated with Fisher -- and is the basis for the statistical definition in the cited Wikipedia article.

Suppose you have random variates $X$ which arise from a parameterized distribution $F(X; \theta)$, where $\theta$ is the parameter characterizing $F$. Then the probability of $X = x$ would be: $P(X = x) = F(x; \theta)$, with known $\theta$.

More often, you have data $X$ and $\theta$ is unknown. Given the assumed model $F$, the likelihood is defined as the probability of observed data as a function of $\theta$: $L(\theta) = P(\theta; X = x)$. Note that $X$ is known, but $\theta$ is unknown; in fact the motivation for defining the likelihood is to determine the parameter of the distribution.

Although it seems like we have simply re-written the probability function, a key consequences of this is that the likelihood function does not obey the laws of probability (for example, it's not bound to the [0, 1] interval). However, the likelihood function is proportional to the probabiilty of the observed data.

This concept of likelihood actually leads to a different school of thought, "likelihoodists" (distinct from frequentist and bayesian) and you can google to search for all the various historical debates. The cornerstone is the Likelihood Principle which essentially says that we can perform inference directly from the likelihood function (neither Bayesians nor frequentists accept this since it is not probability based inference). These days a lot of what is taught as "frequentist" in schools is actually an amalgam of frequentist and likelihood thinking.

For deeper insight, a nice start and historical reference is Edwards' Likelihood. For a modern take, I'd recommend Richard Royall's wonderful monograph, Statistical Evidence: A Likelihood Paradigm.

-
Interesting answer, I actually thought that the "likelihood school" was basically the "frequentists who don't design samples school", while the "design school" was the rest of the frequentists. I actually find it hard myself to say which "school" I am, as I have a bit of knowledge from every school. The "Probability as extended logic" school is my favourite (duh), but I don't have enough practical experience in applying it to real problems to be dogmatic about it. –  probabilityislogic Jan 28 '11 at 13:53

This is the kind of question that just about everybody is going to answer and I would expect all the answers to be good. But you're a mathematician, Douglas, so let me offer a mathematical reply. A statistical model has to connect two distinct conceptual entities: data, which are elements $x$ of some set (such as a vector space), and a possible quantitative model of the data behavior. Models are usually represented by points $\theta$ on a finite dimensional manifold, a manifold with boundary, or a function space (the latter is termed a "non-parametric" problem). The data $x$ are connected to the possible models $\theta$ by means of a function $\Lambda(x, \theta)$. For any given $\theta$, $\Lambda(x, \theta)$ is intended to be the probability (or probability density) of $x$. For any given $x$, on the other hand, $\Lambda(x, \theta)$ can be viewed as a function of $\theta$ and is usually assumed to have certain nice properties, such as being continuously second differentiable. The intention to view $\Lambda$ in this way and to invoke these assumptions is announced by calling $\Lambda$ the "likelihood." It's quite like the distinction between variables and parameters in a differential equation: sometimes we want to study the solution (i.e., we focus on the variables as the argument) and sometimes we want to study how the solution varies with the parameters. The main distinction is that in statistics we rarely need to study the simultaneous variation of both sets of arguments; there is no statistical object that naturally corresponds to changing both the data $x$ and the model parameters $\theta$. That's why you hear more about this dichotomy than you would in analogous mathematical settings.

-
+1, what a cool answer. Analogy with differential equations seems very apropriate. –  mpiktas Mar 5 '12 at 20:15

I'll try and minimise the mathematics in my explanation as there are some good mathematical explanations already.

As Robin Girand points out the difference between probability and likelihood is closely related to the difference between probability and statistics. In a sense probability and statistics concern themselves with problems that are opposite or inverse to one another.

Consider a coin toss. (My answer will be similar to Example 1 on Wikipedia.) If we know the coin is fair (p=0.5) a typical probability question is: What is the probability of getting two heads in a row. The answer is P(HH) = P(H)*P(H) = 0.5*0.5 = 0.25.

A typical statistical question is: Is the coin fair? To answer this we need to ask: To what extent does our sample support the our hypothesis that P(H) = P(T) = 0.5?

The first point to note is that the direction of the question has reversed. In probability we start with an assumed parameter (P(head)) and estimate the probability of a given sample (two heads in a row). In statistics we start with the observation (two heads in a row) and make INFERENCE about our parameter (p = P(H) = 1- P(T) = 1 - q).

Example 1 on Wikipedia shows us that the maximum likelihood estimate of P(H) after 2 heads in a row is p_MLE = 1. But the data in no way rule out the the true parameter value p(H) = 0.5 (let's not concern ourselves with the details at the moment). Indeed only very small values of p(H) and particularly p(H)=0 can be reasonably eliminated after n = 2 (two throws of the coin). After the third throw we can now eliminate the possibility that P(H) = 1.0 (i.e. it is not a two-headed coin), but most values in between can be reasonably supported by the data. (An exact binomial 95% confidence interval for p(H) is 0.094 to 0.992.

After 100 coin tosses and (say) 70 heads, we now have a reasonable basis for the suspicion that the coin is not in fact fair. An exact 95% CI on p(H) is now 0.600 to 0.787 and the probability of observing a result as extreme as 70 or more heads (or tails) from 100 tosses given p(H) = 0.5 is 0.0000785.

Although I have not explicitly used likelihood calculations this example captures the concept of likelihood: Likelihood is a measure of the extent to which a sample provides support for particular values of a parameter in a parametric model.

-
I'm having trouble doing hyperlinks. Can someone offer some pointers please? –  Thylacoleo Sep 14 '10 at 8:47
In the editor you will see icons for bold, italic and right next to that you would see an icon that looks like a 'linked-chain'. Click that icon and paste the hyperlink in the dialog box. In the main text of the question you will something along the following lines: "[link text][1]" Replace link text with what you want the link to say. You can do links in comments also like so: [link text] (url to link). Remove the space between the closing ']' and the opening '(' when you actually use this methid in the comments. –  user28 Sep 14 '10 at 14:10
Thanks Srikant, that was great! –  Thylacoleo Sep 15 '10 at 0:22

If I have a fair coin (parameter value) then the probability that it will come up heads is 0.5. If I flip a coin 100 times and it comes up heads 52 times then it has a high likelihood of being fair (the numeric value of likelihood potentially taking a number of forms).

-

Suppose you have a coin with probability $p$ to land heads and $(1-p)$ to land tails. Let $x=1$ indicate heads and $x=0$ indicate tails. Define $f$ as follows

$$f(x,p)=p^x (1-p)^{1-x}$$

$f(x,2/3)$ is probability of x given $p=2/3$, $f(1,p)$ is likelihood of $p$ given $x=1$. Basically likelihood vs. probability tells you which parameter of density is considered to be the variable

-
Nice complement to the theoretical definitions used above! –  Frank Meulenaar Sep 17 '11 at 10:47
shouldn't that be "f(x,2/3) is the probability of x given p=2/3"?? If not could you explain why? –  qi5d02lx Oct 20 '11 at 23:41
good catch, fixed –  Yaroslav Bulatov Nov 2 '11 at 10:00