What is likelihood actually?

Question

I have been pretty confused about maximum likelihood as expressed by my question here. But this question is not about MLE.

It occurs to me my confusion may have been because the likelihood function does not return likelihood. It returns probability density ( if I understand correctly).

So why is it called the likelihood function?

Could it have been better named?

Is the meaning of "function" in statistics different from the meaning of "function" in programming?

[Update]

I am now understanding from Glen_b that the likelihood function does return likelihood. My problem may be that I don't have an intuitive understanding of what likelihood is. Especially since it can be infinity

[Update]

Is likelihood a ratio? As per this picture in Laken's Coursera course: Improving your statistical inference Lecture 2.1

[Update]

On this post's comment, I see that "The likelihood is the joint density of the data given a parameter value".

Yet in Tim's answer it is the data that is "given" (If I interpret the vertical bar correctly)

Answer 1

The likelihood function parametrized by a parameter $\theta$ in statistics is defined as

$$ \mathcal{L}(\theta \mid x) = f_{\theta}(x) $$

where $f_{\theta}$ is the probability density or mass function with parameter $\theta$ and $x$ is the data.

If for some data $x$ you evaluate the function for the parameter $\theta$ we call the result the “likelihood” of $\theta$. There's no other “likelihood”, because this is how we define it.

Using a code example, Gaussian likelihood could be implemented in Python as below.

import numpy as np
from scipy.stats import norm

def likelihood(loc, scale):
   return np.prod(norm.pdf(X, loc=loc, scale=scale))

where norm.pdf is the Gaussian probability density function and np.prod calculates the product of the probability density values returned for each value in the array X. Notice that X is not an argument of the function, it is fixed, and the only arguments of the likelihood function are the parameters (here loc and scale). What the function returns, is the likelihood for the parameters passed as arguments. If you maximize this function, the result would be a maximum likelihood estimate for the parameters.

Could it have been better named?

Maybe, but it wasn't. But the same applies to all the other names in mathematics or names in general. For example, “isomorphism” or “monoid” may also not be a great names, but this is how we called them.

Answer 2

There have been numerous responses including some to your very posts earlier and the present one too.

It should be reiterated that $\mathcal L(\theta\mid \mathbf x) $ or $\ell_\mathbf x(\theta)$ (to emphasize what the argument here is) even though has the same functional form as the corresponding density function of the distribution, in the former, what varies is the value of $\theta$ over the parameter space given the observed sample value $\mathbf x. $ As has been noted earlier too, $\ell_\mathbf x(\theta) $ , as a function of $\theta$, doesn't have to be a legit density function.

It returns likelihood as codified in the Likelihood Principle which basically says that two likelihood functions have same information about $\theta$ if they are proportional to one another or stating in more formal terms, if $E:=(\mathbf X, \theta,\{f_\theta(\mathbf x) \}),$ is the experiment, then any conclusion about $\theta$ (measured by the evidence function $\textrm{EV}(E,\mathbf x) $) should depend on $E,~\mathbf x$ only via $\ell_\theta(\mathbf x).$ So, if $\ell_\mathbf x(\theta)=C(\mathbf x, \mathbf y) \ell_\mathbf y, ~\forall\theta\in\Theta,$ ($C(\mathbf x, \mathbf y) $ would be independent of $\theta$) for two sample values $\mathbf x, \mathbf y, $ then the inference on $\theta$ based on either of the sample observation is equivalent.

Thus likelihood functions enable us to assess the "plausibility" of $\theta:$ if $\ell_\mathbf x(\theta_2) =c\ell_\mathbf x(\theta_1),$ then it is likely that $\theta_2$ is $c ~(c>0) $ (say) times as plausible as $\theta_1.$ By the likelihood principle, for the sample value $\mathbf y, ~\ell_\mathbf y(\theta_2) =c\ell_\mathbf y(\theta_1)$ and likely $\theta_2$ is $c$ times as plausible as $\theta_1$ irrespective of whether $\mathbf x$ or $\mathbf y$ is the realized observation of the sample.

---

Since the confusion still lingers in light of the likelihoods and priors, let me quote verbatim from $\rm [II]$ (to articulate the relationship of Bayes' theorem and likelihood function; emphasis mine):

[...] given the data $\mathbf y, ~p(\mathbf y\mid\boldsymbol\theta) $ in $$p(\boldsymbol\theta\mid\mathbf y) =cp(\mathbf y\mid\boldsymbol\theta) p(\boldsymbol\theta)$$ may be regarded as a function not of $\bf y$ but of $\boldsymbol\theta.$ When so regarded, following Fisher ($1922$), it is called the likelihood function of $\boldsymbol\theta$ for given $\mathbf y$ and can be written $l(\boldsymbol\theta\mid\mathbf y). $ We can thus write Bayes' formula as $$ p(\boldsymbol\theta\mid\mathbf y) =l(\boldsymbol\theta\mid\mathbf y)p(\boldsymbol\theta).$$ In other words, Bayes' theorem tells us that the probability distribution for $\boldsymbol \theta$ posterior to the data $\bf y$ is proportional to the product of the distribution for $\boldsymbol\theta$ prior to the data and the likelihood for $\boldsymbol\theta$ given $\mathbf y. $

---

Reference:

$\rm [I]$ Statistical Inference, George Casella, Roger L. Berger, Wadsworth, $2002, $ sec. $6.3.1, $ pp. $290-291, ~293-294.$

$\rm [II]$ Bayesian Inference in Statistical Analysis, George E. P. Box, George C. Tiao, Wiley Classics, $1992, $ sec. $1.2.1, $ pp. $10-11.$

Answer 3

As stated by many others: the likelihood function $\mathcal L_y$ is the probability density function $f_\theta$ of the observed data $y$, but viewed as a function of the (unknown) parameter $\theta$, i.e., $\mathcal L_y(\theta) = f_\theta(y)$.

To provide some intuition, let's consider discrete data¹ and continuous data separately:

• For discrete data we have $\mathcal L_y(\theta) = \mathbb P_\theta(Y = y)$, i.e., the likelihood function is the probability of the observed data viewed as a function of the parameter $\theta$.
• For continuous data let's assume that in practice we can only measure, and thus observe, data with limited accuracy. Then, observing $Y = y$ (say, for the sake of simplicity, for real-valued $Y$) can be understood as indicating that $Y$ took a value in a small interval $[y - \delta, y + \delta]$. For the probability of the observed datum $y$ we then have $$ \mathbb P_\theta(Y \in [y - \delta, y + \delta]) = \mathbb P_\theta(y - \delta \leq Y \leq y + \delta) = \int_{y - \delta}^{y + \delta} f_\theta(y) \,\mathrm d y. $$ Now, the approximation $$\int_{y - \delta}^{y + \delta} f_\theta(y) \,\mathrm d y \approx f_\theta(y) \cdot \left[\left(y + \delta\right) -\left(y - \delta\right)\right] = 2\delta \cdot f_\theta(y) $$ suggests that the probability of the observed datum $y$ is approximately proportional to $f_\theta(y)$.
  In this sense $\mathcal L_y(\theta) = f_\theta(y) \overset{\text{approx.}}\propto \mathbb P_\theta(Y \in [y - \delta, y + \delta])$ still indicates how "likely" a paramter value $\theta$ is for the observed datum $y$.

---

¹ here the probability mass function referred to in some of the other answers can be seen as probability density function w.r.t. the counting measure.

---

Reference

Held, L., & Sabanés Bové, D. (2020). Likelihood and Bayesian inference: With applications in biology and medicine (Second edition). Springer.

Answer 4

Likelihood is a slippery concept. The likelihood function, $L(t | w)$, expresses how probable the data $t_n$ are in relation to the model function $y(x,w)$.

The uncertainty in the empirical measurements enters the modeling through the misfits $ε_n$; one misfit for each data point. We introduce a misfit probability distribution, usually a Gaussian.

The values of the misfits have no specific meaning. But the introduction of the misfit probability distribution has changed our state of knowledge in a rather fundamental way. As data scientists, we prefer models with small misfits and look suspiciously at large outliers. The misfits $ε_n$ are now subjected to an overarching probability distribution $p(ε_n)$, which “connects” the previously unrelated individual misfit values by their probabilities. Conceptually, this metamorphosis is a big step.

The likelihood value is the product of the $p(ε_n)$.

The likelihood function is the generalization of the likelihood value. The likelihood function is the product of the misfit probability densities but with the dependency on the coefficient $w$ of the model function $y(x,w)$ taken into account.

Technically, the likelihood function is not a normalized probability distribution over $w$. It is a factor in Bayes' theorem, and does not necessarily integrate to unity. $$\int L(t | w) dw \ne 1.$$ However, it is a normalized probability distribution when integrated over the data values $t$ $$\int L(t | w) dt = 1.$$ Which makes it confusing.

---

The above texts are excerpts from my new tutorial book "What is your model?, a Bayesian tutorial". It is a short self-study book on Bayesian data analysis. One can read 80% for free on Amazon under the Look Inside feature, thereby avoiding bad buys.

Answer 5

I am not exactly sure I fully understand the question but I suspect it might come down to understanding what the likelihood function is measuring exactly.

Let $X$ be a (absolutely) continuous random variable and let $f_X(\cdot)$ denote its PDF, also known as the likelihood function. The value of $f_X(a)$ is not equal to $P(X = a)$, indeed since $X$ is continuous it means that $P(X=a)=0$ for every real number $a$.

So then what is the interpretation of $f_X(a)$? It denotes the probability of landing inside an "infinitesimal neighborhood of $a$". To make this more precise, let $\varepsilon > 0$, then we can ask $P(|X-a|\leq \varepsilon)$. We thicken the point $a$ by a width of $2\varepsilon$. Then $f_X(a)$ is the limit of $\frac{P(|X-a|\leq \varepsilon)}{2\varepsilon}$ as we shrink $\varepsilon$.

For instance, if $f_X(0) = 2$, this means that if we drew a tiny neighborhood of thickness $\ell$, then the probability of landing inside that neighborhood is approximately equal to $2\ell$. This explains why likelihood can exceed the value of $1$ whereas the probability does not.

Answer 6

After studying Tim's answer and realising that code can give me the best short sentences. I realised that I could understand best via code.

The following code outputs the likelihood of a particular mean and variance combination, for a normal distribution and some generated data.

import numpy as np
from scipy.stats import norm
RNG = np.random.default_rng(seed=0)

def likelihood(pMean, pStdDev):
   # construct the probability density function for the model this particular likelihood function will use, using the supplied parameters.
   gausianPDF = norm.pdf(X, loc=pMean, scale=pStdDev)

   # construct the product of the PDF for each data point
   productOfPDFs = np.prod(gausianPDF)

   # that is the answer.
   return productOfPDFs

X = RNG.choice(20,30)
mean = np.mean(X)
stdDev = np.std(X)

lh = likelihood(mean, stdDev)
print('The likelihood of mean ',mean,'and stdDev',stdDev, 'for the data is ',lh)    

When I ran it I got

This is for a normal distribution.

I see that the likelihood is the product of a probability density function. The particular probability density function is based on a distribution model and some data distributed according to the model.

To get the likelihood of some parameters we evaluate the likelihood function for the particular distribution and data, using the parameters we are interested in.

Thus,

The likelihood of the parameters of a model is given by the joint probability density of the data, as modelled using those parameters.