11
$\begingroup$

I have been pretty confused about maximum likelihood as expressed by my question here. But this question is not about MLE.

It occurs to me my confusion may have been because the likelihood function does not return likelihood. It returns probability density ( if I understand correctly).

So why is it called the likelihood function?

Could it have been better named?

Is the meaning of "function" in statistics different from the meaning of "function" in programming?

[Update]

I am now understanding from Glen_b that the likelihood function does return likelihood. My problem may be that I don't have an intuitive understanding of what likelihood is. Especially since it can be infinity

[Update]

Is likelihood a ratio? As per this picture in Laken's Coursera course: Improving your statistical inference Lecture 2.1

Likelihood curves

[Update]

On this post's comment, I see that "The likelihood is the joint density of the data given a parameter value".

Yet in Tim's answer it is the data that is "given" (If I interpret the vertical bar correctly)

$\endgroup$
16
  • 7
    $\begingroup$ The question seems to be based on a mistaken premise. The likelihood function doesn't "return" the density; it really does give the likelihood -- it is correctly named. While likelihood function is defined in terms of density it is not a density function. $\endgroup$
    – Glen_b
    Mar 12, 2023 at 8:03
  • 3
    $\begingroup$ I think you'll find these helpful stats.stackexchange.com/questions/2641/… and stats.stackexchange.com/questions/112451/… The analogy to computer programming doesn't help a reader understand what needs explanation, because there are literally millions of computer programs that define functions to compute likelihoods. Perhaps you're mistaking the likelihood function with the MLE, which is the estimated location of the maximum of the likelihood function? $\endgroup$
    – Sycorax
    Mar 12, 2023 at 22:02
  • 4
    $\begingroup$ The question is titled "Why is the likelihood function named as such when it does not appear to me to return likelihood?" to which the only possible answer is "the likelihood function does, in fact, return a likelihood." You acknowledge this answer in your post: "I am now understanding from Glen_b that the likelihood function does return likelihood." Is the only outstanding question you have, effectively, "What is a likelihood function?" Or something else? Can you edit your question to explain what you know, what you want to know, and where you are stuck? $\endgroup$
    – Sycorax
    Mar 12, 2023 at 23:48
  • 3
    $\begingroup$ If you're going to endlessly edit and rephrase your question it's not going to make people answer it. It's a Q&A site, where people ask questions, and others answer them, the question gets marked as solved and everybody moves on. If you're going to ignore this convention, editing the question and making the answers irrelevant, people gonna think that you don't value the time they spend on answering your question and won't care about answering them. $\endgroup$
    – Tim
    Mar 14, 2023 at 7:35
  • 3
    $\begingroup$ @Kirsten I don't believe you should change the question so dramatically after it amassed such a large number of answers. Now some of the answers don't make sense. You should create follow up questions instead. $\endgroup$
    – Firebug
    Mar 14, 2023 at 12:10

6 Answers 6

14
+50
$\begingroup$

The likelihood function parametrized by a parameter $\theta$ in statistics is defined as

$$ \mathcal{L}(\theta \mid x) = f_{\theta}(x) $$

where $f_{\theta}$ is the probability density or mass function with parameter $\theta$ and $x$ is the data.

If for some data $x$ you evaluate the function for the parameter $\theta$ we call the result the “likelihood” of $\theta$. There's no other “likelihood”, because this is how we define it.

Using a code example, Gaussian likelihood could be implemented in Python as below.

import numpy as np
from scipy.stats import norm

def likelihood(loc, scale):
   return np.prod(norm.pdf(X, loc=loc, scale=scale))

where norm.pdf is the Gaussian probability density function and np.prod calculates the product of the probability density values returned for each value in the array X. Notice that X is not an argument of the function, it is fixed, and the only arguments of the likelihood function are the parameters (here loc and scale). What the function returns, is the likelihood for the parameters passed as arguments. If you maximize this function, the result would be a maximum likelihood estimate for the parameters.

Could it have been better named?

Maybe, but it wasn't. But the same applies to all the other names in mathematics or names in general. For example, “isomorphism” or “monoid” may also not be a great names, but this is how we called them.

$\endgroup$
12
  • $\begingroup$ Thanks Tim. In my world (coding) I have never seen a function evaluated for a parameter. Parameters are inputs the function evaluation is an output. $\endgroup$
    – Kirsten
    Mar 12, 2023 at 20:05
  • 3
    $\begingroup$ @Kirsten I'm not sure what you mean. When you call f(x) in any programming language it evaluates the function f with the x argument. Same here, you call the likelihood function $\mathcal{L}$ with $\theta$ as an argument. $\endgroup$
    – Tim
    Mar 12, 2023 at 20:28
  • 2
    $\begingroup$ @Kirsten I don't know what you mean by those words. MLE is the maximum of the likelihood function with respect to $\theta$. It is not "given", it is something you find by maximizing the function. $\endgroup$
    – Tim
    Mar 12, 2023 at 22:56
  • 3
    $\begingroup$ @Kirsten Before going deeper you should probably familiarize yourself better with some basic concepts in probability theory and statistics, otherwise it might be hard to understand more advanced concepts stats.stackexchange.com/questions/4220/… $\endgroup$
    – Tim
    Mar 13, 2023 at 10:50
  • 1
    $\begingroup$ @Kirsten the definition and usage of likelihood are the same regardless if it is used with discrete or continuous variables. $\endgroup$
    – Tim
    Mar 13, 2023 at 14:51
12
$\begingroup$

There have been numerous responses including some to your very posts earlier and the present one too.

It should be reiterated that $\mathcal L(\theta\mid \mathbf x) $ or $\ell_\mathbf x(\theta)$ (to emphasize what the argument here is) even though has the same functional form as the corresponding density function of the distribution, in the former, what varies is the value of $\theta$ over the parameter space given the observed sample value $\mathbf x. $ As has been noted earlier too, $\ell_\mathbf x(\theta) $ , as a function of $\theta$, doesn't have to be a legit density function.

It returns likelihood as codified in the Likelihood Principle which basically says that two likelihood functions have same information about $\theta$ if they are proportional to one another or stating in more formal terms, if $E:=(\mathbf X, \theta,\{f_\theta(\mathbf x) \}),$ is the experiment, then any conclusion about $\theta$ (measured by the evidence function $\textrm{EV}(E,\mathbf x) $) should depend on $E,~\mathbf x$ only via $\ell_\theta(\mathbf x).$ So, if $\ell_\mathbf x(\theta)=C(\mathbf x, \mathbf y) \ell_\mathbf y, ~\forall\theta\in\Theta,$ ($C(\mathbf x, \mathbf y) $ would be independent of $\theta$) for two sample values $\mathbf x, \mathbf y, $ then the inference on $\theta$ based on either of the sample observation is equivalent.

Thus likelihood functions enable us to assess the "plausibility" of $\theta:$ if $\ell_\mathbf x(\theta_2) =c\ell_\mathbf x(\theta_1),$ then it is likely that $\theta_2$ is $c ~(c>0) $ (say) times as plausible as $\theta_1.$ By the likelihood principle, for the sample value $\mathbf y, ~\ell_\mathbf y(\theta_2) =c\ell_\mathbf y(\theta_1)$ and likely $\theta_2$ is $c$ times as plausible as $\theta_1$ irrespective of whether $\mathbf x$ or $\mathbf y$ is the realized observation of the sample.


Since the confusion still lingers in light of the likelihoods and priors, let me quote verbatim from $\rm [II]$ (to articulate the relationship of Bayes' theorem and likelihood function; emphasis mine):

[...] given the data $\mathbf y, ~p(\mathbf y\mid\boldsymbol\theta) $ in $$p(\boldsymbol\theta\mid\mathbf y) =cp(\mathbf y\mid\boldsymbol\theta) p(\boldsymbol\theta)$$ may be regarded as a function not of $\bf y$ but of $\boldsymbol\theta.$ When so regarded, following Fisher ($1922$), it is called the likelihood function of $\boldsymbol\theta$ for given $\mathbf y$ and can be written $l(\boldsymbol\theta\mid\mathbf y). $ We can thus write Bayes' formula as $$ p(\boldsymbol\theta\mid\mathbf y) =l(\boldsymbol\theta\mid\mathbf y)p(\boldsymbol\theta).$$ In other words, Bayes' theorem tells us that the probability distribution for $\boldsymbol \theta$ posterior to the data $\bf y$ is proportional to the product of the distribution for $\boldsymbol\theta$ prior to the data and the likelihood for $\boldsymbol\theta$ given $\mathbf y. $


Reference:

$\rm [I]$ Statistical Inference, George Casella, Roger L. Berger, Wadsworth, $2002, $ sec. $6.3.1, $ pp. $290-291, ~293-294.$

$\rm [II]$ Bayesian Inference in Statistical Analysis, George E. P. Box, George C. Tiao, Wiley Classics, $1992, $ sec. $1.2.1, $ pp. $10-11.$

$\endgroup$
9
  • $\begingroup$ Thank you. I am having trouble processing the second paragraph. Could you add some punctuation please? $\endgroup$
    – Kirsten
    Mar 13, 2023 at 2:58
  • 1
    $\begingroup$ The second para basically ruminates that $\ell_\mathbf x(\theta) $ is a function of $\theta$ with its domain as the parameter space. While it resembles the density function, it is different as the domains are not the same. $\endgroup$ Mar 13, 2023 at 3:05
  • 3
    $\begingroup$ @Kirsten, here is how you break down the really long sentence into three shorter ones: It returns likelihood as codified in the Likelihood Principle. The principle says that two likelihood functions have same information about $\theta$ if they are proportional to one another. More formally, ... $\endgroup$ Mar 14, 2023 at 7:49
  • 1
    $\begingroup$ (+1) for the informative and precise answer! @Kuku the change you are referring to is only a matter of different notations for the same thing. $\endgroup$
    – utobi
    Mar 27, 2023 at 20:31
  • 1
    $\begingroup$ @Kuku as explained in the answer, the likelihood is a function of the parameter no matter what is the data. Thus, denoting the likelihood by $p(y|\theta)$ or by $l(\theta|y)$, being you bayesian or frequentist, doesn't really matter. $\endgroup$
    – utobi
    Mar 28, 2023 at 9:08
5
$\begingroup$

As stated by many others: the likelihood function $\mathcal L_y$ is the probability density function $f_\theta$ of the observed data $y$, but viewed as a function of the (unknown) parameter $\theta$, i.e., $\mathcal L_y(\theta) = f_\theta(y)$.

To provide some intuition, let's consider discrete data1 and continuous data separately:

  • For discrete data we have $\mathcal L_y(\theta) = \mathbb P_\theta(Y = y)$, i.e., the likelihood function is the probability of the observed data viewed as a function of the parameter $\theta$.
  • For continuous data let's assume that in practice we can only measure, and thus observe, data with limited accuracy. Then, observing $Y = y$ (say, for the sake of simplicity, for real-valued $Y$) can be understood as indicating that $Y$ took a value in a small interval $[y - \delta, y + \delta]$. For the probability of the observed datum $y$ we then have $$ \mathbb P_\theta(Y \in [y - \delta, y + \delta]) = \mathbb P_\theta(y - \delta \leq Y \leq y + \delta) = \int_{y - \delta}^{y + \delta} f_\theta(y) \,\mathrm d y. $$ Now, the approximation $$\int_{y - \delta}^{y + \delta} f_\theta(y) \,\mathrm d y \approx f_\theta(y) \cdot \left[\left(y + \delta\right) -\left(y - \delta\right)\right] = 2\delta \cdot f_\theta(y) $$ suggests that the probability of the observed datum $y$ is approximately proportional to $f_\theta(y)$.
    In this sense $\mathcal L_y(\theta) = f_\theta(y) \overset{\text{approx.}}\propto \mathbb P_\theta(Y \in [y - \delta, y + \delta])$ still indicates how "likely" a paramter value $\theta$ is for the observed datum $y$.

1 here the probability mass function referred to in some of the other answers can be seen as probability density function w.r.t. the counting measure.


Reference

Held, L., & Sabanés Bové, D. (2020). Likelihood and Bayesian inference: With applications in biology and medicine (Second edition). Springer.

$\endgroup$
3
$\begingroup$

Likelihood is a slippery concept. The likelihood function, $L(t | w)$, expresses how probable the data $t_n$ are in relation to the model function $y(x,w)$.

The uncertainty in the empirical measurements enters the modeling through the misfits $ε_n$; one misfit for each data point. We introduce a misfit probability distribution, usually a Gaussian.

The values of the misfits have no specific meaning. But the introduction of the misfit probability distribution has changed our state of knowledge in a rather fundamental way. As data scientists, we prefer models with small misfits and look suspiciously at large outliers. The misfits $ε_n$ are now subjected to an overarching probability distribution $p(ε_n)$, which “connects” the previously unrelated individual misfit values by their probabilities. Conceptually, this metamorphosis is a big step.

The likelihood value is the product of the $p(ε_n)$.

The likelihood function is the generalization of the likelihood value. The likelihood function is the product of the misfit probability densities but with the dependency on the coefficient $w$ of the model function $y(x,w)$ taken into account.

Technically, the likelihood function is not a normalized probability distribution over $w$. It is a factor in Bayes' theorem, and does not necessarily integrate to unity. $$\int L(t | w) dw \ne 1.$$ However, it is a normalized probability distribution when integrated over the data values $t$ $$\int L(t | w) dt = 1.$$ Which makes it confusing.


The above texts are excerpts from my new tutorial book "What is your model?, a Bayesian tutorial". It is a short self-study book on Bayesian data analysis. One can read 80% for free on Amazon under the Look Inside feature, thereby avoiding bad buys.

$\endgroup$
0
$\begingroup$

I am not exactly sure I fully understand the question but I suspect it might come down to understanding what the likelihood function is measuring exactly.

Let $X$ be a (absolutely) continuous random variable and let $f_X(\cdot)$ denote its PDF, also known as the likelihood function. The value of $f_X(a)$ is not equal to $P(X = a)$, indeed since $X$ is continuous it means that $P(X=a)=0$ for every real number $a$.

So then what is the interpretation of $f_X(a)$? It denotes the probability of landing inside an "infinitesimal neighborhood of $a$". To make this more precise, let $\varepsilon > 0$, then we can ask $P(|X-a|\leq \varepsilon)$. We thicken the point $a$ by a width of $2\varepsilon$. Then $f_X(a)$ is the limit of $\frac{P(|X-a|\leq \varepsilon)}{2\varepsilon}$ as we shrink $\varepsilon$.

For instance, if $f_X(0) = 2$, this means that if we drew a tiny neighborhood of thickness $\ell$, then the probability of landing inside that neighborhood is approximately equal to $2\ell$. This explains why likelihood can exceed the value of $1$ whereas the probability does not.

$\endgroup$
2
  • 5
    $\begingroup$ However the likelihood function is not a pdf stats.stackexchange.com/questions/31238/… $\endgroup$
    – Kirsten
    Mar 12, 2023 at 20:26
  • $\begingroup$ @Kirsten Yes, you are correct, however, the objection is semantics. One can select "PDF" and replace it with "likelihood", in appropriate places, and my answer would make sense. $\endgroup$ Mar 13, 2023 at 5:48
0
$\begingroup$

After studying Tim's answer and realising that code can give me the best short sentences. I realised that I could understand best via code.

The following code outputs the likelihood of a particular mean and variance combination, for a normal distribution and some generated data.

import numpy as np
from scipy.stats import norm
RNG = np.random.default_rng(seed=0)

def likelihood(pMean, pStdDev):
   # construct the probability density function for the model this particular likelihood function will use, using the supplied parameters.
   gausianPDF = norm.pdf(X, loc=pMean, scale=pStdDev)

   # construct the product of the PDF for each data point
   productOfPDFs = np.prod(gausianPDF)

   # that is the answer.
   return productOfPDFs

X = RNG.choice(20,30)
mean = np.mean(X)
stdDev = np.std(X)

lh = likelihood(mean, stdDev)
print('The likelihood of mean ',mean,'and stdDev',stdDev, 'for the data is ',lh)    

When I ran it I got Output

This is for a normal distribution.

I see that the likelihood is the product of a probability density function. The particular probability density function is based on a distribution model and some data distributed according to the model.

To get the likelihood of some parameters we evaluate the likelihood function for the particular distribution and data, using the parameters we are interested in.

Thus,

The likelihood of the parameters of a model is given by the joint probability density of the data, as modelled using those parameters.

$\endgroup$
2
  • 4
    $\begingroup$ Is this intended as an answer, an edit to your question, a comment, or something else? Answers really shouldn't be code-only: at the very least, explain what you intend the code to do and demonstrate that it does, indeed, answer the question. $\endgroup$
    – whuber
    Mar 15, 2023 at 23:24
  • $\begingroup$ @whuber Thank you. I have updated my answer (again) $\endgroup$
    – Kirsten
    Mar 16, 2023 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.