17
$\begingroup$

According to the Wikipedia article Likelihood function, the likelihood function is defined as:

$$ \mathcal{L}(\theta|x)=P(x|\theta), $$

with parameters $\theta$ and observed data $x$. This equals $p(x|\theta)$ or $p_\theta(x)$ depending on notation and whether $\theta$ is treated as random variable or fixed value.

The notation $\mathcal{L}(\theta|x)$ seems like an unnecessary abstraction to me. Is there any benefit to using $\mathcal{L}(\theta|x)$, or could one equivalently use $P(x|\theta)$? Why was $\mathcal{L}(\theta|x)$ introduced?

$\endgroup$
3
  • 5
    $\begingroup$ In this context, it reminds us that the likelihood function is a function of $\theta$ with the data $x$ fixed. On the other hand, the joint distribution is a function of the data $x$ given $\theta$. $\endgroup$
    – knrumsey
    Jun 11 '17 at 22:25
  • 1
    $\begingroup$ @BigAgnes Thanks. Aren't observed variables fixed by definition, though? I'm also confused why we can call $p(x|\theta)$ a joint distribution. Isn't it a scalar since both $x$ and $\theta$ are fixed (assuming a Frequentist approach where $\theta$ is not a random variable). $\endgroup$
    – danijar
    Jun 11 '17 at 22:34
  • 1
    $\begingroup$ Closely related: stats.stackexchange.com/questions/224037/… $\endgroup$
    – Tim
    Jun 12 '17 at 7:50
17
$\begingroup$

Likelihood is a function of $\theta$, given $x$, while $P$ is a function of $x$, given $\theta$.

  • The likelihood function is not a density (or pmf) -- it doesn't integrate (/sum) to 1.

  • Indeed, $\mathcal L$ may be continuous while $P$ is discrete (e.g. likelihood for a binomial parameter) or vice-versa (e.g. likelihood for an Erlang distribution with unit rate parameter but unspecified shape)

Imagine a bivariate function of a single potential observation $x$ (say a Poisson count) and a single parameter (e.g. $\lambda$) -- in this example discrete in $x$ and continuous in $\lambda$ -- then when you slice that bivariate function of $(x,\lambda)$ one way you get $p_\lambda(x)$ (each slice gives a different pmf) and when you slice it the other way you get $\mathcal L_x(\lambda)$ (each a different likelihood function).

(That bivariate function simply expresses the way $x$ and $\lambda$ are related via your model)

[Alternatively, consider a discrete $\theta$ and a continuous $x$; here the likelihood is discrete and the density continuous.]

As soon as you specify $x$, you identify a particular $\mathcal L$, that we call the likelihood function of that sample. It tells you about $\theta$ for that sample -- in particular what values had more or less likelihood of giving that sample.

Likelihood is a function that tell you about the relative chance (in that ratios of likelihoods can be thought of as ratios of probabilities of being in $x+dx$) that this value of $\theta$ could produce your data.

$\endgroup$
3
  • $\begingroup$ It's not a density. For any given $\theta$ its value is equal to that of a density evaluated at a specific $x$, but it's equal to a different density at every $\theta$. Imagine you took every possible value of $\theta$ (imagine for the moment a discrete $\theta$ but continuous $p$) and for each one, you drew the probability density, $p$). Then at the specific sample value ($x$), you slice orthogonally across all those different densities. That slice is a likelihood function --- and it is not itself a density. $\endgroup$
    – Glen_b
    Nov 12 '19 at 22:45
  • $\begingroup$ The second thing. For each specific value of $\theta$ and a given $x$, $L$ is equal to the value of the density evaluated at that $x$, given that $\theta$. but the density changes with $\theta$, so $L$ is equal to the value of a different density (each evaluated at $x$) at every point on $L$. $\endgroup$
    – Glen_b
    Nov 12 '19 at 23:27
  • $\begingroup$ @Glen_b-ReinstateMonica I know its bit of an ask but can you draw pictorially what you mentioned about taking a slice in an instant so we can create same mental picture of what you have? $\endgroup$ Jan 23 '20 at 14:37
3
$\begingroup$

According to the Bayesian theory, $f(\theta|x_1,...,x_n) = \frac{f(x_1,...,x_n|\theta) * f(\theta)}{f(x_1,...,x_n)}$ holds, that is $\text{posterior} = \frac{\text{likelihood} * \text{prior}}{evidence}$.

Notice that the maximum likelihood estimate omits the prior beliefs(or defaults it to zero-mean Gaussian and count on it as the L2 regularization or weight decay) and treats the evidence as constant(when calculating the partial derivative with respect to $\theta$).

It tries to maximize the likelihood by adjusting $\theta$ and just treating $f(\theta|x_1,...,x_n)$ equal to $f(x_1,...,x_n|\theta)$ which we can easily get(usually the loss) and keep the likelihood as $\mathcal{L}(\theta|\mathbf x)$. The true probability $\frac{f(x_1,...,x_n|\theta) * f(\theta)}{f(x_1,...,x_n)}$ can hardly be worked out because the evidence(the denominator), $\int_{\theta} f(x_1, ...,x_n, \theta)d\theta$, is intractable.

Hope this helps.

$\endgroup$
2
$\begingroup$

I agree with @Big Agnes. Here is what my professor taught in class: One way is to think of likelihood function $L(\theta | \mathbf{x})$ as a random function which depends on the data. Different data gives us different likelihood functions. So you may say conditioning on data. Given a realization of data, we want to find a $\hat{\theta}$ such that $L(\theta | \mathbf{x})$ is maximized or you can say $\hat{\theta}$ is most consistent with data. This is same to say we maximize "observed probability" $P (\mathbf{x} | \theta)$. We use $P(\mathbf{x} | \theta)$ to do calculation but it is different from $P(\mathbf{X} | \theta)$. Small $\mathbf{x}$ stands for observed values, while $\mathbf{X}$ stands for random variable. If you know $\theta$, then $P(\mathbf{x} | \theta)$ is the probability/ density of observing $\mathbf{x}$.

$\endgroup$
6
  • $\begingroup$ Thanks. Could I equivalently use $P(x|\theta)$ (with lowercase x) instead of $\mathcal{L}(\theta|x)$? When we write ${max}_\theta{P(x|\theta)}$ it should be clear that $x$ is fixed and we're trying to find the most consistent $\theta$. Or does $\mathcal{L}(\theta|x)$ refer to something more abstract that has a different implementation in some situations? $\endgroup$
    – danijar
    Jun 11 '17 at 22:52
  • $\begingroup$ Also, could you elaborate why $\mathcal{L}(\theta|x)$ is a random function? It seems like it should be deterministic since both $x$ and $\theta$ are fixed (unless we give $\theta$ a Bayesian treatment and consider it a random variable). $\endgroup$
    – danijar
    Jun 11 '17 at 22:54
  • $\begingroup$ It is better to use $L(\theta | \mathbf{x})$ (actually likelihood is defined in such way), because it is a function of $\theta$ rather than $\mathbf{x}$. I don't know if $L(\theta | \mathbf{x})$ refers to something abstract. As for random function argument, it is just a way of thinking of likelihood function. The true $\theta$ is fixed, but we don't know it. That's why we estimate it. You plug in your observations into $L(\theta | \mathbf{x})$, and different data gives you different functions. So likelihood function depends on your observation, so it is like a function of random variables. $\endgroup$
    – jwyao
    Jun 11 '17 at 23:09
  • $\begingroup$ $L(\theta | \mathbf{x})$ looks like a posterior distribution but in fact, it isn't. There is no assumption on the (prior) distribution $\pi (\theta)$ of $\theta$. $\endgroup$
    – jwyao
    Jun 11 '17 at 23:11
  • $\begingroup$ So one could write $\mathcal{L}_x(\theta)$ to express this more clearly? (I know we shouldn't write this in practice since it's not common notation.) $\endgroup$
    – danijar
    Jun 11 '17 at 23:17
1
$\begingroup$

I think the other answers given by jwyao and Glen_b are quite good. I just wanted to add a very simple example which is too long for a comment.

Consider one observation $X$ from a Bernoulli distribution with probability of success $\theta$. With $\theta$ fixed (known or unknown), the distribution of $X$ is given by $p(X|\theta)$.

$$P(x|\theta) = \theta^x(1-\theta)^{1-x}$$

In other words, we know that $P(X=1) = 1 - P(X=0) = \theta$.

Alternatively, we could look treat the observation as fixed and view this as a function of $\theta$.

$$L(\theta | x) = \theta^x(1-\theta)^{1-x}$$

For example, in a maximum likelihood setting, we seek to find $\theta$ which maximizes the likelihood as a function of $\theta$. For example, if we observe $X = 1$, then the likelihood becomes

$$L(\theta | x) = \begin{cases} \theta, & \theta \in [0,1] \\ 0, & else \end{cases}$$

and we see that the MLE would be $\hat\theta = 1$.

Not sure that I've really added any value to the discussion, but I just wanted to give a simple example of the different ways of viewing the same function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.