5
$\begingroup$

According to the Wikipedia article Likelihood function, the likelihood function is defined as:

$$ \mathcal{L}(\theta|x)=P(x|\theta), $$

with parameters $\theta$ and observed data $x$. This equals $p(x|\theta)$ or $p_\theta(x)$ depending on notation and whether $\theta$ is treated as random variable or fixed value.

The notation $\mathcal{L}(\theta|x)$ seems like an unnecessary abstraction to me. Is there any benefit to using $\mathcal{L}(\theta|x)$, or could one equivalently use $P(x|\theta)$? Why was $\mathcal{L}(\theta|x)$ introduced?

$\endgroup$
  • 5
    $\begingroup$ In this context, it reminds us that the likelihood function is a function of $\theta$ with the data $x$ fixed. On the other hand, the joint distribution is a function of the data $x$ given $\theta$. $\endgroup$ – knrumsey Jun 11 '17 at 22:25
  • $\begingroup$ @BigAgnes Thanks. Aren't observed variables fixed by definition, though? I'm also confused why we can call $p(x|\theta)$ a joint distribution. Isn't it a scalar since both $x$ and $\theta$ are fixed (assuming a Frequentist approach where $\theta$ is not a random variable). $\endgroup$ – danijar Jun 11 '17 at 22:34
  • 1
    $\begingroup$ Closely related: stats.stackexchange.com/questions/224037/… $\endgroup$ – Tim Jun 12 '17 at 7:50
7
$\begingroup$

Likelihood is a function of $\theta$, given $x$, while $P$ is a function of $x$, given $\theta$.

  • The likelihood function is not a density (or pmf) -- it doesn't integrate (/sum) to 1.

  • Indeed, $\mathcal L$ may be continuous while $P$ is discrete (e.g. likelihood for a binomial parameter) or vice-versa (e.g. likelihood for an Erlang distribution with unit rate parameter but unspecified shape)

Imagine a bivariate function of a single potential observation $x$ (say a Poisson count) and a single parameter (e.g. $\lambda$) -- in this example discrete in $x$ and continuous in $\lambda$ -- then when you slice that bivariate function of $(x,\lambda)$ one way you get $p_\lambda(x)$ (each slice gives a different pmf) and when you slice it the other way you get $\mathcal L_x(\lambda)$ (each a different likelihood function).

(That bivariate function simply expresses the way $x$ and $\lambda$ are related via your model)

As soon as you specify $x$, you identify a particular $\mathcal L$, that we call the likelihood of that sample. It tells you about $\theta$ for that sample -- in particular what values had more or less likelihood of giving that sample.

Likelihood is a function that tell you about the relative chance (in that ratios of likelihoods can be thought of as ratios of probabilities of being in $x+dx$) that this value of $\theta$ could produce your data.

$\endgroup$
2
$\begingroup$

I agree with @Big Agnes. Here is what my professor taught in class: One way is to think of likelihood function $L(\theta | \mathbf{x})$ as a random function which depends on the data. Different data gives us different likelihood functions. So you may say conditioning on data. Given a realization of data, we want to find a $\hat{\theta}$ such that $L(\theta | \mathbf{x})$ is maximized or you can say $\hat{\theta}$ is most consistent with data. This is same to say we maximize "observed probability" $P (\mathbf{x} | \theta)$. We use $P(\mathbf{x} | \theta)$ to do calculation but it is different from $P(\mathbf{X} | \theta)$. Small $\mathbf{x}$ stands for observed values, while $\mathbf{X}$ stands for random variable. If you know $\theta$, then $P(\mathbf{x} | \theta)$ is the probability/ density of observing $\mathbf{x}$.

$\endgroup$
  • $\begingroup$ Thanks. Could I equivalently use $P(x|\theta)$ (with lowercase x) instead of $\mathcal{L}(\theta|x)$? When we write ${max}_\theta{P(x|\theta)}$ it should be clear that $x$ is fixed and we're trying to find the most consistent $\theta$. Or does $\mathcal{L}(\theta|x)$ refer to something more abstract that has a different implementation in some situations? $\endgroup$ – danijar Jun 11 '17 at 22:52
  • $\begingroup$ Also, could you elaborate why $\mathcal{L}(\theta|x)$ is a random function? It seems like it should be deterministic since both $x$ and $\theta$ are fixed (unless we give $\theta$ a Bayesian treatment and consider it a random variable). $\endgroup$ – danijar Jun 11 '17 at 22:54
  • $\begingroup$ It is better to use $L(\theta | \mathbf{x})$ (actually likelihood is defined in such way), because it is a function of $\theta$ rather than $\mathbf{x}$. I don't know if $L(\theta | \mathbf{x})$ refers to something abstract. As for random function argument, it is just a way of thinking of likelihood function. The true $\theta$ is fixed, but we don't know it. That's why we estimate it. You plug in your observations into $L(\theta | \mathbf{x})$, and different data gives you different functions. So likelihood function depends on your observation, so it is like a function of random variables. $\endgroup$ – jwyao Jun 11 '17 at 23:09
  • $\begingroup$ $L(\theta | \mathbf{x})$ looks like a posterior distribution but in fact, it isn't. There is no assumption on the (prior) distribution $\pi (\theta)$ of $\theta$. $\endgroup$ – jwyao Jun 11 '17 at 23:11
  • $\begingroup$ So one could write $\mathcal{L}_x(\theta)$ to express this more clearly? (I know we shouldn't write this in practice since it's not common notation.) $\endgroup$ – danijar Jun 11 '17 at 23:17
2
$\begingroup$

In logist regression, for example, the objective function defaults to maximum likeihood. The likelihood is defined as $\mathcal{L}(\theta; x_1,...,x_n) = f(x_1,...,x_n; \theta)$, where f is a probability density function or probability massive function.

According to the Bayesian inference, $f(x_1,...,x_n; \theta) = \frac{f(\theta; x_1,...,x_n) * f(x_1,...,x_n)}{f(\theta)}$ holds, that is $likelihood = \frac{posterior * prior}{evidence}$. Notice that the maximum likelihood estimate treats the ratio of prior to evidence as a constant, which omits the prior beliefs, thus MLE is considered to be a frequentist technique(rather than Bayesian).

Hope this helps.

$\endgroup$
1
$\begingroup$

I think the other answers given by jwyao and Glen_b are quite good. I just wanted to add a very simple example which is too long for a comment.

Consider one observation $X$ from a Bernoulli distribution with probability of success $\theta$. With $\theta$ fixed (known or unknown), the distribution of $X$ is given by $p(X|\theta)$.

$$P(x|\theta) = \theta^x(1-\theta)^{1-x}$$

In other words, we know that $P(X=1) = 1 - P(X=0) = \theta$.

Alternatively, we could look treat the observation as fixed and view this as a function of $\theta$.

$$L(\theta | x) = \theta^x(1-\theta)^{1-x}$$

For example, in a maximum likelihood setting, we seek to find $\theta$ which maximizes the likelihood as a function of $\theta$. For example, if we observe $X = 1$, then the likelihood becomes

$$L(\theta | x) = \begin{cases} \theta, & \theta \in [0,1] \\ 0, & else \end{cases}$$

and we see that the MLE would be $\hat\theta = 1$.

Not sure that I've really added any value to the discussion, but I just wanted to give a simple example of the different ways of viewing the same function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.