26
$\begingroup$

I have a simple question regarding "conditional probability" and "Likelihood". (I have already surveyed this question here but to no avail.)

It starts from the Wikipedia page on likelihood. They say this:

The likelihood of a set of parameter values, $\theta$, given outcomes $x$, is equal to the probability of those observed outcomes given those parameter values, that is

$$\mathcal{L}(\theta \mid x) = P(x \mid \theta)$$

Great! So in English, I read this as: "The likelihood of parameters equaling theta, given data X = x, (the left-hand-side), is equal to the probability of the data X being equal to x, given that the parameters are equal to theta". (Bold is mine for emphasis).

However, no less than 3 lines later on the same page, the Wikipedia entry then goes on to say:

Let $X$ be a random variable with a discrete probability distribution $p$ depending on a parameter $\theta$. Then the function

$$\mathcal{L}(\theta \mid x) = p_\theta (x) = P_\theta (X=x), \, $$

considered as a function of $\theta$, is called the likelihood function (of $\theta$, given the outcome $x$ of the random variable $X$). Sometimes the probability of the value $x$ of $X$ for the parameter value $\theta$ is written as $P(X=x\mid\theta)$; often written as $P(X=x;\theta)$ to emphasize that this differs from $\mathcal{L}(\theta \mid x) $ which is not a conditional probability, because $\theta$ is a parameter and not a random variable.

(Bold is mine for emphasis). So, in the first quote, we are literally told about a conditional probability of $P(x\mid\theta)$, but immediately afterwards, we are told that this is actually NOT a conditional probability, and should be in fact written as $P(X = x; \theta)$?

So, which one is is? Does the likelihood actually connote a conditional probability ala the first quote? Or does it connote a simple probability ala the second quote?

EDIT:

Based on all the helpful and insightful answers I have received thus far, I have summarized my question - and my understanding thus far as so:

  • In English, we say that: "The likelihood is a function of parameters, GIVEN the observed data." In math, we write it as: $L(\mathbf{\Theta}= \theta \mid \mathbf{X}=x)$.
  • The likelihood is not a probability.
  • The likelihood is not a probability distribution.
  • The likelihood is not a probability mass.
  • The likelihood is however, in English: "A product of probability distributions, (continuous case), or a product of probability masses, (discrete case), at where $\mathbf{X} = x$, and parameterized by $\mathbf{\Theta}= \theta$." In math, we then write it as such: $L(\mathbf{\Theta}= \theta \mid \mathbf{X}=x) = f(\mathbf{X}=x ; \mathbf{\Theta}= \theta) $ (continuous case, where $f$ is a PDF), and as
    $L(\mathbf{\Theta}= \theta \mid \mathbf{X}=x) = P(\mathbf{X}=x ; \mathbf{\Theta}= \theta) $ (discrete case, where $P$ is a probability mass). The takeaway here is that at no point here whatsoever is a conditional probability coming into play at all.
  • In Bayes theorem, we have: $P(\mathbf{\Theta}= \theta \mid \mathbf{X}=x) = \frac{P(\mathbf{X}=x \mid \mathbf{\Theta}= \theta) \ P(\mathbf{\Theta}= \theta)}{P(\mathbf{X}=x)}$. Colloquially, we are told that "$P(\mathbf{X}=x \mid \mathbf{\Theta}= \theta)$ is a likelihood", however, this is not true, since $\mathbf{\Theta}$ might be an actual random variable. Therefore, what we can correctly say however, is that this term $P(\mathbf{X}=x \mid \mathbf{\Theta}= \theta)$ is simply "similar" to a likelihood. (?) [On this I am not sure.]

EDIT II:

Based on @amoebas answer, I have drawn his last comment. I think it's quite elucidating, and I think it clears up the main contention I was having. (Comments on the image).

enter image description here

EDIT III:

I extended @amoebas comments to the Bayesian case just now as well:

enter image description here

$\endgroup$
  • $\begingroup$ You already got two nice answers but check also stats.stackexchange.com/q/112451/35989 $\endgroup$ – Tim Jul 16 '16 at 7:24
  • $\begingroup$ @Tim Excellent link thanks! Unfortunately I am still unclear as to the specific questions I have vis-a-vis Likelihood and the conditional probability (?) that it seems to conjures. On this, I am still unclear. :-/ $\endgroup$ – Creatron Jul 17 '16 at 19:47
  • 2
    $\begingroup$ "Given that" does not always mean conditional probability. Sometimes this phrase is merely an attempt to indicate what symbols are intended to be fixed in a calculation or conceptually. $\endgroup$ – whuber Jul 18 '16 at 14:49
  • 2
    $\begingroup$ Some people do indeed use such a typographic convention with semicolons. There are many, many conventions: subscripts, superscripts, etc. You often have to figure out what somebody means from the context or their text descriptions of what they are doing. $\endgroup$ – whuber Jul 18 '16 at 20:15
  • 4
    $\begingroup$ When $\theta$ is a random variate (that is, a value considered to arise from random variable $\Theta$), nothing in the definition of likelihood changes. It's still a likelihood. Logically, this is no different than saying that a blue butterfly is still a butterfly. Technically, it raises issues about the joint distribution of $\Theta$ and $x$. Evidently this joint distribution must be well defined and enjoy certain "regularity conditions" before you may identify the likelihood with a conditional probability. $\endgroup$ – whuber Jul 18 '16 at 22:10
18
$\begingroup$

I think this is largely unnecessary splitting hairs.

Conditional probability $P(x\mid y)\equiv P(X=x \mid Y=y)$ of $x$ given $y$ is defined for two random variables $X$ and $Y$ taking values $x$ and $y$. But we can also talk about probability $P(x\mid\theta)$ of $x$ given $\theta$ where $\theta$ is not a random variable but a parameter.

Note that in both cases the same term "given" and the same notation $P(\cdot\mid\cdot)$ can be used. There is no need to invent different notations. Moreover, what is called "parameter" and what is called "random variable" can depend on your philosophy, but the math does not change.

The first quote from Wikipedia states that $\mathcal{L}(\theta \mid x) = P(x \mid \theta)$ by definition. Here it is assumed that $\theta$ is a parameter. The second quote says that $\mathcal{L}(\theta \mid x)$ is not a conditional probability. This means that it is not a conditional probability of $\theta$ given $x$; and indeed it cannot be, because $\theta$ is assumed to be a parameter here.

In the context of Bayes theorem $$P(a\mid b)=\frac{P(b\mid a)P(a)}{P(b)},$$ both $a$ and $b$ are random variables. But we can still call $P(b\mid a)$ "likelihood" (of $a$), and now it is also a bona fide conditional probability (of $b$). This terminology is standard in Bayesian statistics. Nobody says it is something "similar" to the likelihood; people simply call it the likelihood.

Note 1: In the last paragraph, $P(b\mid a)$ is obviously a conditional probability of $b$. As a likelihood $\mathcal L(a\mid b)$ it is seen as a function of $a$; but it is not a probability distribution (or conditional probability) of $a$! Its integral over $a$ does not necessarily equal $1$. (Whereas its integral over $b$ does.)

Note 2: Sometimes likelihood is defined up to an arbitrary proportionality constant, as emphasized by @MichaelLew (because most of the time people are interested in likelihood ratios). This can be useful, but is not always done and is not essential.


See also What is the difference between "likelihood" and "probability"? and in particular @whuber's answer there.

I fully agree with @Tim's answer in this thread too (+1).

$\endgroup$
  • 1
    $\begingroup$ So a likelihood, can in fact, be equal to, a conditional probability (as per the last paragraph), correct? This is what I am trying to square. For example in one of the first answers, we have: "First, likelihood cannot be generally equal to a the probability of the data given the parameter value, as likelihood is only defined up to a proportionality constant. Fisher was explicit about that when he first formalised likelihood (Fisher, 1922). " This is what I am trying to square. Is the likelihood - can the likelihood - ever be equal to a conditional probability? $\endgroup$ – Creatron Jul 19 '16 at 0:53
  • $\begingroup$ @Creatron I added two Notes to my answer. Do they clarify it? $\endgroup$ – amoeba says Reinstate Monica Jul 19 '16 at 9:54
  • 1
    $\begingroup$ In regards to Note1: Since $P(b|a)$ is a conditional probability distribution, and since $L(a|b)$ cannot be a probability distribution, then it seems to me that the most 'correct' way we can write the equation for likelihood in this context is: $L(a|b) \propto P(b|a)$, and not as, $L(a|b) = P(b|a)$. (I know that in optimization this doesn't make a difference, but I am trying to nail down the correctness of what the likelihood is here). Is my understanding right? Thank you for your patience. $\endgroup$ – Creatron Jul 19 '16 at 20:41
  • 1
    $\begingroup$ @Creatron I think you are confusing several distinct issues here. I assume that you are talking about a Bayes theorem setting (which is what my Note 1 refers to), where both $a$ and $b$ are random events. Okay, so $P(b|a)$ is a conditional probability distribution of $b$ given $a$. But $L(a|b)$ is supposed to be seen as a function of $a$, not of $b$! And it is not the probability distribution of $a$ because it does not sum to one. This has nothing to do with the issue or proportionality (which is my Note 2). I think we can write $L(a|b)=P(b|a)$. $\endgroup$ – amoeba says Reinstate Monica Jul 19 '16 at 21:21
  • 1
    $\begingroup$ Amoeba, thank you!! You have been instrumental in un-knotting those concepts for me, thank you so much!! :) I just "extended" the diagram to the Bayesian case, and would appreciate your feedback to make sure I have understood that correctly as well. I have also accepted your answer. Once again, massively gracious! $\endgroup$ – Creatron Jul 21 '16 at 1:54
10
$\begingroup$

You already got two nice answers, but since it still seems unclear for you let me provide one. Likelihood is defined as

$$ \mathcal{L}(\theta|X) = P(X|\theta) = \prod_i f_\theta(x_i) $$

so we have likelihood of some parameter value $\theta$ given the data $X$. It is equal to product of probability mass (discrete case), or density (continuous case) functions $f$ of $X$ parametrized by $\theta$. Likelihood is a function of parameter given the data. Notice that $\theta$ is a parameter that we are optimizing, not a random variable, so it does not have any probabilities assigned to it. This is why Wikipedia states that using conditional probability notation may be ambiguous, since we are not conditioning on any random variable. On another hand, in Bayesian setting $\theta$ is a random variable and does have distribution, so we can work with it as with any other random variable and we can use Bayes theorem to calculate the posterior probabilities. Bayesian likelihood is still likelihood since it tells us about likelihood of data given the parameter, the only difference is that the parameter is considered as random variable.

If you know programming, you can think of likelihood function as of overloaded function in programming. Some programming languages allow you to have function that works differently when called using different parameter types. If you think of likelihood like this, then by default if takes as argument some parameter value and returns likelihood of data given this parameter. On another hand, you can use such function in Bayesian setting, where parameter is random variable, this leads to basically the same output, but that can be understood as conditional probability since we are conditioning on random variable. In both cases the function works the same, just you use it and understand it a little bit differently.

// likelihood "as" overloaded function
Default Likelihood(Numeric theta, Data X) {
    return f(X, theta); // returns likelihood, not probability
}

Bayesian Likelihood(RandomVariable theta, Data X) {
    return f(X, theta); // since theta is r.v., the output can be
                        // understood as conditional probability
}

Moreover, you rather won't find Bayesians who write Bayes theorem as

$$ P(\theta|X) \propto \mathcal{L}(\theta|X) P(\theta) $$

...this would be very confusing. First, you would have $\theta|X$ on both sides of equation and it wouldn't have much sense. Second, we have posterior probability to know about probability of $\theta$ given data (i.e. the thing that you would like to know in likelihoodist framework, but you don't when $\theta$ is not a random variable). Third, since $\theta$ is a random variable, we have and write it as conditional probability. The $L$-notation is generally reserved for likelihoodist setting. The name likelihood is used by convention in both approaches to denote similar thing: how probability of observing such data changes given your model and the parameter.

$\endgroup$
  • $\begingroup$ Thank you Tim, this has been very helpful in my understanding. I have re-consolidated my question (see under "Edit") with this new knowledge. I believe everything I have now written there is true. The only holdout is the last point in the list on Bayes rule. If you could take a look I would appreciate that a lot. Thanks again, and have an upvote! $\endgroup$ – Creatron Jul 18 '16 at 21:39
  • 1
    $\begingroup$ @Creatron I added a sentence commenting your last bullet to my answer, hope it is now clear -- if not please say so. $\endgroup$ – Tim Jul 19 '16 at 7:37
  • $\begingroup$ (1/2) Your edits on the overloaded operator helps me a lot. In this case, it seems to me that we can say this: 1) Under the 'mathematically pure' (historical case in the sense of what Fisher probably meant), case, where $\theta$ is not a random variable, and instead is a parameter of a PDF, (or a function of a parameter?), then the likelihood is equal to the probability of $P(X=x ; \theta)$. The likelihood function is NOT a probability distribution, sure, but it is EQUAL TO the probability of $P(X=x ; \theta)$. Is this correct? $\endgroup$ – Creatron Jul 19 '16 at 23:00
  • $\begingroup$ (2/2) In the second case however, (2), when the context is a Bayesian setting, then in this case our parameters are a r.v, and so in this case the likelihood IS in fact, a conditional probability distribution, of P(b|a), written however, as L(a|b). So in the first 'default' case, the likelihood was definitely NOT a probability distribution, (but was equal to a probability value), however in the second case, the likelihood IS in fact a probability distribution, and that probability distribution is a conditional probability, written as P(b|a). Is this correct? $\endgroup$ – Creatron Jul 19 '16 at 23:04
  • 2
    $\begingroup$ Thank you Tim, even though I accepted @amoeba 's answer, your post truly helped me understand this varied and deep concept, esp your analogy to overloaded functions. Thank you again! $\endgroup$ – Creatron Jul 21 '16 at 2:01
7
$\begingroup$

There are several aspects of the common descriptions of likelihood that are imprecise or omit detail in a way that engenders confusion. The Wikipedia entry is a good example.

First, likelihood cannot be generally equal to a the probability of the data given the parameter value, as likelihood is only defined up to a proportionality constant. Fisher was explicit about that when he first formalised likelihood (Fisher, 1922). The reason for that seems to be the fact that there is no restraint on the integral (or sum) of a likelihood function, and the probability of observing data $x$ within a statistical model given any value of the parameter(s) is strongly affected by the precision of the data values and of the granularity of specification of the parameter values.

Second, it is more helpful to think about the likelihood function than individual likelihoods. The likelihood function is a function of the model parameter value(s), as is obvious from a graph of a likelihood function. Such a graph also makes it easy to see that the likelihoods allow a ranking of the various values of the parameter(s) according to how well the model predicts the data when set to those parameter values. Exploration of likelihood functions makes the roles of the data and the parameter values much more clear, in my opinion, than can cogitation of the various formulas given in the original question.

The use a ratio of pairs of likelihoods within a likelihood function as the relative degree of support offered by the observed data for the parameter values (within the model) gets around the problem of unknown proportionality constants because those constants cancel in the ratio. It is important to note that the constants would not necessarily cancel in a ratio of likelihoods that come from separate likelihood functions (i.e. from different statistical models).

Finally, it is useful to be explicit about the role of the statistical model because likelihoods are determined by the statistical model as well as the data. If you choose a different model you get a different likelihood function, and you can get a different unknown proportionality constant.

Thus, to answer the original question, likelihoods are not a probability of any sort. They do not obey Kolmogorov's axioms of probability, and they play a different role in statistical support of inference from the roles played by the various types of probability.

  1. Fisher (1922) On the mathematical foundations of statistics http://rsta.royalsocietypublishing.org/content/222/594-604/309
$\endgroup$
  • 1
    $\begingroup$ The first line in your post summarizes my frustration with this topic. At any rate, some questions based on your post, sir: 1) The bayesian formula is often written as $P(a|b) = \frac{P(b|a)P(a)}{P(b)}$, where (we are told) that $P(b|a)$ is a 'likelihood', and that $P(a)$ is a 'prior'. If likelihood is not a probability, then is this statement false? 2) My motivation for the question is in the context of deriving a maximum likelihood estimator, which inevitably links a likelihood to a (seemingly) concrete (conditional) probability. Given those two examples, how then to reconcile those? Thanks. $\endgroup$ – Creatron Jul 16 '16 at 1:36
  • $\begingroup$ @Creatron 1. No, the statement is not necessarily wrong. The likelihood function is how the evidence enters the calculation, and combining it with a probability distribution yields a probability distribution. In that context the unknown proportionality constant is not a problem because after the product of the likelihood function and prior probability distribution is arbitrarily scaled so that it has the correct unity integral (or sum). $\endgroup$ – Michael Lew Jul 16 '16 at 2:59
  • $\begingroup$ 2. In the context of finding a maximum likelihood estimate it makes no difference whether you use a conditional probability or a likelihood, as they will be proportional over the entire range of parameter values. $\endgroup$ – Michael Lew Jul 16 '16 at 3:05
  • 1
    $\begingroup$ Can we then say that while $L(\theta|x) = P(x|\theta)$ is technically wrong, $L(\theta|x) \propto P(x|\theta)$ is technically and formally correct? Is that all there is to it? $\endgroup$ – Creatron Jul 16 '16 at 3:36
  • $\begingroup$ Thank you Micheal Lew, your post has really helped in my understanding of this problem, much appreciated. $\endgroup$ – Creatron Jul 21 '16 at 2:00
7
$\begingroup$

Wikipedia should have said that $L(\theta)$ is not a conditional probability of $\theta$ being in some specified set, nor a probability density of $\theta$. Indeed, if there are infinitely many values of $\theta$ in the parameter space, you can have $$ \sum_\theta L(\theta) = \infty, $$ for example by having $L(\theta)=1$ regardless of the value of $\theta$, and if there is some standard measure $d\theta$ on the parameter space $\Theta$, then in the same way one can have $$ \int_\Theta L(\theta)\,d\theta =\infty. $$ An essential point that the article should emphasize is that $L$ is the function $$ \theta \mapsto P(x\mid\theta) \text{ and NOT } x\mapsto P(x\mid\theta). $$

$\endgroup$
  • 2
    $\begingroup$ +1 and thanks for the edit of my answer; I forgot that \mid exists. $\endgroup$ – amoeba says Reinstate Monica Jul 19 '16 at 23:41
  • $\begingroup$ @amoeba : Glad to help. $\qquad$ $\endgroup$ – Michael Hardy Jul 19 '16 at 23:52
3
$\begingroup$

"I read this as: "The likelihood of parameters equaling theta, given data X = x, (the left-hand-side), is equal to the probability of the data X being equal to x, given that the parameters are equal to theta". (Bold is mine for emphasis)."

It's the probability of the set of observations given the parameter is theta. This is perhaps confusing because they write $P(x|\theta)$ but then $\mathcal{L}(\theta|x)$.

The explanation (somewhat objectively) implies that $\theta$ is not a random variable. It could, for example, be a random variable with some prior distribution in a Bayesian setting. The point however, is that we suppose $\theta=\theta$, a concrete value and then make statements about the likelihood of our observations. This is because there is only one true value of $\theta$ in whatever system we're interested in.

$\endgroup$
  • $\begingroup$ Ok, so I then conclude based on this that i) The first image on the wikipedia is wrong, because (to my knowledge at least), $P(a|b)$ is always read as a conditional probability, and what they SEEM to want to say, is that it's not - or ever - "probability of the data GIVEN this theta", it's rather, "probability of the data, PARAMETERIZED by this theta". Is this correct? Thanks. (To summarize, it seems that $L(\theta|x) = P(X=x; \theta)$. $\endgroup$ – Creatron Jul 16 '16 at 0:03
  • $\begingroup$ This however is problematic, because in a Bayesian formulation, $P(a|b) = \frac{P(b|a) \ P(a)}{P(b)}$, the $P(b|a)$ we are told is in fact the likelihood, (and is in fact a conditional probability). However this contradicts what we just said, and also contradicts what the wiki says in image 2. $\endgroup$ – Creatron Jul 16 '16 at 0:06
  • $\begingroup$ $L(\theta|x):=P(x|\theta)$. The $\theta$ is to the left of $x$ in $L$ to emphasize that we think of $L$ as a function of $\theta$, the parameter we wish to optimize. So there's no contradiction. $\endgroup$ – Alex R. Jul 16 '16 at 0:19
  • $\begingroup$ Is the right-hand-side of $L(\theta|x)$ := $P(x|\theta)$ a conditional probability? $\endgroup$ – Creatron Jul 16 '16 at 0:26
  • $\begingroup$ This makes more sense to me now. Thanks for your initial help, @Alex. $\endgroup$ – Creatron Jul 21 '16 at 2:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.