Skip to main content
6 of 12
added 1708 characters in body
Creatron
  • 1.7k
  • 2
  • 20
  • 24

Wikipedia entry on likelihood seems ambiguous

I have a simple question regarding "conditional probability" and "Likelihood". (I have already surveyed this question here but to no avail.)

It starts from the Wikipedia page on likelihood. They say this:

The likelihood of a set of parameter values, $\theta$, given outcomes $x$, is equal to the probability of those observed outcomes given those parameter values, that is

$$\mathcal{L}(\theta |x) = P(x | \theta)$$

Great! So in English, I read this as: "The likelihood of parameters equaling theta, given data X = x, (the left-hand-side), is equal to the probability of the data X being equal to x, given that the parameters are equal to theta". (Bold is mine for emphasis).

However, no less than 3 lines later on the same page, the Wikipedia entry then goes on to say:

Let $X$ be a random variable with a discrete probability distribution $p$ depending on a parameter $\theta$. Then the function

$$\mathcal{L}(\theta |x) = p_\theta (x) = P_\theta (X=x), \, $$

considered as a function of $\theta$, is called the likelihood function (of $\theta$, given the outcome $x$ of the random variable $X$). Sometimes the probability of the value $x$ of $X$ for the parameter value $\theta$ is written as $P(X=x|\theta)$; often written as $P(X=x;\theta)$ to emphasize that this differs from $\mathcal{L}(\theta |x) $ which is not a conditional probability, because $\theta$ is a parameter and not a random variable.

(Bold is mine for emphasis). So, in the first quote, we are literally told about a conditional probability of $P(x|\theta)$, but immediately afterwards, we are told that this is actually NOT a conditional probability, and should be in fact written as $P(X = x; \theta)$?

So, which one is is? Does the likelihood actually connote a conditional probability ala the first quote? Or does it connote a simple probability ala the second quote?

EDIT:

Based on all the helpful and insightful answers I have received thus far, I have summarized my question - and my understanding thus far as so:

  • In English, we say that: "The likelihood is a function of parameters, GIVEN the observed data." In math, we write it as: $L(\mathbf{\Theta}= \theta \ | \ \mathbf{X}=x)$.
  • The likelihood is not a probability.
  • The likelihood is not a probability distribution.
  • The likelihood is not a probability mass.
  • The likelihood is however, in English: "A product of probability distributions, (continuous case), or a product of probability masses, (discrete case), at where $\mathbf{X} = x$, and parameterized by $\mathbf{\Theta}= \theta$." In math, we then write it as such: $L(\mathbf{\Theta}= \theta \ | \ \mathbf{X}=x) = f(\mathbf{X}=x ; \mathbf{\Theta}= \theta) $ (continuous case, where $f$ is a PDF), and as
    $L(\mathbf{\Theta}= \theta \ | \ \mathbf{X}=x) = P(\mathbf{X}=x ; \mathbf{\Theta}= \theta) $ (discrete case, where P is a probability mass). The takeaway here is that at no point here whatsoever is a conditional probability coming into play at all.
  • In Bayes theorem, we have: $P(\mathbf{\Theta}= \theta \ | \ \mathbf{X}=x) = \frac{P(\mathbf{X}=x \ | \ \mathbf{\Theta}= \theta) \ P(\mathbf{\Theta}= \theta)}{P(\mathbf{X}=x)}$. Colloquially, we are told that "$P(\mathbf{X}=x \ | \ \mathbf{\Theta}= \theta)$ is a likelihood", however, this is not true, since $\mathbf{\Theta}$ might be an actual random variable. Therefore, what we can correctly say however, is that this term $P(\mathbf{X}=x \ | \ \mathbf{\Theta}= \theta)$ is simply "similar" to a likelihood. (?) [On this I am not sure.]
Creatron
  • 1.7k
  • 2
  • 20
  • 24