I have a simple question regarding "conditional probability" and "Likelihood". (I have already surveyed this question here but to no avail.)
It starts from the Wikipedia page on likelihood. They say this:
The likelihood of a set of parameter values, $\theta$, given outcomes $x$, is equal to the probability of those observed outcomes given those parameter values, that is
$$\mathcal{L}(\theta |x) = P(x | \theta)$$
Great! So in English, I read this as: "The likelihood of parameters equaling theta, given data X = x, (the left-hand-side), is equal to the probability of the data X being equal to x, given that the parameters are equal to theta". (Bold is mine for emphasis).
However, no less than 3 lines later on the same page, the Wikipedia entry then goes on to say:
Let $X$ be a random variable with a discrete probability distribution $p$ depending on a parameter $\theta$. Then the function
$$\mathcal{L}(\theta |x) = p_\theta (x) = P_\theta (X=x), \, $$
considered as a function of $\theta$, is called the likelihood function (of $\theta$, given the outcome $x$ of the random variable $X$). Sometimes the probability of the value $x$ of $X$ for the parameter value $\theta$ is written as $P(X=x|\theta)$; often written as $P(X=x;\theta)$ to emphasize that this differs from $\mathcal{L}(\theta |x) $ which is not a conditional probability, because $\theta$ is a parameter and not a random variable.
(Bold is mine for emphasis). So, in the first quote, we are literally told about a conditional probability of $P(x|\theta)$, but immediately afterwards, we are told that this is actually NOT a conditional probability, and should be in fact written as $P(X = x; \theta)$?
So, which one is is? Does the likelihood actually connote a conditional probability ala the first quote? Or does it connote a simple probability ala the second quote?
EDIT:
Based on all the helpful and insightful answers I have received thus far, I have summarized my question - and my understanding thus far as so:
- In English, we say that: "The likelihood is a function of parameters, GIVEN the observed data." In math, we write it as: $L(\mathbf{\Theta}= \theta \ | \ \mathbf{X}=x)$.
- The likelihood is not a probability.
- The likelihood is not a probability distribution.
- The likelihood is not a probability mass.
- The likelihood is however, in English: "A product of probability distributions, (continuous case), or a product of probability masses, (discrete case), at where $\mathbf{X} = x$, and parameterized by $\mathbf{\Theta}= \theta$." In math, we then write it as such: $L(\mathbf{\Theta}= \theta \ | \ \mathbf{X}=x) = f(\mathbf{X}=x ; \mathbf{\Theta}= \theta) $ (continuous case, where $f$ is a PDF), and as
$L(\mathbf{\Theta}= \theta \ | \ \mathbf{X}=x) = P(\mathbf{X}=x ; \mathbf{\Theta}= \theta) $ (discrete case, where P is a probability mass). The takeaway here is that at no point here whatsoever is a conditional probability coming into play at all. - In Bayes theorem, we have: $P(\mathbf{\Theta}= \theta \ | \ \mathbf{X}=x) = \frac{P(\mathbf{X}=x \ | \ \mathbf{\Theta}= \theta) \ P(\mathbf{\Theta}= \theta)}{P(\mathbf{X}=x)}$. Colloquially, we are told that "$P(\mathbf{X}=x \ | \ \mathbf{\Theta}= \theta)$ is a likelihood", however, this is not true, since $\mathbf{\Theta}$ might be an actual random variable. Therefore, what we can correctly say however, is that this term $P(\mathbf{X}=x \ | \ \mathbf{\Theta}= \theta)$ is simply "similar" to a likelihood. (?) [On this I am not sure.]