Is likelihood a conditional probability?

Question

If we have a set of observations $\mathcal{D} = \{x_i\}_{i=1}^n$ then the likelihood $\mathcal{L}$ is:

$$ \mathcal{L}(\theta \mid \mathcal{D}) = P_\theta(\mathcal{D})$$ and if the observations are independent then:

$$\mathcal{L}(\theta \mid \mathcal{D}) = \prod_{i=1}^{n}P_\theta(x_i)$$

where $P$ is the probability mass function for a given parameter $\theta$.

In maximum likelihood estimation we treat $\theta$ as a parameter for estimation. Furthermore, in MAP:

$$P(\theta \mid \mathcal{D}) \propto P(\mathcal{D} \mid \theta) P(\theta)$$

If we assume uniform prior then:

$$P(\theta \mid \mathcal{D}) \propto P(\mathcal{D} \mid \theta)$$

and it is said that MLE and MAP give the same point estimate.

What troubles me is that in Bayesian context the likelihood is $P(\mathcal{D} \mid \theta)$ where we treat $\theta$ as a random variable so conditional probabilities make sense. But in a Frequentist approach we don't treat the parameters as random variables.

Should we view likelihood as a conditional probability or it depends on the approach (Bayesian vs Frequentist)?

Answer 1

Yes, likelihood $P(X|\theta)$ is always (technically) a conditional probability. It tells you, how likely the occurrence of the observed data is, given the fixed values of parameters. The usage is drastically different in the two approaches. I agree the notation $L(\theta|X)$ can be confusing.

Role of likelihood for a frequentist

Frequentists work with the likelihood as a loss function. It is used to indicate how well the model with parameters $\theta$ fits the data. The goal is to maximize the likelihood, e.g., using EM.

$$\hat{\theta}=\arg\max_\theta P(X|\theta)$$

Because of monotonicity of natural logarithm and its better numerical stability in the interval $[0,1]$, you can see log-likelihood used instead.

$$\hat{\theta}=\arg\max_\theta \log P(X|\theta)$$

Role of likelihood for a bayesian

Bayesian approach thinks of likelihood as an increment of information. Model can get updated by adding new information. E.g., we add five samples to the model. The equations are of course derived from Bayes' theorem.

$$P(\theta|X_{1:5})\propto P(X_{1:5}|\theta)\cdot P(\theta)$$

Now we have our updated model, which has seen the five samples. We can add new information, $X_6$.

$$P(\theta|X_{1:6})\propto P(X_{6}|\theta)\cdot P(\theta|X_{1:5})$$