2
$\begingroup$

If we have a set of observations $\mathcal{D} = \{x_i\}_{i=1}^n$ then the likelihood $\mathcal{L}$ is:

$$ \mathcal{L}(\theta \mid \mathcal{D}) = P_\theta(\mathcal{D})$$ and if the observations are independent then:

$$\mathcal{L}(\theta \mid \mathcal{D}) = \prod_{i=1}^{n}P_\theta(x_i)$$

where $P$ is the probability mass function for a given parameter $\theta$.

In maximum likelihood estimation we treat $\theta$ as a parameter for estimation. Furthermore, in MAP:

$$P(\theta \mid \mathcal{D}) \propto P(\mathcal{D} \mid \theta) P(\theta)$$

If we assume uniform prior then:

$$P(\theta \mid \mathcal{D}) \propto P(\mathcal{D} \mid \theta)$$

and it is said that MLE and MAP give the same point estimate.

What troubles me is that in Bayesian context the likelihood is $P(\mathcal{D} \mid \theta)$ where we treat $\theta$ as a random variable so conditional probabilities make sense. But in a Frequentist approach we don't treat the parameters as random variables.

Should we view likelihood as a conditional probability or it depends on the approach (Bayesian vs Frequentist)?

$\endgroup$
3
  • 1
    $\begingroup$ Not sure to grasp your question. Are you asking about how to interpret the likelihood function? $\endgroup$
    – utobi
    Oct 16, 2022 at 20:11
  • 2
    $\begingroup$ The likelihood of $\theta$ given the observations $\{x_i\}_{i=1}^n$ is not a conditional probability for $\theta$ (the suggestion is meaningless in frequentist terms and in general wrong in Bayesian terms - there is no reason to expect it to sum or integrate over $\theta$ to $1$), though it is proportional to the conditional probability $\mathbb P(\{x_i\}_{i=1}^n \mid \theta)$ $\endgroup$
    – Henry
    Oct 16, 2022 at 21:11
  • $\begingroup$ stats.stackexchange.com/questions/2641/… may be helpful $\endgroup$
    – Henry
    Oct 16, 2022 at 21:14

1 Answer 1

1
$\begingroup$

Yes, likelihood $P(X|\theta)$ is always (technically) a conditional probability. It tells you, how likely the occurrence of the observed data is, given the fixed values of parameters. The usage is drastically different in the two approaches. I agree the notation $L(\theta|X)$ can be confusing.

Role of likelihood for a frequentist

Frequentists work with the likelihood as a loss function. It is used to indicate how well the model with parameters $\theta$ fits the data. The goal is to maximize the likelihood, e.g., using EM.

$$\hat{\theta}=\arg\max_\theta P(X|\theta)$$

Because of monotonicity of natural logarithm and its better numerical stability in the interval $[0,1]$, you can see log-likelihood used instead.

$$\hat{\theta}=\arg\max_\theta \log P(X|\theta)$$

Role of likelihood for a bayesian

Bayesian approach thinks of likelihood as an increment of information. Model can get updated by adding new information. E.g., we add five samples to the model. The equations are of course derived from Bayes' theorem.

$$P(\theta|X_{1:5})\propto P(X_{1:5}|\theta)\cdot P(\theta)$$

Now we have our updated model, which has seen the five samples. We can add new information, $X_6$.

$$P(\theta|X_{1:6})\propto P(X_{6}|\theta)\cdot P(\theta|X_{1:5})$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.