I am told

the method of maximum likelihood says we should use the model that assigns the greatest probability to the data we have observed; formally, the maximum likelihood estimator is found by solving
$\hat{\theta}= arg_{\theta}max\{p(x|\theta)\}$
where $p(x|\theta)$ is called the likelihood function.

Am I correct reading $p(x|\theta)$ in English as "the probability of the data given the parameters"? I am confused because we at first seem to be told that MLE is about the probability of the parameters given the data.

  • $\begingroup$ Thank you, In English "Likelihood" is similar to "probability" $\endgroup$
    – Kirsten
    Jan 4 at 19:27
  • $\begingroup$ Thank you. How would you read $p(x|\theta)$ in English? $\endgroup$
    – Kirsten
    Jan 4 at 19:32

The parameters are not random variables but fixed unknowns (at least in the likelihood approach to inference). It is thus incorrect to talk of a probability distribution on the parameters. The MLE is the value of the parameters that makes the actual observations the most likely to have occurred $$p(x|\hat\theta) = \max_\theta p(x|\theta)$$ The quantity $p(x|θ)$ thus reads as

  1. the probability of observing $x$ when the parameter value is equal to $θ$ in a discrete setting and
  2. the density of $x$ when the parameter value is equal to $θ$ in a continuous setting.

(I would even avoid given since this could be interpreted as a conditional probability or density, which does not make sense if $\theta$ is not a random variable, i.e., outside a Bayesian framework.)

To quote the originator of the notion of likelihood, R.A. Fisher :

I suggest that we may speak without confusion of the likelihood of one value of p being thrice the likelihood of another (…) likelihood is not here used loosely as a synonym of probability, but simply to express the relative frequencies with which such values of the hypothetical quantity p would in fact yield the observed sample.

  • 1
    $\begingroup$ So is the vertical line symbol ambiguous? Sometimes meaning "given" and sometimes meaning "when" ? As a programmer I feel like I can't get maths to compile! $\endgroup$
    – Kirsten
    Jan 5 at 8:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.