Is there any difference between Frequentist and Bayesian on the definition of Likelihood?

Question

Some sources say likelihood function is not conditional probability, some say it is. This is very confusing to me.

According to most sources I have seen, the likelihood of a distribution with parameter $\theta$, should be a product of probability mass functions given $n$ samples of $x_i$:

$$L(\theta) = L(x_1,x_2,...,x_n;\theta) = \prod_{i=1}^n p(x_i;\theta)$$

For example in Logistic Regression, we use an optimization algorithm to maximize the likelihood function (Maximum Likelihood Estimation) to obtain the optimal parameters and therefore the final LR model. Given the $n$ training samples, which we assume to be independent from each other, we want to maximize the product of probabilities (or the joint probability mass functions). This seems quite obvious to me.

According to Relation between: Likelihood, conditional probability and failure rate, "likelihood is not a probability and it is not a conditional probability". It also mentioned, "likelihood is a conditional probability only in Bayesian understanding of likelihood, i.e., if you assume that $\theta$ is a random variable."

I read about the different perspectives of treating a learning problem between frequentist and Bayesian.

According to a source, for Bayesian inference, we have a priori $P(\theta)$, likelihood $P(X|\theta)$, and we want to obtain the posterior $P(\theta|X)$, using Bayesian theorem:

$$P(\theta|X)=\dfrac{P(X|\theta) \times P(\theta)}{P(X)}$$

I'm not familiar with Bayesian Inference. How come $P(X|\theta)$ which is the distribution of the observed data conditional on its parameters, is also termed the likelihood? In Wikipedia, it says sometimes it is written $L(\theta|X)=p(X|\theta)$. What does this mean?

is there a difference between Frequentist and Bayesian's definitions on likelihood??

Thanks.

---

EDIT:

There are different ways of interpreting Bayes' theorem - Bayesian interpretation and Frequentist interpretation (See: Bayes' theorem - Wikipedia).

Answer 1

There is no difference in the definition - in both cases, the likelihood function is any function of the parameter that is proportional to the sampling density. Strictly speaking we do not require that the likelihood be equal to the sampling density; it needs only be proportional, which allows removal of multiplicative parts that do not depend on the parameters.

Whereas the sampling density is interpreted as a function of the data, conditional on a specified value of the parameter, the likelihood function is interpreted as a function of the parameter for a fixed data vector. So in the standard case of IID data you have:

$$L_\mathbf{x}(\theta) \propto \prod_{i=1}^n p(x_i|\theta).$$

In Bayesian statistics, we usually express Bayes' theorem in its simplest form as:

$$\pi (\theta|\mathbf{x}) \propto \pi(\theta) \cdot L_\mathbf{x}(\theta).$$

This expression for Bayes' theorem stresses that both of its multilicative elements are functions of the parameter, which is the object of interest in the posterior density. (This proportionality result fully defines the rule, since the posterior is a density, and so there is a unique multiplying constant that makes it integrate to one.) As you point out in your update, Bayesian and frequentist philosophy have different interpretive structures. Within the frequentist paradigm the parameter is generally treated as a "fixed constant" and so it is not ascribed a probability measure. Frequentists therefore reject the ascription of a prior or posterior distribution to the parameter (for more discussion on these philosophic and interpretive differences, see e.g., O'Neill 2009).

Answer 2

The likelihood function is defined independently from $-$or prior to$-$ the statistical paradigm that is used for inference, as a function, $L(\theta;x)$ (or $L(\theta|x)$), of the parameter $\theta$, function that depends on $-$or is indexed by$-$ the observation(s) $x$ available for this inference. And also implicitly depending on the family of probability models chosen to represent the variability or randomness in the data. For a given value of the pair $(\theta,x)$, the value of this function is exactly identical to the value of the density of the model at $x$ when indexed with the parameter $\theta$. Which is often crudely translated as the "probability of the data".

To quote more authoritative and historical sources than an earlier answer on this forum,

"We may discuss the probability of occurrence of quantities which can be observed . . . in relation to any hypotheses which may be suggested to explain these observations. We can know nothing of the probability of hypotheses . . . [We] may ascertain the likelihood of hypotheses . . . by calculation from observations: . . . to speak of the likelihood . . . of an observable quantity has no meaning." R.A. Fisher, On the ``probable error’’ of a coefficient of correlation deduced from a small sample. Metron 1, 1921, p.25

and

"What we can find from a sample is the likelihood of any particular value of r, if we define the likelihood as a quantity proportional to the probability that, from a population having the particular value of r, a sample having the observed value of r, should be obtained." R.A. Fisher, On the ``probable error’’ of a coefficient of correlation deduced from a small sample. Metron 1, 1921, p.24

which mentions a proportionality that Jeffreys (and I) find superfluous:

"..likelihood, a convenient term introduced by Professor R.A. Fisher, though in his usage it is sometimes multiplied by a constant factor. This is the probability of the observations given the original information and the hypothesis under discussion." H. Jeffreys, Theory of Probability, 1939, p.28

To quote but one sentence from the excellent historical entry to the topic by John Aldrich (Statistical Science, 1997):

"Fisher (1921, p. 24) redrafted what he had written in 1912 about inverse probability, distinguishing between the mathematical operations that can be performed on probability densities and likelihoods: likelihood is not a ‘‘differential element,’’ it cannot be integrated." J. Aldrich, R. A. Fisher and the Making of Maximum Likelihood 1912 – 1922, 1997, p.9

When adopting a Bayesian approach, the likelihood function does not change in shape or in nature. It keeps being the density at $x$ indexed by $\theta$. The additional feature is that, since $\theta$ is also endowed with a probabilistic model, the prior distribution, the density at $x$ indexed by $\theta$ can also be interpreted as a conditional density, conditional on a realisation of $\theta$: in a Bayesian modelling, one realisation of $\theta$ is produced from the prior, with density $\pi(\cdot)$, then a realisation of $X$, $x$, is produced from the distribution with density $L(\theta|\cdot)$, indexed by $\theta$. In other words, and with respect to the proper dominating measure, the pair $(\theta,x)$ has joint density $$\pi(\theta) \times L(\theta|x)$$ from which one derives the posterior density of $\theta$, that is, the conditional density of $\theta$, conditional on a realisation of $x$ as $$\pi(\theta|x) \propto \pi(\theta) \times L(\theta|x)$$ also expressed as $$\text{posterior} \propto \text{prior} \times \text{likelihood}$$ found since Jeffreys (1939).

Note: I find the distinction made in the introduction of the Wikipedia page about likelihood functions between frequentist and Bayesian likelihoods confusing and unnecessary, or just plain wrong as the large majority of current Bayesian statisticians does not use likelihood as a substitute for posterior probability. Similarly, the "difference" pointed out in the Wikipedia page about Bayes Theorem sounds more confusing than anything else, as this theorem is a probability statement about a change of conditioning, independent from the paradigm or from the meaning of a probability statement. (In my opinion, it is more a definition than a theorem!)

Answer 3

As a small addendum:

The name "Likelihood" is entirely misleading, because there are very many different possible meanings. Not only the "normal language" one, but also in statistics. I can think of at least three different, but even related expressions that are all called Likelihood; even in text books.

That said, when taking the multiplicative definition of Likelihood, there is nothing in it that will turn it into any kind of probability in the sense of its (e.g. axiomatic) definition. It is a real-valued number. You can do lots of things to compute or relate it to a probability (taking ratios, calculating priors and posteriors, etc.) -- but on itself it has no meaning in terms of probability.

The answer has been more or less obsoleted by the much more informative and comprehensive answer by Xi'an. But by request, some text book definitions of Likelihood:

• the function $L (\vec{x}; \theta)$
• the method of finding a 'best' value of the parameter $\theta$ under the condition of some observed data (Maximum L., Minimum L., log-L., etc.)
• the ratio of Likelihood values for different priors (e.g. in a classification task) ... and moreover the different meanings one can try to attribute to the (ab)use of the aforementioned elements.