# Is there any difference between Frequentist and Bayesian on the definition of Likelihood?

Some sources say likelihood function is not conditional probability, some say it is. This is very confusing to me.

According to most sources I have seen, the likelihood of a distribution with parameter $\theta$, should be a product of probability mass functions given $n$ samples of $x_i$:

$$L(\theta) = L(x_1,x_2,...,x_n;\theta) = \prod_{i=1}^n p(x_i;\theta)$$

For example in Logistic Regression, we use an optimization algorithm to maximize the likelihood function (Maximum Likelihood Estimation) to obtain the optimal parameters and therefore the final LR model. Given the $n$ training samples, which we assume to be independent from each other, we want to maximize the product of probabilities (or the joint probability mass functions). This seems quite obvious to me.

According to Relation between: Likelihood, conditional probability and failure rate, "likelihood is not a probability and it is not a conditional probability". It also mentioned, "likelihood is a conditional probability only in Bayesian understanding of likelihood, i.e., if you assume that $\theta$ is a random variable."

I read about the different perspectives of treating a learning problem between frequentist and Bayesian.

According to a source, for Bayesian inference, we have a priori $P(\theta)$, likelihood $P(X|\theta)$, and we want to obtain the posterior $P(\theta|X)$, using Bayesian theorem:

$$P(\theta|X)=\dfrac{P(X|\theta) \times P(\theta)}{P(X)}$$

I'm not familiar with Bayesian Inference. How come $P(X|\theta)$ which is the distribution of the observed data conditional on its parameters, is also termed the likelihood? In Wikipedia, it says sometimes it is written $L(\theta|X)=p(X|\theta)$. What does this mean?

is there a difference between Frequentist and Bayesian's definitions on likelihood??

Thanks.

EDIT:

There are different ways of interpreting Bayes' theorem - Bayesian interpretation and Frequentist interpretation (See: Bayes' theorem - Wikipedia).

• Two key properties of likelihood are that (a) that it is a function of $\theta$ for a particular $X$ rather than the other way round, and (b) it can only be known up to a positive constant of proportionality. It is not a probability (conditional or otherwise), because it does not need to sum or integrate to $1$ over all $\theta$ – Henry May 30 '18 at 19:57
• – Tim May 31 '18 at 6:39

There is no difference in the definition - in both cases, the likelihood function is any function of the parameter that is proportional to the sampling density. Strictly speaking we do not require that the likelihood be equal to the sampling density; it needs only be proportional, which allows removal of multiplicative parts that do not depend on the parameters.

Whereas the sampling density is interpreted as a function of the data, conditional on a specified value of the parameter, the likelihood function is interpreted as a function of the parameter for a fixed data vector. So in the standard case of IID data you have:

$$L_\mathbf{x}(\theta) \propto \prod_{i=1}^n p(x_i|\theta).$$

In Bayesian statistics, we usually express Bayes' theorem in its simplest form as:

$$\pi (\theta|\mathbf{x}) \propto \pi(\theta) \cdot L_\mathbf{x}(\theta).$$

This expression for Bayes' theorem stresses that both of its multilicative elements are functions of the parameter, which is the object of interest in the posterior density. (This proportionality result fully defines the rule, since the posterior is a density, and so there is a unique multiplying constant that makes it integrate to one.) As you point out in your update, Bayesian and frequentist philosophy have different interpretive structures. Within the frequentist paradigm the parameter is generally treated as a "fixed constant" and so it is not ascribed a probability measure. Frequentists therefore reject the ascription of a prior or posterior distribution to the parameter (for more discussion on these philosophic and interpretive differences, see e.g., O'Neill 2009).

• Isnt there the fundamental difference, that the likelihood in Bayes theorem has to be a (conditional) probability distribution, $p(x|\theta)$, i.e., a distribution over random variables x conditioned on random variables $\theta$. Whereas in the frequentist framework, for MLE, the likelihood is assumed to be a function (not a probability distribution) over parameters $\theta$ (which are no random variables in the frequentist framework), while the actual random variables x are held fixed? – guest1 Feb 11 at 19:06
• No. The function $p(x|\theta)$ is the sampling density, not the likelihood function. The likelihood $L_\mathbf{x}(\theta)$ does not need to be a conditional probability distribution over $x$. You can multiply the likelihood function by any function of $x$ and it is still considered to be a valid likelihood. – Reinstate Monica Feb 11 at 21:31
• Yes but my point is that $p(x|\theta)$ used in Bayes theorem is given by the same formula as the likelihood $L_x$ just that the role of the parameters and variables is different. That is to say in $p(x|\theta)$ both x and $\theta$ are random variables whilst in the likelihood $L_x$, x is a random variable but held fixed and $\theta$ is a parameter (which is treated as a non random variable in MLE to find its maximum value). So even these two quantities are given by the same formula they are fundamentally different in their behaviour. – guest1 Feb 12 at 7:49
• Hence you shouldnt be allowed to use the likelihood $L_x$ in Bayes theorem because it isnt even a probability distribution – guest1 Feb 12 at 7:51
• Or where am I wrong here in this thought process? – guest1 Feb 14 at 7:41

The likelihood function is defined independently from $-$or prior to$-$ the statistical paradigm that is used for inference, as a function, $L(\theta;x)$ (or $L(\theta|x)$), of the parameter $\theta$, function that depends on $-$or is indexed by$-$ the observation(s) $x$ available for this inference. And also implicitly depending on the family of probability models chosen to represent the variability or randomness in the data. For a given value of the pair $(\theta,x)$, the value of this function is exactly identical to the value of the density of the model at $x$ when indexed with the parameter $\theta$. Which is often crudely translated as the "probability of the data".

To quote more authoritative and historical sources than an earlier answer on this forum,

"We may discuss the probability of occurrence of quantities which can be observed . . . in relation to any hypotheses which may be suggested to explain these observations. We can know nothing of the probability of hypotheses . . . [We] may ascertain the likelihood of hypotheses . . . by calculation from observations: . . . to speak of the likelihood . . . of an observable quantity has no meaning." R.A. Fisher, On the probable error’’ of a coefficient of correlation deduced from a small sample. Metron 1, 1921, p.25

and

"What we can find from a sample is the likelihood of any particular value of r, if we define the likelihood as a quantity proportional to the probability that, from a population having the particular value of r, a sample having the observed value of r, should be obtained." R.A. Fisher, On the probable error’’ of a coefficient of correlation deduced from a small sample. Metron 1, 1921, p.24

which mentions a proportionality that Jeffreys (and I) find superfluous:

"..likelihood, a convenient term introduced by Professor R.A. Fisher, though in his usage it is sometimes multiplied by a constant factor. This is the probability of the observations given the original information and the hypothesis under discussion." H. Jeffreys, Theory of Probability, 1939, p.28

To quote but one sentence from the excellent historical entry to the topic by John Aldrich (Statistical Science, 1997):

"Fisher (1921, p. 24) redrafted what he had written in 1912 about inverse probability, distinguishing between the mathematical operations that can be performed on probability densities and likelihoods: likelihood is not a ‘‘differential element,’’ it cannot be integrated." J. Aldrich, R. A. Fisher and the Making of Maximum Likelihood 1912 – 1922, 1997, p.9

When adopting a Bayesian approach, the likelihood function does not change in shape or in nature. It keeps being the density at $x$ indexed by $\theta$. The additional feature is that, since $\theta$ is also endowed with a probabilistic model, the prior distribution, the density at $x$ indexed by $\theta$ can also be interpreted as a conditional density, conditional on a realisation of $\theta$: in a Bayesian modelling, one realisation of $\theta$ is produced from the prior, with density $\pi(\cdot)$, then a realisation of $X$, $x$, is produced from the distribution with density $L(\theta|\cdot)$, indexed by $\theta$. In other words, and with respect to the proper dominating measure, the pair $(\theta,x)$ has joint density $$\pi(\theta) \times L(\theta|x)$$ from which one derives the posterior density of $\theta$, that is, the conditional density of $\theta$, conditional on a realisation of $x$ as $$\pi(\theta|x) \propto \pi(\theta) \times L(\theta|x)$$ also expressed as $$\text{posterior} \propto \text{prior} \times \text{likelihood}$$ found since Jeffreys (1939).

Note: I find the distinction made in the introduction of the Wikipedia page about likelihood functions between frequentist and Bayesian likelihoods confusing and unnecessary, or just plain wrong as the large majority of current Bayesian statisticians does not use likelihood as a substitute for posterior probability. Similarly, the "difference" pointed out in the Wikipedia page about Bayes Theorem sounds more confusing than anything else, as this theorem is a probability statement about a change of conditioning, independent from the paradigm or from the meaning of a probability statement. (In my opinion, it is more a definition than a theorem!)

• the function $L (\vec{x}; \theta)$
• the method of finding a 'best' value of the parameter $\theta$ under the condition of some observed data (Maximum L., Minimum L., log-L., etc.)