26
$\begingroup$

I'm reading a paper where the authors are leading from a discussion of maximum likelihood estimation to Bayes' Theorem, ostensibly as an introduction for beginners.

As a likelihood example, they start with a binomial distribution:

$$p(x|n,\theta) = \binom{n}{x}\theta^x(1-\theta)^{n-x}$$

and then log both sides

$$\ell(\theta|x, n) = x \ln (\theta) + (n-x)\ln (1-\theta)$$

with the rationale that:

"Because the likelihood is only defined up to a multiplicative constant of proportionality (or an additive constant for the log-likelihood), we can rescale ... by dropping the binomial coefficient and writing the log-likelihood in place of the likelihood"

The math makes sense, but I can't understand what is meant by "the likelihood is only defined up to a multiplicative constant of proportionality" and how this allows dropping the binomial coefficient and going from $p(x|n,\theta)$ to $\ell(\theta|x,n)$.

Similar terminology has come up in other questions (here and here), but it still not clear what, practically, likelihood being defined or bringing information up to a multiplicative constant means. Is it possible to explain this in layman's terms?

$\endgroup$

5 Answers 5

29
$\begingroup$

The point is that sometimes, different models (for the same data) can lead to likelihood functions which differ by a multiplicative constant, but the information content must clearly be the same. An example:

We model $n$ independent Bernoulli experiments, leading to data $X_1, \dots, X_n$, each with a Bernoulli distribution with (probability) parameter $p$. This leads to the likelihood function $$ \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i} $$ Or we can summarize the data by the binomially distributed variable $Y=X_1+X_2+\dotsm+X_n$, which has a binomial distribution, leading to the likelihood function $$ \binom{n}{y} p^y (1-p)^{n-y} $$ which, as a function of the unknown parameter $p$, is proportional to the former likelihood function. The two likelihood functions clearly contains the same information, and should lead to the same inferences!

And indeed, by definition, they are considered the same likelihood function.


Another viewpoint: observe that when the likelihood functions are used in Bayes theorem, as needed for bayesian analysis, such multiplicative constants simply cancel! so they are clearly irrelevant to bayesian inference. Likewise, it will cancel when calculating likelihood ratios, as used in optimal hypothesis tests (Neyman-Pearson lemma.) And it will have no influence on the value of maximum likelihood estimators. So we can see that in much of frequentist inference it cannot play a role.


We can argue from still another viewpoint. The Bernoulli probability function (hereafter we use the term "density") above is really a density with respect to counting measure, that is, the measure on the non-negative integers with mass one for each non-negative integer. But we could have defined a density with respect to some other dominating measure. In this example this will seem (and is) artificial, but in larger spaces (function spaces) it is really fundamental! Let us, for the purpose of illustration, use the specific geometric distribution, written $\lambda$, with $\lambda(0)=1/2$, $\lambda(1)=1/4$, $\lambda(2)=1/8$ and so on. Then the density of the Bernoulli distribution with respect to $\lambda$ is given by $$ f_{\lambda}(x) = p^x (1-p)^{1-x}\cdot 2^{x+1} $$ meaning that $$ P(X=x)= f_\lambda(x) \cdot \lambda(x) $$ With this new, dominating, measure, the likelihood function becomes (with notation from above) $$ \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i} 2^{x_i+1} = p^y (1-p)^{n-y} 2^{y+n} $$ note the extra factor $2^{y+n}$. So when changing the dominating measure used in the definition of the likelihood function, there arises a new multiplicative constant, which does not depend on the unknown parameter $p$, and is clearly irrelevant. That is another way to see how multiplicative constants must be irrelevant. This argument can be generalized using Radon-Nikodym derivatives (as the argument above is an example of.)

$\endgroup$
5
  • $\begingroup$ "the information content must clearly be the same" This is only true if you believe in the likelihood principle! $\endgroup$
    – jsk
    Commented May 16, 2014 at 5:30
  • $\begingroup$ Yes, maybe, but I did show how it follows from bayesian principles. $\endgroup$ Commented May 16, 2014 at 12:38
  • 1
    $\begingroup$ @kjetilbhalvorsen Thank you for the thoughtful answer! One thing I'm still confused about is why the likelihood of bernoulli distribution doesn't include a binomial coefficient. Your answer makes it clear why it doesn't matter, but I don't understand why it's left off of the likelihood in the first place. $\endgroup$
    – jvans
    Commented Apr 30, 2018 at 14:43
  • $\begingroup$ @jvans: It's because the binomial coefficient do not depend on the unknown parameter, so cannot influence the shape of the likelihood function $\endgroup$ Commented May 3, 2018 at 12:54
  • 1
    $\begingroup$ @jvans to asnwer this: "One thing I'm still confused about is why the likelihood of bernoulli distribution doesn't include a binomial coefficient.". Intuitively, the bernoulli likelihood assigns probability to a specific order of the sequence of trials (e.g., 0011). However, the binomial likelihood assigns probability to the total number of successes (number of 1's in the sequence), that is $\sum_i x_i$. The order here does not matter (sums are order-invariant) so you have to count all the possible combinations that would have generated the 2 successes in 0011. The bin coeff does the count $\endgroup$
    – Pietro
    Commented Jun 9, 2021 at 15:35
14
$\begingroup$

It basically means that only relative value of the PDF matters. For instance, the standard normal (Gaussian) PDF is: $f(x)=\frac{1}{\sqrt{2\pi}}e^{-x^2/2}$, your book is saying that they could use $g(x)=e^{-x^2/2}$ instead, because they don't care for the scale, i.e. $c=\frac{1}{\sqrt{2\pi}}$.

This happens because they maximize likelihood function, and $c\cdot g(x)$ and $g(x)$ will have the same maximum. Hence, maximum of $e^{-x^2/2}$ will be the same as of $f(x)$. So, they don't bother about the scale.

$\endgroup$
8
$\begingroup$

I cannot explain the meaning of the quotation, but for maximum-likelihood estimation, it does not matter whether we choose to find the maximum of the likelihood function $L(\mathbf x; \theta)$ (regarded as a function of $\theta$ or the maximum of $aL(\mathbf x; \theta)$ where $a$ is some constant. This is because we are not interested in the maximum value of $L(\mathbf x; \theta)$ but rather the value $\theta_{\text{ML}}$ where this maximum occurs, and both $L(\mathbf x; \theta)$ and $aL(\mathbf x; \theta)$ achieve their maximum value at the same $\theta_{\text{ML}}$. So, multiplicative constants can be ignored. Similarly, we could choose to consider any monotone function $g(\cdot)$ (such as the logarithm) of the likelihood function $L(\mathbf x; \theta)$, determine the maximum of $g(L(\mathbf x;\theta))$, and infer the value of $\theta_{\text{ML}}$ from this. For the logarithm, the multipliative constant $a$ becomes the additive constant $\ln(a)$ and this too can be ignored in the process of finding the location of the maximum: $\ln(a)+\ln(L(\mathbf x; \theta)$ is maximized at the same point as $\ln(L(\mathbf x; \theta)$.

Turning to maximum a posteriori probability (MAP) estimation, $\theta$ is regarded as a realization of a random variable $\Theta$ with a priori density function $f_{\Theta}(\theta)$, the data $\mathbf x$ is regarded as a realization of a random variable $\mathbf X$, and the likelihood function is considered to be the value of the conditional density $f_{\mathbf X\mid \Theta}(\mathbf x\mid \Theta=\theta)$ of $\mathbf X$ conditioned on $\Theta = \theta$; said conditional density function being evaluated at $\mathbf x$. The a posteriori density of $\Theta$ is $$f_{\Theta\mid \mathbf X}(\theta \mid \mathbf x) = \frac{f_{\mathbf X\mid \Theta}(\mathbf x\mid \Theta=\theta)f_\Theta(\theta)}{f_{\mathbf X}(\mathbf x)} \tag{1}$$ in which we recognize the numerator as the joint density $f_{\mathbf X, \Theta}(\mathbf x, \theta)$ of the data and the parameter being estimated. The point $\theta_{\text{MAP}}$ where $f_{\Theta\mid \mathbf X}(\theta \mid \mathbf x)$ attains its maximum value is the MAP estimate of $\theta$, and, using the same arguments as in the paragraph, we see that we can ignore $[f_{\mathbf X}(\mathbf x)]^{-1}$ on the right side of $(1)$ as a multiplicative constant just as we can ignore multiplicative constants in both $f_{\mathbf X\mid \Theta}(\mathbf x\mid \Theta=\theta)$ and in $f_\Theta(\theta)$. Similarly when log-likelihoods are being used, we can ignore additive constants.

$\endgroup$
1
  • $\begingroup$ This line of thinking can be done via bayes also: If you put $L$ or $aL$ into Bayes' theorem doesn't matter, the $a$ will cancel so the posterior is the same. $\endgroup$ Commented Nov 20, 2018 at 14:24
6
$\begingroup$

In layman's terms, you'll often look for the maximum likelihood and $f(x)$ and $kf(x)$ share the same critical points.

$\endgroup$
3
  • 4
    $\begingroup$ So do $f(x)$ and $f(x)+2$ but they would not be equivalent likelihood functions $\endgroup$
    – Henry
    Commented May 13, 2014 at 18:18
  • $\begingroup$ Please, as Alecos Papadopoulos writes in his answer, "the likelihood is first a joint probability density function". Because of the iid assumption for random samples, that joint function is a product of simple density functions, so multiplicative factors do arise, addends do not. $\endgroup$
    – Sergio
    Commented May 13, 2014 at 18:36
  • 3
    $\begingroup$ The joint function is such a product if and only if the data are independent. But MLE extends to dependent variables, so the product argument appears unconvincing. $\endgroup$
    – whuber
    Commented Feb 20, 2017 at 20:32
1
$\begingroup$

I would suggest not to drop from sight any constant terms in the likelihood function (i.e. terms that do not include the parameters). In usual circumstances, they do not affect the $\text {argmax}$ of the likelihood, as already mentioned. But:

There may be unusual circumstances when you will have to maximize the likelihood subject to a ceiling -and then you should "remember" to include any constants in the calculation of its value.

Also, you may be performing model selection tests for non-nested models, using the value of the likelihood in the process -and since the models are non-nested the two likelihoods will have different constants.

Apart from these, the sentence

"Because the likelihood is only defined up to a multiplicative constant of proportionality (or an additive constant for the log-likelihood)"

is wrong, because the likelihood is first a joint probability density function, not just "any" objective function to be maximized.

$\endgroup$
12
  • 4
    $\begingroup$ Hmmm... When wearing a Bayesian hat, I always thought of the likelihood function as the conditional density function of the data given the parameter and not as a joint probability density function. The location of the maximum of the joint probability density of the data and the parameter (regarded as a function of the unknown parameter $\theta$; the data being fixed) gives the maximum a posteriori probability (MAP) estimate of $\theta$, does it not? $\endgroup$ Commented May 13, 2014 at 16:21
  • 3
    $\begingroup$ I think you need to be a bit more careful with the language. The likelihood is a function of the parameters for a fixed sample, but is equivalent to the joint density over the sample space. That is, $$L(\boldsymbol \theta \mid \boldsymbol x) = f(\boldsymbol x \mid \boldsymbol \theta).$$ This will integrate to $1$ over the sample space, but is not necessarily $1$ when integrated over the parameter space. When you say "the likelihood is a density, viewed as a function of the parameters," that makes it sound as if you mean "density with respect to the parameters," which it isn't. $\endgroup$
    – heropup
    Commented May 13, 2014 at 19:09
  • 1
    $\begingroup$ @heropup I have already wrote that it doesn't necessarily integrate to unity over the parameter space, and so, immediately, it cannot be considered as a "density function" when it is viewed as a "function of the parameters". $\endgroup$ Commented May 13, 2014 at 19:13
  • 1
    $\begingroup$ Yes, I know. My point is that the phrase "The likelihood function is a density function, viewed as a function of the parameters" is itself confusing. It would be more precise to say something like, "The likelihood function is a function of the parameters for a fixed sample, and is equivalent (or proportional) to the joint density over the sample space." $\endgroup$
    – heropup
    Commented May 13, 2014 at 19:15
  • 3
    $\begingroup$ @heropup Your desired statement that "The likelihood function ... is equivalent (or proportional) to the joint density over the sample space" would indeed be much more precise but equally incorrect. The likelihood function is neither equivalent nor proportional to the joint density because the "coefficient of proportionality" is not a constant (unless the prior distribution of the unknown parameter is uniformly distributed over an interval). The joint density is $L(x\mid \theta)f(\theta)$ where $L$ is the likelihood and $f(\theta)$ is the prior distribution of the parameter. $\endgroup$ Commented May 13, 2014 at 22:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.