What does "likelihood is only defined up to a multiplicative constant of proportionality" mean in practice?

Question

I'm reading a paper where the authors are leading from a discussion of maximum likelihood estimation to Bayes' Theorem, ostensibly as an introduction for beginners.

As a likelihood example, they start with a binomial distribution:

$$p(x|n,\theta) = \binom{n}{x}\theta^x(1-\theta)^{n-x}$$

and then log both sides

$$\ell(\theta|x, n) = x \ln (\theta) + (n-x)\ln (1-\theta)$$

with the rationale that:

"Because the likelihood is only defined up to a multiplicative constant of proportionality (or an additive constant for the log-likelihood), we can rescale ... by dropping the binomial coefficient and writing the log-likelihood in place of the likelihood"

The math makes sense, but I can't understand what is meant by "the likelihood is only defined up to a multiplicative constant of proportionality" and how this allows dropping the binomial coefficient and going from $p(x|n,\theta)$ to $\ell(\theta|x,n)$.

Similar terminology has come up in other questions (here and here), but it still not clear what, practically, likelihood being defined or bringing information up to a multiplicative constant means. Is it possible to explain this in layman's terms?

Answer 1

The point is that sometimes, different models (for the same data) can lead to likelihood functions which differ by a multiplicative constant, but the information content must clearly be the same. An example:

We model $n$ independent Bernoulli experiments, leading to data $X_1, \dots, X_n$, each with a Bernoulli distribution with (probability) parameter $p$. This leads to the likelihood function $$ \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i} $$ Or we can summarize the data by the binomially distributed variable $Y=X_1+X_2+\dotsm+X_n$, which has a binomial distribution, leading to the likelihood function $$ \binom{n}{y} p^y (1-p)^{n-y} $$ which, as a function of the unknown parameter $p$, is proportional to the former likelihood function. The two likelihood functions clearly contains the same information, and should lead to the same inferences!

And indeed, by definition, they are considered the same likelihood function.

---

Another viewpoint: observe that when the likelihood functions are used in Bayes theorem, as needed for bayesian analysis, such multiplicative constants simply cancel! so they are clearly irrelevant to bayesian inference. Likewise, it will cancel when calculating likelihood ratios, as used in optimal hypothesis tests (Neyman-Pearson lemma.) And it will have no influence on the value of maximum likelihood estimators. So we can see that in much of frequentist inference it cannot play a role.

---

We can argue from still another viewpoint. The Bernoulli probability function (hereafter we use the term "density") above is really a density with respect to counting measure, that is, the measure on the non-negative integers with mass one for each non-negative integer. But we could have defined a density with respect to some other dominating measure. In this example this will seem (and is) artificial, but in larger spaces (function spaces) it is really fundamental! Let us, for the purpose of illustration, use the specific geometric distribution, written $\lambda$, with $\lambda(0)=1/2$, $\lambda(1)=1/4$, $\lambda(2)=1/8$ and so on. Then the density of the Bernoulli distribution with respect to $\lambda$ is given by $$ f_{\lambda}(x) = p^x (1-p)^{1-x}\cdot 2^{x+1} $$ meaning that $$ P(X=x)= f_\lambda(x) \cdot \lambda(x) $$ With this new, dominating, measure, the likelihood function becomes (with notation from above) $$ \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i} 2^{x_i+1} = p^y (1-p)^{n-y} 2^{y+n} $$ note the extra factor $2^{y+n}$. So when changing the dominating measure used in the definition of the likelihood function, there arises a new multiplicative constant, which does not depend on the unknown parameter $p$, and is clearly irrelevant. That is another way to see how multiplicative constants must be irrelevant. This argument can be generalized using Radon-Nikodym derivatives (as the argument above is an example of.)

Answer 2

It basically means that only relative value of the PDF matters. For instance, the standard normal (Gaussian) PDF is: $f(x)=\frac{1}{\sqrt{2\pi}}e^{-x^2/2}$, your book is saying that they could use $g(x)=e^{-x^2/2}$ instead, because they don't care for the scale, i.e. $c=\frac{1}{\sqrt{2\pi}}$.

This happens because they maximize likelihood function, and $c\cdot g(x)$ and $g(x)$ will have the same maximum. Hence, maximum of $e^{-x^2/2}$ will be the same as of $f(x)$. So, they don't bother about the scale.

Answer 3

I cannot explain the meaning of the quotation, but for maximum-likelihood estimation, it does not matter whether we choose to find the maximum of the likelihood function $L(\mathbf x; \theta)$ (regarded as a function of $\theta$ or the maximum of $aL(\mathbf x; \theta)$ where $a$ is some constant. This is because we are not interested in the maximum value of $L(\mathbf x; \theta)$ but rather the value $\theta_{\text{ML}}$ where this maximum occurs, and both $L(\mathbf x; \theta)$ and $aL(\mathbf x; \theta)$ achieve their maximum value at the same $\theta_{\text{ML}}$. So, multiplicative constants can be ignored. Similarly, we could choose to consider any monotone function $g(\cdot)$ (such as the logarithm) of the likelihood function $L(\mathbf x; \theta)$, determine the maximum of $g(L(\mathbf x;\theta))$, and infer the value of $\theta_{\text{ML}}$ from this. For the logarithm, the multipliative constant $a$ becomes the additive constant $\ln(a)$ and this too can be ignored in the process of finding the location of the maximum: $\ln(a)+\ln(L(\mathbf x; \theta)$ is maximized at the same point as $\ln(L(\mathbf x; \theta)$.

Turning to maximum a posteriori probability (MAP) estimation, $\theta$ is regarded as a realization of a random variable $\Theta$ with a priori density function $f_{\Theta}(\theta)$, the data $\mathbf x$ is regarded as a realization of a random variable $\mathbf X$, and the likelihood function is considered to be the value of the conditional density $f_{\mathbf X\mid \Theta}(\mathbf x\mid \Theta=\theta)$ of $\mathbf X$ conditioned on $\Theta = \theta$; said conditional density function being evaluated at $\mathbf x$. The a posteriori density of $\Theta$ is $$f_{\Theta\mid \mathbf X}(\theta \mid \mathbf x) = \frac{f_{\mathbf X\mid \Theta}(\mathbf x\mid \Theta=\theta)f_\Theta(\theta)}{f_{\mathbf X}(\mathbf x)} \tag{1}$$ in which we recognize the numerator as the joint density $f_{\mathbf X, \Theta}(\mathbf x, \theta)$ of the data and the parameter being estimated. The point $\theta_{\text{MAP}}$ where $f_{\Theta\mid \mathbf X}(\theta \mid \mathbf x)$ attains its maximum value is the MAP estimate of $\theta$, and, using the same arguments as in the paragraph, we see that we can ignore $[f_{\mathbf X}(\mathbf x)]^{-1}$ on the right side of $(1)$ as a multiplicative constant just as we can ignore multiplicative constants in both $f_{\mathbf X\mid \Theta}(\mathbf x\mid \Theta=\theta)$ and in $f_\Theta(\theta)$. Similarly when log-likelihoods are being used, we can ignore additive constants.

Answer 4

In layman's terms, you'll often look for the maximum likelihood and $f(x)$ and $kf(x)$ share the same critical points.

Answer 5

I would suggest not to drop from sight any constant terms in the likelihood function (i.e. terms that do not include the parameters). In usual circumstances, they do not affect the $\text {argmax}$ of the likelihood, as already mentioned. But:

There may be unusual circumstances when you will have to maximize the likelihood subject to a ceiling -and then you should "remember" to include any constants in the calculation of its value.

Also, you may be performing model selection tests for non-nested models, using the value of the likelihood in the process -and since the models are non-nested the two likelihoods will have different constants.

Apart from these, the sentence

"Because the likelihood is only defined up to a multiplicative constant of proportionality (or an additive constant for the log-likelihood)"

is wrong, because the likelihood is first a joint probability density function, not just "any" objective function to be maximized.