Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm reading a paper where the authors are leading from a discussion of maximum likelihood estimation to Bayes' Theorem, ostensibly as an introduction for beginners.

As a likelihood example, they start with a binomial distribution:

$$p(x|n,\theta) = \binom{n}{x}\theta^x(1-\theta)^{n-x}$$

and then log both sides

$$\ell(\theta|x, n) = x \ln (\theta) + (n-x)\ln (1-\theta)$$

with the rationale that:

"Because the likelihood is only defined up to a multiplicative constant of proportionality (or an additive constant for the log-likelihood), we can rescale ... by dropping the binomial coefficient and writing the log-likelihood in place of the likelihood"

The math makes sense, but I can't understand what is meant by "the likelihood is only defined up to a multiplicative constant of proportionality" and how this allows dropping the binomial coefficient and going from $p(x|n,\theta)$ to $\ell(\theta|x,n)$.

Similar terminology has come up in other questions (here and here), but it still not clear what, practically, likelihood being defined or bringing information up to a multiplicative constant means. Is it possible to explain this in layman's terms?

share|improve this question

5 Answers 5

up vote 8 down vote accepted

The point is that sometimes, different models (for the same data) can lead to likelihood functions which differ by a multiplicative constant, but the information content must clearly be the same. An example:

We model $n$ independent bernoulli experiments, leading to data $X_1, \dots, X_n$, each with a bernoulli distribution with (probability) parameter $p$. This leads to the likelihood function $$ \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i} $$ Or we can summarize the data by the binomially distributed variable $Y=X_1+X_2+\dots+X_n$, which has a binomial distribution, leading to the likelihood function $$ \binom{n}{y} p^y (1-p)^{n-y} $$ which, as a function of the unknown parameter $p$, is proportional to the former likelihood function. The two likelihood functions clearly contains the same information, and should lead to the same inferences!

And indeed, by definition, they are considered the same likelihood function.

Another wievpoint: observe that when the likelihood functions are used in Bayes theorem, as needed for bayesian analysis, such multiplicative constants simply cancel! so they are clearly irrelevant to bayesian inference.

share|improve this answer
    
No, they are not identical! as long as $\binom{n}{y} \not= 1$, they are different, but proportional. –  kjetil b halvorsen May 13 at 15:25
    
Yes, I'm sorry. –  Scortchi May 13 at 15:43
    
"the information content must clearly be the same" This is only true if you believe in the likelihood principle! –  jsk May 16 at 5:30
    
Yes, maybe, but I did show how it follows from bayesian principles. –  kjetil b halvorsen May 16 at 12:38

I cannot explain the meaning of the quotation, but for maximum-likelihood estimation, it does not matter whether we choose to find the maximum of the likelihood function $L(\mathbf x; \theta)$ (regarded as a function of $\theta$ or the maximum of $aL(\mathbf x; \theta)$ where $a$ is some constant. This is because we are not interested in the maximum value of $L(\mathbf x; \theta)$ but rather the value $\theta_{\text{ML}}$ where this maximum occurs, and both $L(\mathbf x; \theta)$ and $aL(\mathbf x; \theta)$ achieve their maximum value at the same $\theta_{\text{ML}}$. So, multiplicative constants can be ignored. Similarly, we could choose to consider any monotone function $g(\cdot)$ (such as the logarithm) of the likelihood function $L(\mathbf x; \theta)$, determine the maximum of $g(L(\mathbf x;\theta))$, and infer the value of $\theta_{\text{ML}}$ from this. For the logarithm, the multipliative constant $a$ becomes the additive constant $\ln(a)$ and this too can be ignored in the process of finding the location of the maximum: $\ln(a)+\ln(L(\mathbf x; \theta)$ is maximized at the same point as $\ln(L(\mathbf x; \theta)$.

Turning to maximum a posteriori probability (MAP) estimation, $\theta$ is regarded as a realization of a random variable $\Theta$ with a priori density function $f_{\Theta}(\theta)$, the data $\mathbf x$ is regarded as a realization of a random variable $\mathbf X$, and the likelihood function is considered to be the value of the conditional density $f_{\mathbf X\mid \Theta}(\mathbf x\mid \Theta=\theta)$ of $\mathbf X$ conditioned on $\Theta = \theta$; said conditional density function being evaluated at $\mathbf x$. The a posteriori density of $\Theta$ is $$f_{\Theta\mid \mathbf X}(\theta \mid \mathbf x) = \frac{f_{\mathbf X\mid \Theta}(\mathbf x\mid \Theta=\theta)f_\Theta(\theta)}{f_{\mathbf X}(\mathbf x)} \tag{1}$$ in which we recognize the numerator as the joint density $f_{\mathbf X, \Theta}(\mathbf x, \theta)$ of the data and the parameter being estimated. The point $\theta_{\text{MAP}}$ where $f_{\Theta\mid \mathbf X}(\theta \mid \mathbf x)$ attains its maximum value is the MAP estimate of $\theta$, and, using the same arguments as in the paragraph, we see that we can ignore $[f_{\mathbf X}(\mathbf x)]^{-1}$ on the right side of $(1)$ as a multiplicative constant just as we can ignore multiplicative constants in both $f_{\mathbf X\mid \Theta}(\mathbf x\mid \Theta=\theta)$ and in $f_\Theta(\theta)$. Similarly when log-likelihoods are being used, we can ignore additive constants.

share|improve this answer

It basically means that only relative value of the PDF matters. For instance, the standard normal (Gaussian) PDF is: $f(x)=\frac{1}{\sqrt{2\pi}}e^{-x^2/2}$, your book is saying that they could use $g(x)=e^{-x^2/2}$ instead, because they don't care for the scale, i.e. $c=\frac{1}{\sqrt{2\pi}}$.

This happens because they maximize likelihood function, and $c\cdot g(x)$ and $g(x)$ will have the same maximum. Hence, maximum of $e^{-x^2/2}$ will be the same as of $f(x)$. So, they don't bother about the scale.

share|improve this answer

In layman's terms, you'll often look for the maximum likelihood and $f(x)$ and $kf(x)$ share the same critical points.

share|improve this answer
1  
So do $f(x)$ and $f(x)+2$ but they would not be equivalent likelihood functions –  Henry May 13 at 18:18
    
Please, as Alecos Papadopoulos writes in his answer, "the likelihood is first a joint probability density function". Because of the iid assumption for random samples, that joint function is a product of simple density functions, so multiplicative factors do arise, addends do not. –  Sergio May 13 at 18:36

I would suggest not to drop from sight any constant terms in the likelihood function (i.e. terms that do not include the parameters). In usual circumstances, they do not affect the $\text {argmax}$ of the likelihood, as already mentioned. But:

There may be unusual circumstances when you will have to maximize the likelihood subject to a ceiling -and then you should "remember" to include any constants in the calculation of its value.

Also, you may be performing model selection tests for non-nested models, using the value of the likelihood in the process -and since the models are non-nested the two likelihoods will have different constants.

Apart from these, the sentence

"Because the likelihood is only defined up to a multiplicative constant of proportionality (or an additive constant for the log-likelihood)"

is wrong, because the likelihood is first a joint probability density function, not just "any" objective function to be maximized.

share|improve this answer
3  
Hmmm... When wearing a Bayesian hat, I always thought of the likelihood function as the conditional density function of the data given the parameter and not as a joint probability density function. The location of the maximum of the joint probability density of the data and the parameter (regarded as a function of the unknown parameter $\theta$; the data being fixed) gives the maximum a posteriori probability (MAP) estimate of $\theta$, does it not? –  Dilip Sarwate May 13 at 16:21
1  
I think you need to be a bit more careful with the language. The likelihood is a function of the parameters for a fixed sample, but is equivalent to the joint density over the sample space. That is, $$L(\boldsymbol \theta \mid \boldsymbol x) = f(\boldsymbol x \mid \boldsymbol \theta).$$ This will integrate to $1$ over the sample space, but is not necessarily $1$ when integrated over the parameter space. When you say "the likelihood is a density, viewed as a function of the parameters," that makes it sound as if you mean "density with respect to the parameters," which it isn't. –  heropup May 13 at 19:09
1  
@heropup I have already wrote that it doesn't necessarily integrate to unity over the parameter space, and so, immediately, it cannot be considered as a "density function" when it is viewed as a "function of the parameters". –  Alecos Papadopoulos May 13 at 19:13
1  
@heropup Your desired statement that "The likelihood function ... is equivalent (or proportional) to the joint density over the sample space" would indeed be much more precise but equally incorrect. The likelihood function is neither equivalent nor proportional to the joint density because the "coefficient of proportionality" is not a constant (unless the prior distribution of the unknown parameter is uniformly distributed over an interval). The joint density is $L(x\mid \theta)f(\theta)$ where $L$ is the likelihood and $f(\theta)$ is the prior distribution of the parameter. –  Dilip Sarwate May 13 at 22:09
1  
Alecos Papadopoulos: I understand you do not like the standard definition, but it is still the standard definition. In my answer I explained the thinking behind that choice of definition! You bring to market another argument: the "constant of proportionality" might be of interest for model choice---like, making AIC comparable across different model families. Somebody used that argument and asked for R's calculated likelihoods to incorporate those constants. That was rejected, because the R Gods (like B Ripley) dont believe in that argument. –  kjetil b halvorsen May 14 at 10:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.