8
$\begingroup$

Many books and many posts on this site define the likelihood as a function of model parameters. However, does the output associated with every possible model parameter have to be unique? For example, it seems that for some two configurations of the model parameter, the observed data can be equally likely.

So, my question is whether we are playing fast and loose with the word "function" when talking about the likelihood or really the likelihood by definition has to be a function and each input for the model parameter must yield a unique $P(x|\theta)$?

$\endgroup$
4
  • $\begingroup$ @Xi'an: To pick on your correct answer, the likelihood function is really an equivalence class of functions, since there is an arbitrary (but positive) multiplicative constant. $\endgroup$ Feb 21, 2015 at 21:43
  • $\begingroup$ Ok, so the likelihood does not have to be a bijective function. So different inputs can correspond to the same output? $\endgroup$
    – Luca
    Feb 21, 2015 at 21:45
  • 2
    $\begingroup$ @Luca why shouldn't they? Take a constant function: each of its outputs is the same and it is most of the basic functions described in the textbooks. $\endgroup$
    – Tim
    Feb 21, 2015 at 21:51
  • 2
    $\begingroup$ @kjetilbhalvorsen: while I can see the motivations behind your remark (reference measure, likelihood principle, and all that), the likelihood function can be defined uniquely once the dominating measure on the sampling space is set. $\endgroup$
    – Xi'an
    Feb 22, 2015 at 6:29

3 Answers 3

18
$\begingroup$

Many books and many posts on this site define the likelihood as a function of model parameters.

If you specify a value for each of the parameters*, you will have at most one value for the likelihood.

  • (along with everything else you need to have specified, of course)

However, does the output associated with every possible model parameter have to be unique? For example, it seems that for some two configurations of the model parameter, the observed data can be equally likely.

You're confused -- take some function $f$ -- it's fine for $f(x_1)$ and $f(x_2)$ to be equal. $f(x)=(x-3)^2$ is a function, even though $f(2)=f(4)$.

That's two different arguments having the same function value, not the function having two different values for a given argument.

enter image description here

So, my question is whether we are playing fast and loose with the word "function" when talking about the likelihood

Nope

or really the likelihood by definition has to be a function and each input for the model parameter must yield a unique P(x|θ)?

Yes. But you seem to have gotten a little confused about what that means.

$\endgroup$
2
  • $\begingroup$ Thanks. I got confused with whether bijection was needed as well. $\endgroup$
    – Luca
    Feb 22, 2015 at 6:39
  • 7
    $\begingroup$ Yeah. I think this is sometimes a confusion for people, so it's a good question to have on site. The discussion between kjetil and Xi'an raises an important but somewhat more subtle issue than I think the one you were having. $\endgroup$
    – Glen_b
    Feb 22, 2015 at 6:40
10
$\begingroup$

A real-valued function $f$ associates to a vector or real entry $θ\in\Theta$ a real number $f(θ)$, that is, \begin{align*}f:\ \Theta &\longrightarrow \mathbb{R}\\ \theta &\longrightarrow f(\theta)\end{align*}Once the data $(x_1,\ldots,x_n)$ is observed and thus fixed, the likelihood associates with a given value of the parameter $θ$ the real number $$\prod_{i=1}^n p(x_i|\theta)$$ where $p(x|\theta)$ is the density of the random variable $X_i$. (I assume i.i.d.-ness in this answer to keep notations at a minimum complexity.) It is therefore a well-defined function in the mathematical sense.

$\endgroup$
1
$\begingroup$

This is closely related to the concept of identifiability in mathematical statistics, which I think relates to your question. Identifiability deals with the possibility that a poorly-specified model may lack a one-to-one relationship between parameter sets and probability distributions over the data, and this causes problems with regard to inference.

For instance, take the "over-parameterized" ANOVA model,

$$ Y_{ij} = \mu + \alpha_i + \epsilon_{ij} , $$

where $1 \leq i \leq k$, $1 \leq j \leq n$, $\epsilon_{ij} \sim$ normal$(0, \sigma^2)$, and no restrictions are placed on$\{ \alpha_i \}_{i=1}^{k}$. Now suppose we were told by an oracle the exact distribution of $Y_{ij}$ within each group, so that we know both its mean and variance for every $i$. (This is in fact the maximum we could ever hope to learn from the data.) Can we recover the model parameters? We cannot, because there's an infinite number of ways we could specify $\mu, \alpha_1, \ldots , \alpha_k$ so that $\text{E}(Y_{ij}) = \mu + \alpha_i$ for each $i$. This would show up in the likelihood function as well, where different parameters sets would give exactly the same likelihood for all possible configurations of the data. The model is not identifiable, and we can't obtain even consistent estimates for any of the mean parameters. For this reason one usually imposes the identifiability constraint $\sum_{i=1}^{k} \alpha_i = 0$.

So while it's important that the parameters of the model specify the distributions involved, it's also important that we be able to go in the other direction and infer parameters from distributions, else we could never uncover the "true" model.

$\endgroup$
1
  • $\begingroup$ Nevertheless, even in an unidentifiable model the likelihood is a function of all its arguments: for any given set of data and any given set of parameter values, the likelihood has a unique value. That's the very definition of a function. Please read the other answers to this question for more about this. $\endgroup$
    – whuber
    Jul 7, 2015 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.