7
$\begingroup$

Many books and many posts on this site define the likelihood as a function of model parameters. However, does the output associated with every possible model parameter have to be unique? For example, it seems that for some two configurations of the model parameter, the observed data can be equally likely.

So, my question is whether we are playing fast and loose with the word "function" when talking about the likelihood or really the likelihood by definition has to be a function and each input for the model parameter must yield a unique $P(x|\theta)$?

$\endgroup$
  • $\begingroup$ @Xi'an: To pick on your correct answer, the likelihood function is really an equivalence class of functions, since there is an arbitrary (but positive) multiplicative constant. $\endgroup$ – kjetil b halvorsen Feb 21 '15 at 21:43
  • 2
    $\begingroup$ @Luca why shouldn't they? Take a constant function: each of its outputs is the same and it is most of the basic functions described in the textbooks. $\endgroup$ – Tim Feb 21 '15 at 21:51
  • 5
    $\begingroup$ I have overthought this. $\endgroup$ – Luca Feb 21 '15 at 21:57
  • 2
    $\begingroup$ @kjetilbhalvorsen: while I can see the motivations behind your remark (reference measure, likelihood principle, and all that), the likelihood function can be defined uniquely once the dominating measure on the sampling space is set. $\endgroup$ – Xi'an Feb 22 '15 at 6:29
  • 3
    $\begingroup$ Why the bayesian tag? $\endgroup$ – boscovich Feb 22 '15 at 7:53
18
$\begingroup$

Many books and many posts on this site define the likelihood as a function of model parameters.

If you specify a value for each of the parameters*, you will have at most one value for the likelihood.

  • (along with everything else you need to have specified, of course)

However, does the output associated with every possible model parameter have to be unique? For example, it seems that for some two configurations of the model parameter, the observed data can be equally likely.

You're confused -- take some function $f$ -- it's fine for $f(x_1)$ and $f(x_2)$ to be equal. $f(x)=(x-3)^2$ is a function, even though $f(2)=f(4)$.

That's two different arguments having the same function value, not the function having two different values for a given argument.

enter image description here

So, my question is whether we are playing fast and loose with the word "function" when talking about the likelihood

Nope

or really the likelihood by definition has to be a function and each input for the model parameter must yield a unique P(x|θ)?

Yes. But you seem to have gotten a little confused about what that means.

$\endgroup$
  • $\begingroup$ Thanks. I got confused with whether bijection was needed as well. $\endgroup$ – Luca Feb 22 '15 at 6:39
  • 7
    $\begingroup$ Yeah. I think this is sometimes a confusion for people, so it's a good question to have on site. The discussion between kjetil and Xi'an raises an important but somewhat more subtle issue than I think the one you were having. $\endgroup$ – Glen_b Feb 22 '15 at 6:40
10
$\begingroup$

A real-valued function $f$ associates to a vector or real entry $θ\in\Theta$ a real number $f(θ)$, that is, \begin{align*}f:\ \Theta &\longrightarrow \mathbb{R}\\ \theta &\longrightarrow f(\theta)\end{align*}Once the data $(x_1,\ldots,x_n)$ is observed and thus fixed, the likelihood associates with a given value of the parameter $θ$ the real number $$\prod_{i=1}^n p(x_i|\theta)$$ where $p(x|\theta)$ is the density of the random variable $X_i$. (I assume i.i.d.-ness in this answer to keep notations at a minimum complexity.) It is therefore a well-defined function in the mathematical sense.

$\endgroup$
1
$\begingroup$

This is closely related to the concept of identifiability in mathematical statistics, which I think relates to your question. Identifiability deals with the possibility that a poorly-specified model may lack a one-to-one relationship between parameter sets and probability distributions over the data, and this causes problems with regard to inference.

For instance, take the "over-parameterized" ANOVA model,

$$ Y_{ij} = \mu + \alpha_i + \epsilon_{ij} , $$

where $1 \leq i \leq k$, $1 \leq j \leq n$, $\epsilon_{ij} \sim$ normal$(0, \sigma^2)$, and no restrictions are placed on$\{ \alpha_i \}_{i=1}^{k}$. Now suppose we were told by an oracle the exact distribution of $Y_{ij}$ within each group, so that we know both its mean and variance for every $i$. (This is in fact the maximum we could ever hope to learn from the data.) Can we recover the model parameters? We cannot, because there's an infinite number of ways we could specify $\mu, \alpha_1, \ldots , \alpha_k$ so that $\text{E}(Y_{ij}) = \mu + \alpha_i$ for each $i$. This would show up in the likelihood function as well, where different parameters sets would give exactly the same likelihood for all possible configurations of the data. The model is not identifiable, and we can't obtain even consistent estimates for any of the mean parameters. For this reason one usually imposes the identifiability constraint $\sum_{i=1}^{k} \alpha_i = 0$.

So while it's important that the parameters of the model specify the distributions involved, it's also important that we be able to go in the other direction and infer parameters from distributions, else we could never uncover the "true" model.

$\endgroup$
  • $\begingroup$ Nevertheless, even in an unidentifiable model the likelihood is a function of all its arguments: for any given set of data and any given set of parameter values, the likelihood has a unique value. That's the very definition of a function. Please read the other answers to this question for more about this. $\endgroup$ – whuber Jul 7 '15 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.