4
$\begingroup$

Suppose you want an estimator for a parameter theta. What happens when two different values for theta yield the same likelihood? (I mean there are two equally likely values for theta that are more likely than every other.) Which is selected?

Would a Bayesian procedure yield a posterior distribution for theta that has two peaks?

Is there any example of such situation?

$\endgroup$
3
$\begingroup$

If two values of the parameter of interest have the same likelihood then it means that those values of theta are equally well supported by the data, according to the statistical model. It does not mean that those values are equally believable or that an inferential decision has to treat them the same.

Any Bayesian prior that gave more weight to one of the values than the other would give a posterior that renders one of the values more probable than the other.

You have not indicated the form of the likelihood function, or the nature of theta. It is possible for a likelihood function to be bimodal, but not every statistical model allows more than one peak and simple models involving the normal distribution will generally yield a bell-shaped likelihood function with a single mode. If theta is discrete then it is possible for two adjacent values to have the same likelihood even when the function has a single peak.

$\endgroup$
2
$\begingroup$

The definition of non-identifiability is when multiple values of $\theta$ yield the same likelihood value. A scenario where this occurs is when there is exact collinearity, e.g. ${\rm cor}(X_1, X_2) = 1$ and

$$ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \varepsilon, \ \ \ \ \ \ \ \ \ \varepsilon \sim N(0, \sigma^2) $$

Under that model, $\beta_1$ and $\beta_2$ are not uniquely identified (although $\beta_1 + \beta_2$ is); this means, for example, $\hat \beta_1 = 1, \hat \beta_2 = 1$ is not distinguishable from $\hat \beta_1 = 2, \hat \beta_2 = 0$ (i.e. those two parameter values would generate identical likelihood values).

(Note: in that case the fitting software typically would flag the solution as non-identified and therefore would drop one of the variables automatically)

The other poster is correct that using a prior (or a likelihood penalty in regularized frequentist estimation) can often help to resolve stuff like this. That's one reason why ridge regression is used to combat the instability generated by collinearity.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.