38
$\begingroup$

I know that with a model that is not identifiable the data can be said to be generated by multiple different assignments to the model parameters. I know that sometimes it's possible to constrain parameters so that all are identifiable, as in the example in Cassella & Berger 2nd ed, section 11.2.

Given a particular model, how can I evaluate whether or not it's identifiable?

$\endgroup$
46
$\begingroup$

For identifiability we are talking about a parameter $\theta$ (which could be a vector), which ranges over a parameter space $\Theta$, and a family of distributions (for simplicity, think PDFs) indexed by $\theta$ which we typically write something like $\{ f_{\theta}|\, \theta \in \Theta\}$. For instance, $\theta$ could be $\theta = \beta$ and $f$ could be

$$ f_{\theta}(x) = \frac{1}{\beta}\mathrm{e}^{-x/\beta}, \ x>0,\ \beta >0, $$ which would mean that $\Theta = (0,\infty)$. In order for the model to be identifiable, the transformation which maps $\theta$ to $f_{\theta}$ should be one-to-one. Given a model in your lap, the most straightforward way to check this is to start with the equation $f_{\theta_{1}} = f_{\theta_{2}}$, (this equality should hold for (almost) all $x$ in the support) and to try to use algebra (or some other argument) to show that just such an equation implies that, in fact, $\theta_{1} = \theta_{2}$.

If you succeed with this plan, then your model is identifiable; go on with your business. If you don't, then either your model isn't identifiable, or you need to find another argument. The intuition is the same, regardless: in an identifiable model it is impossible for two distinct parameters (which could be vectors) to give rise to the same likelihood function.

This makes sense, because if, for fixed data, two unique parameters gave rise to the same likelihood, then it would be impossible to distinguish between the two candidate parameters based on the data alone. It would be impossible to identify the true parameter, in that case.

For the example above, the equation $f_{\theta_{1}} = f_{\theta_{2}}$ is $$ \frac{1}{\beta_{1}}\mathrm{e}^{-x/\beta_{1}} = \frac{1}{\beta_{2}}\mathrm{e}^{-x/\beta_{2}}, $$ for (almost) all $x > 0$. If we take logs of both sides we get $$ -\ln\,\beta_{1} - \frac{x}{\beta_{1}} = -\ln\,\beta_{2} - \frac{x}{\beta_{2}} $$ for $x > 0$, which implies the linear function $$ -\left(\frac{1}{\beta_{1}} - \frac{1}{\beta_{2}}\right)x - (\ln\,\beta_{1} - \ln\,\beta_{2}) $$ is (almost) identically zero. The only line which does such a thing is the one which has slope 0 and y-intercept zero. Hopefully you can see the rest.

By the way, if you can tell by looking at your model that it isn't identifiable (sometimes you can), then it is common to introduce additional constraints on it to make it identifiable (as you mentioned). This is akin to recognizing that the function $f(y) = y^{2}$ isn't one-to-one for $y$ in $[-1,1]$, but it is one-to-one if we restrict $y$ to lie inside $[0,1]$. In more complicated models the equations are tougher but the idea is the same.

$\endgroup$
  • 2
    $\begingroup$ (+1) Nice, comprehensive, down-to-earth explanation. The analogies you draw make the concepts clear. $\endgroup$ – cardinal Jan 5 '12 at 12:09
  • $\begingroup$ You certainly answered the question I asked, but I'm too much of a novice to really understand your answer. If you know of an explanation that's better for a novice, please let me know. $\endgroup$ – Jack Tanner Jan 5 '12 at 17:19
  • 1
    $\begingroup$ @cardinal, thanks. To Jack, alright, I see. How about this: if there's something above that isn't clear yet, and if you point it out to me, then I can try to flesh it out some more. Or, if you'd prefer, you could write another question that asks for a "layman's" explanation or examples of these ideas. I think it's fair to say that identifiability is a topic that usually comes up after the typical introductory period of study, so if you'd like to provide some context of why you're encountering this now it might help potential answerers. $\endgroup$ – user1108 Jan 5 '12 at 18:42
  • 2
    $\begingroup$ +1, nice answer. It might be worth pointing out one classic & easy to see example of an unidentifiable model is the unconstrained version of the ANOVA: $$y_{ij}=\mu+\alpha_1+\alpha_2+\ldots+\alpha_k+\varepsilon_i$$ To remedy this, reference cell coding is typically used, wherein the mean of one level is set as the reference (which is estimated by the intercept), & the grand mean is not explicitly estimated. $\endgroup$ – gung - Reinstate Monica Dec 25 '12 at 5:14
23
$\begingroup$

One way is to inspect the covariance matrix, $\Sigma$, of your parameter estimates. If two parameter estimates are perfectly (approximately) correlated with each other or one parameter estimate is a (approximately) linear combination of several others, then your model is not identified; the parameters that are functions of the others are not necessary. In each of these cases, $\Sigma$ will also be (approximately) singular. So, if $\Sigma$ is approximately singular, this may give you reason to be concerned about identifiability issues. (Although I don't think this would detect non-linear relationships between parameter estimates that would give rise to non-identifiability).

The practical problem is that it is often difficult to calculate $\Sigma$ for even mildly complicated models.

If you are doing a maximum likelihood problem, then you know the asymptotic covariance matrix of your estimates is equal to the inverse of the fisher information evaluated at the MLE. So, checking the fisher information matrix for (approximate) singularity is also a reasonable way of assessing identifiability. This also works where the theoretical fisher information is difficult to calculate because it is often possible to very accurately numerically approximate a consistent estimator of the fisher information matrix by, for example, estimating the expected outer product of the score function by the observed average outer product.

In you are not doing an ML problem then you may be able to get a handle on $\Sigma$ by simulating data from the model and estimating parameters a large number of times and calculating a sample covariance matrix.

$\endgroup$
  • 2
    $\begingroup$ (+1) Well done. I hadn't even thought to approach this question from that direction. $\endgroup$ – user1108 Jan 5 '12 at 18:50
  • 1
    $\begingroup$ One reason the idea about calculating a covariance matrix based on simulated data is especially neat, is that one should simulate the data anyway to do a Cook-Gelman-Rubin check. $\endgroup$ – Jack Tanner Jan 5 '12 at 22:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.