How do we Deal with Identifiability Problems in Statistics?

Question

In statistics, are there any common strategies to deal with non-identifiable models?

For example, I have heard that mixture models (i.e. based on weighted sums of normal probability distributions) can be quite useful at modelling complex datasets believed to contain different underlying populations - but on the other hand, they can also be "non-identifiable" (i.e. the likelihood equations corresponding to these models can have multiple possible solutions).

In general, are there any "strategies" to deal with this problem? For instance, can the likelihood function in these cases be somehow modified (e.g. adding some constraint) to make it identifiable? Or is this problem of non-identifiability not that serious and can be overlooked?

Answer 1

There are different steps to deal with the issue of identifiability.

The first one is to make sure that your issue is related to practical identifiability, and not structural identifiability. The latter refers to an intrinsic property of a model, for which different combinations of its parameters would yield exactly the same output. For instance, if the relation between the input $x$ and the output $y$ is $y = (A+B)x$ (with free parameters $A$ and $B$), then the model is said to be structurally non-identifiable. The former refers to a property of your experimental setting, in which the parameters of an otherwise structurally identifiable model cannot be accurately determined due to observations being scarce and/or noisy, for instance. Structurally non-identifiable models should be avoided. Different methods exist to assess the structural identifiability of a model, e.g.:

Massonis, G., & Villaverde, A. F. (2020). Finding and breaking Lie symmetries: implications for structural identifiability and observability in biological modelling. Symmetry, 12(3), 469.

For practical identifiability, it then all depends on your set up:

• 1st possibility: you have one statistical model, but the possibility to acquire more data. You may be only interested in a specific model, which parameters you try to infer from a set of data, but these said data yield unreliable or inaccurate estimate of the parameters (in the sense that different combinations of parameters will yield the same likelihood). The best step is then to acquire more or more informative data, through processes known as Optimal Experiment Design or Bayesian Active Learning. The points of these methods is to obtain data that will be the most informative about the parameters you are trying to infer and are more likely to lead you to their ground-truth values. The following paper sums up different methods and reviews previous applications: https://arxiv.org/pdf/2201.07539.pdf

• 2nd possibility: you can choose from different possible models to fit your data, but cannot acquire more data. In this situation, performing model selection will guide you to the candidate model having the best explanatory power while minimizing its risk of overfitting and its number of free parameters (and hence guiding you to a unique set of estimated parameters). Model selection can be performed either

o Using cross validation, see chapter 1.6 of the following thesis: https://core.ac.uk/reader/535018070

o Using criteria such as the AIC or the BIC, as described in the following paper: Gontier, C., & Pfister, J. P. (2020). Identifiability of a binomial synapse. Frontiers in computational neuroscience, 14, 558477.

Answer 2

Identifiability is ultimately about what can be known from (ideally infinitely many) observations. If a model parameter is not identifiable, there is no way to estimate it. Or rather, it may be possible to get an estimate, but even if this is based on a very large sample, it doesn't allow you to tell apart parameter values close to the estimator from other parameter values that are quite far away.

An identifiability problem always means that you can't know certain things. And it is always good for understanding a problem that you know whether there are identifiability issues.

Examples: A mixture model has the form $P=\sum_{i=1}^k \pi_i P_{\theta_i}$, where the $\pi_i\ge 0$ are proportions ($\sum \pi_i=1$) and $P_{\theta},\ \theta\in\Theta$ are distributions from a parameterised family with parameter values in some set $\theta$.

An elementary identifiability problem with mixtures is that the component numbers cannot be identified. If you have a mixture of ${\cal N}(0,1)$ and ${\cal N}(5,1)$, it doesn't make a difference which one of the parameters (say 0 and 5, assuming for simplicity that we know that all component variances are 1) you call $\theta_1$ and which one you call $\theta_2$. So technically you can't identify $\theta_1$, as this could be either of the two, and no amount of data will change that.

Now you might say, fair enough, but for me it is good enough to know that the two parameters are 0 and 5; I don't mind which one is number 1 and which one is number 2. And this is very sensible. The "solution" here is to say, the "parameter of interest" is the set of the two means, 0 and 5, but not the numbering of components (you might also define the problem in such a way that the smaller mean always belongs to mixture component 1 and the larger one to component 2, then the numbering is fixed and unambiguous). This paper...

Redner, R. (1981). Note on the consistency of the maximum likelihood estimate for non-identifiable distributions. The Annals of Statistics 9, 225-228. [link]

...shows that even if you can't estimate non-identifiable parameters consistently, you can estimate the equivalence class of all parameter values that give rise to the same distribution and can therefore not be told apart. This means that you may not be able to identify a certain parameter, but you can identify the set of possibilities.

This however only helps if you have a proper knowledge about the equivalence class so that you know what kind of information you get from an estimator in such a situation.

In fact the "label switching" issue regarding mixtures defined above is mostly harmless as we will not normally be interested in interpreting the component numbering (it can lead to problems with computational methods though; it is known to create trouble for Markov Chain Monte Carlo algorithms if you try to estimate the parameters and their uncertainty in a Bayesian way).

There are less harmless issues in mixture modelling. For example, if $\pi_i=0$, changing $\theta_i$ doesn't change the model and can therefore not be identified, and if $\theta_i=\theta_j$, you can identify $\pi_i+\pi_j$, but you can distribute the sum freely over $\pi_i$ and $\pi_2$. You can "forbid" these problems by assuming that $\pi_i>0,\ \theta_i\neq\theta_j$ for all $i,j$, but they still have practical implications that won't go away that easily. For example, assumptions for standard maximum likelihood theory will be violated, and if you want to estimate the number of mixture components $k$, you will find distributions that are arbitrarily close to a mixture with $k=k_0$ components that have a much larger $k=k^*>k_0$ just by having very small values of $\pi_i$ and/or $\theta_i,\theta_j$ very close. This means that the estimation of $k$ is an ill-posed problem, and for large enough data in practice larger and larger $k$ cannot be told apart from any "true" lower $k_0$. (This is in practice often dealt with using penalties such as the BIC, but to the extent that these rely on model assumptions, for really large data, if in reality the model assumptions are not perfectly fulfilled, a model with larger $k$ will still eventually look better. Ultimately we can only specify a penalty and say, if larger $k$ only looks a little better, we prefer the smaller one; any theoretical basis for this is rather shaky; at best it relies critically on model assumptions being precisely fulfilled, which isn't the case in reality.)

Even more problematic would it be if even imposing conditions as mentioned already, mixtures defined by "really" different parameters would yield the same distribution. For example, if (now allowing also the Gaussian variances to differ), the $0.5{\cal N}(0,1)+0.5{\cal N}(0,10)$ mixture distribution could equally well be written as $0.2{\cal N}(0,0.5)+0.8{\cal N}(0,8)$, you couldn't interpret the mixture proportions (as these may be $(0.5; 0.5)$ as well as $(0.2;0.8)$ giving rise to the same distribution, i.e., observed data will look the same), and neither the variances. This would really be a problem if you'd want to interpret results saying that "there are two groups in the data, both with (roughly) the same proportion", because the data, even arbitrarily large data sets wouldn't allow you to say such a thing if identifiability were not fulfilled.

Now in fact Gaussian mixtures are identifiable (due to a non-trivial mathematical theorem from the 1960s), so you will not run into this problem, but this doesn't apply to all kinds of mixtures. For example mixtures of uniform distributions are not identifiable, so you can have mixtures that look properly different in terms of the parameters but are actually the same.

If you don't know this, you may well overinterpret your results, interpreting features of the data that in fact cannot be identified, i.e., cannot be known. Estimating an equivalence class as in the Redner paper cited above will not help you if you want to give the specific parameter values an interpretation.

Another example for the same issue is if you have a regression model $Y=\beta_0+\beta_1x_1+\beta_2x_2+\epsilon$ and the variables $x_1$ and $x_2$ are perfectly correlated. If you want to make statements about the relative influence of $x_1$ and $x_2$ on $Y$, there is no way to do this as whatever can be explained from $x_1$ can also be explained from $x_2$. You can have numerical methods (such as ridge regression) that enforce a solution, i.e., distribute the influence of $x_1$ and $x_2$ together in an enforced way on $\beta_1$ and $\beta_2$, however this doesn't help if you can't identify what you are actually interested in.

Allow me to link a recent paper of mine in which I show that in a situation in which you want to assume Gaussian observations to be i.i.d., you cannot tell from any amount of data whether there is actually constant larger than zero correlation between any two observations:

C. Hennig (2023) Parameters not empirically identifiable or distinguishable, including correlation between Gaussian observations. Statistical Papers [link]

Baseline: There are many identifiability problems, many much more convoluted than those discussed here. It is always worthwhile to know about such issues. They may be harmless and allow for a solution, however they may also mean that what you actually want to know cannot be known from the data, whatever you do. In which case you better accept that you can't know this and look for other problems to solve...