# Enforce identifiability on model predictioins

I have a model (a neural network) which produces estimates for the parameters of latent random variables (e.g. the $$\lambda$$ param for an exponential distribution).

I don't observe the r.v. directly, but the result of another process which acts on pairs of variables. I then compute the MLE and do usual SGD.

Let's assume that the MLE is in the form $$\lambda_1 - \lambda_2$$ for a pair of observations. This is problematic because the parameters are not identifiable; the model can learn an arbitrary number $$b$$ and emit $$\lambda + b$$ instead of $$\lambda$$, and MLE is exactly the same. So the parameters are not identifiable, only the difference between the parameters are.

This is a problem when you are e.g. doing ensembling, in which N models are trained and the results averaged. If each model is free to learn a different $$b$$, it's not clear that it makes sense to average the parameter estimates (or compute std. deviation, etc.)

In this case we could solve the problem by adding a small regularization loss on $$\lambda$$, to nudge all models to learn $$b = 0$$. However, how about more complex cases? Do you know of some general techniques that deal with this problem, or do you have links to some literature which have considered the problem?

I have been asked to elaborate on more complex cases. Well, let's assume that the MLE is in a complex form like

$$\frac 1Z B\left(\frac{\lambda_1 + \lambda_2}{\lambda_1 \cdot \lambda_2}, \lambda_1\cdot \lambda_2^p\right)$$

where B is e.g. beta function and $$Z$$ is a suitable integration factor. How do I know if the parameters are identifiable? I can try doing linear transformations like $$a\lambda_1 + b$$ and see what happens, but those are only linear. What about other transformations?

Even in the case of linear transformation, let's say I know that the model is free to learn $$a, b$$ s.t. $$a\lambda + b$$ translates into the same MLE. How do I regularize it? A loss towards $$0$$ will incentivize $$b$$ to be close to $$0$$, but it will also try to put $$a$$ as close to $$0$$ as possible. Before you know it your predictions are all around 1e-9 because the model can reduce $$a$$ as much as possible without hurting MLE loss.

I realise there can be no general technique which works in all cases and it's highly dependent on the specific situation, but I assume this problem has been analysed and discussed, so I am interested in pointers to such discussions.

Thanks!

• You ask "how about more complex cases?" but don't elaborate on how those cases are more complex & what your requirements are to solve this problem. Can you edit to explain what a more complex case would look like and the small regularization loss is not a solution to it?
– Sycorax
Commented Nov 17, 2022 at 18:42
• @Sycorax I have added more context, let me know if that's enough!
– Ant
Commented Nov 17, 2022 at 21:25
• SGD = stochastic gradient descent? Or something else? Commented Nov 18, 2022 at 4:07
• @Glen_b yes! Or any other optimization algorithm :)
– Ant
Commented Nov 18, 2022 at 13:35