# Fisher information for MLE with constraint

Supposing I have a probability distribution $f(x|\vec\theta)$, where $x$ is a random variable and $\vec\theta$ is a vector of distribution parameters. I also know that parameters $\vec\theta$ should satisfy some condition that could be expressed with a function $g(\vec\theta) = 0$.

After $n$ measurements of $x$ with results $x_1, x_2, \dots, x_n$ I can estimate $\vec\theta$ using the maximum likelihood estimation (MLE). To satisfy the constraint $g(\vec\theta) = 0$ I'm using the method of Lagrange multipliers. So the task is to find a maxima of a function $$F(\vec\theta) = \sum_{i=1}^n{\log f(x_i|\vec\theta)} + \lambda g(\vec\theta), \tag{1}$$ where $\lambda$ is a Lagrange multiplier.

Now I want to estimate the Fisher information matrix for parameters $\vec\theta$, which is $$I_{ij} = -\mathbb{E}\left[\frac{\partial^2}{\partial\theta_i\partial\theta_j}\log f(x|\vec\theta) \right]. \tag{2}$$ As far as I understand Fisher information describes the sharpness of a likelihood function maxima. But as I perform the maximization with constraint $g(\vec\theta) = 0$ using (1), should I actually estimate the Fisher information matrix as $$I_{ij} = -\mathbb{E}\left[\frac{\partial^2}{\partial\theta_i\partial\theta_j} F(x,\vec\theta) \right], \quad F(x,\vec\theta) = \log f(x|\vec\theta) + \lambda g(\vec\theta) \tag{3}$$ instead of (2)?

## 2 Answers

Constraints greatly complicate the usage of Fisher's information with asymptotic theory for derivation of an estimator's distribution. Two general tips:

1.) If you are dealing with equality constraints (which seems like the case in OP's problem), you can often use a change of parameterizations such that the parameter lies on a lower dimensional space without constraints. This is recommended.

2.) If you are dealing with inequality constraints, and your results imply you are far from the boundary (i.e., you estimate a probability as $$\hat p = 0.25$$ with standard error $$SE(\hat p) = 0.05$$), the issue can likely be safely ignored.

I think not. The Fisher Information is a function of $\theta$, so it specifies what the what kind of performance you can expected of your estimator given a value of $\theta$. In some cases the FI depends on $\theta$, in some cases it does not. I don't think having a constraint on $\theta$ changes that.

What I would recommend however, is to look into Bayesian MMSE estimators. The advantage of the Bayesian approach is that they allow you to use any prior knowledge about the parameter (in your case $g(\theta) = 0$) to improve the estimator. There is a good explanation with an example of a constrained estimator for a DC level in noise in this lecture from the Estimation and Detection course of the TU Delft.. In know it's quite a different approach from solving the Lanrange dual problem as you are trying now, but might be worth having a look!

• What about estimation of mean value $\mu$ for Normal distribution with standard deviation $\sigma$? It doesn't depend on $\mu$ but if I, for example, put constraint that $\mu \in [-0.1;0.1]$ the accuracy of estimation significantly changes for high $\sigma$. P.S. Thanks for the recommendation, I will take a look at Bayesian MMSE. Sep 10, 2018 at 10:41
• The ML estimator for the mean is $1/N \sum x$. How would you incorporate the constraint in your estimator? If you cast all values outside the range, then the variance is indeed at most $0.1^2$ in your example, but it is a step you apply after the estimation step, so it is not fair to compare them in that way. The FI assumes no such prior knowledge. You could use the BMMSE technique with the assumption that $\mu \sim U(-0.1, 0.1)$ however to achieve better results.
– RvdV
Sep 10, 2018 at 12:29
• "...it specifies what the what kind of performance you can expected of your estimator given a value of 𝜃...I don't think having a constraint on 𝜃 changes that.". It does. And it's really hard to deal with in general. Mar 17, 2020 at 23:26