Fisher information for MLE with constraint

Question

Supposing I have a probability distribution $f(x|\vec\theta)$, where $x$ is a random variable and $\vec\theta$ is a vector of distribution parameters. I also know that parameters $\vec\theta$ should satisfy some condition that could be expressed with a function $g(\vec\theta) = 0$.

After $n$ measurements of $x$ with results $x_1, x_2, \dots, x_n$ I can estimate $\vec\theta$ using the maximum likelihood estimation (MLE). To satisfy the constraint $g(\vec\theta) = 0$ I'm using the method of Lagrange multipliers. So the task is to find a maxima of a function $$ F(\vec\theta) = \sum_{i=1}^n{\log f(x_i|\vec\theta)} + \lambda g(\vec\theta), \tag{1} $$ where $\lambda$ is a Lagrange multiplier.

Now I want to estimate the Fisher information matrix for parameters $\vec\theta$, which is $$ I_{ij} = -\mathbb{E}\left[\frac{\partial^2}{\partial\theta_i\partial\theta_j}\log f(x|\vec\theta) \right]. \tag{2} $$ As far as I understand Fisher information describes the sharpness of a likelihood function maxima. But as I perform the maximization with constraint $g(\vec\theta) = 0$ using (1), should I actually estimate the Fisher information matrix as $$ I_{ij} = -\mathbb{E}\left[\frac{\partial^2}{\partial\theta_i\partial\theta_j} F(x,\vec\theta) \right], \quad F(x,\vec\theta) = \log f(x|\vec\theta) + \lambda g(\vec\theta) \tag{3} $$ instead of (2)?

Answer 1

Constraints greatly complicate the usage of Fisher's information with asymptotic theory for derivation of an estimator's distribution. Two general tips:

1.) If you are dealing with equality constraints (which seems like the case in OP's problem), you can often use a change of parameterizations such that the parameter lies on a lower dimensional space without constraints. This is recommended.

2.) If you are dealing with inequality constraints, and your results imply you are far from the boundary (i.e., you estimate a probability as $\hat p = 0.25$ with standard error $SE(\hat p) = 0.05$), the issue can likely be safely ignored.

Answer 2

I think not. The Fisher Information is a function of $\theta$, so it specifies what the what kind of performance you can expected of your estimator given a value of $\theta$. In some cases the FI depends on $\theta$, in some cases it does not. I don't think having a constraint on $\theta$ changes that.

What I would recommend however, is to look into Bayesian MMSE estimators. The advantage of the Bayesian approach is that they allow you to use any prior knowledge about the parameter (in your case $g(\theta) = 0$) to improve the estimator. There is a good explanation with an example of a constrained estimator for a DC level in noise in this lecture from the Estimation and Detection course of the TU Delft.. In know it's quite a different approach from solving the Lanrange dual problem as you are trying now, but might be worth having a look!