3
$\begingroup$

Supposing I have a probability distribution $f(x|\vec\theta)$, where $x$ is a random variable and $\vec\theta$ is a vector of distribution parameters. I also know that parameters $\vec\theta$ should satisfy some condition that could be expressed with a function $g(\vec\theta) = 0$.

After $n$ measurements of $x$ with results $x_1, x_2, \dots, x_n$ I can estimate $\vec\theta$ using the maximum likelihood estimation (MLE). To satisfy the constraint $g(\vec\theta) = 0$ I'm using the method of Lagrange multipliers. So the task is to find a maxima of a function $$ F(\vec\theta) = \sum_{i=1}^n{\log f(x_i|\vec\theta)} + \lambda g(\vec\theta), \tag{1} $$ where $\lambda$ is a Lagrange multiplier.

Now I want to estimate the Fisher information matrix for parameters $\vec\theta$, which is $$ I_{ij} = -\mathbb{E}\left[\frac{\partial^2}{\partial\theta_i\partial\theta_j}\log f(x|\vec\theta) \right]. \tag{2} $$ As far as I understand Fisher information describes the sharpness of a likelihood function maxima. But as I perform the maximization with constraint $g(\vec\theta) = 0$ using (1), should I actually estimate the Fisher information matrix as $$ I_{ij} = -\mathbb{E}\left[\frac{\partial^2}{\partial\theta_i\partial\theta_j} F(x,\vec\theta) \right], \quad F(x,\vec\theta) = \log f(x|\vec\theta) + \lambda g(\vec\theta) \tag{3} $$ instead of (2)?

$\endgroup$
0
$\begingroup$

I think not. The Fisher Information is a function of $\theta$, so it specifies what the what kind of performance you can expected of your estimator given a value of $\theta$. In some cases the FI depends on $\theta$, in some cases it does not. I don't think having a constraint on $\theta$ changes that.

What I would recommend however, is to look into Bayesian MMSE estimators. The advantage of the Bayesian approach is that they allow you to use any prior knowledge about the parameter (in your case $g(\theta) = 0$) to improve the estimator. There is a good explanation with an example of a constrained estimator for a DC level in noise in this lecture from the Estimation and Detection course of the TU Delft.. In know it's quite a different approach from solving the Lanrange dual problem as you are trying now, but might be worth having a look!

$\endgroup$
  • $\begingroup$ What about estimation of mean value $\mu$ for Normal distribution with standard deviation $\sigma$? It doesn't depend on $\mu$ but if I, for example, put constraint that $\mu \in [-0.1;0.1]$ the accuracy of estimation significantly changes for high $\sigma$. P.S. Thanks for the recommendation, I will take a look at Bayesian MMSE. $\endgroup$ – Krivoi Sep 10 '18 at 10:41
  • $\begingroup$ The ML estimator for the mean is $1/N \sum x$. How would you incorporate the constraint in your estimator? If you cast all values outside the range, then the variance is indeed at most $0.1^2$ in your example, but it is a step you apply after the estimation step, so it is not fair to compare them in that way. The FI assumes no such prior knowledge. You could use the BMMSE technique with the assumption that $\mu \sim U(-0.1, 0.1)$ however to achieve better results. $\endgroup$ – RvdV Sep 10 '18 at 12:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.