I`m using the Newton-Raphson method for obtaining MLE for parameters for maximizing my objective function.
At each iteration, I want to check that is the Hessian matrix negative definite or not and I see the Hessian matrix is not negative definite at some iterations. So, I want to use the Fisher scoring algorithm (https://en.wikipedia.org/wiki/Scoring_algorithm#Sketch of derivation)
In this situation, the Fisher information matrix is indefinite!! Is it possible? What are the reasons for this? Are the second-order derivatives wrong? Or are values of expectations incorrect?
$\begingroup$
$\endgroup$
6
-
2$\begingroup$ Could you clarify what you mean by "Fisher information matrix"? As usually defined, this doesn't depend (in any way) on "iterations" towards a solution: it depends only on the likelihood function at the optimal parameter. $\endgroup$– whuber ♦Commented Jun 2, 2021 at 16:04
-
$\begingroup$ I can refer you to this link : en.wikipedia.org/wiki/Scoring_algorithm. I mean expected the observed information matrix calculated at obtained parameter value in previous iteration. $\endgroup$– user321525Commented Jun 2, 2021 at 16:35
-
$\begingroup$ Thanks. That's a gradient descent algorithm. Refer to posts about these algorithms for the (many) problems they can have. $\endgroup$– whuber ♦Commented Jun 2, 2021 at 19:10
-
$\begingroup$ @whuber Thank you. $\endgroup$– user321525Commented Jun 3, 2021 at 5:48
-
$\begingroup$ You need to be aware of the difference between the Fisher Information and the Observed Fisher Information, see for discussion stats.stackexchange.com/questions/154724/… and stats.stackexchange.com/questions/188251/… $\endgroup$– kjetil b halvorsen ♦Commented Jun 3, 2021 at 10:12
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
4
From https://en.wikipedia.org/wiki/Observed_information the observed Fisher information matrix is just the negative Hessian of the log likelihood function. If your log likelihood function is not convex then the hessian will not be positive-definite (and thus indefinite).
This is the case if you are using the "Observed Fisher Information Matrix" from https://en.wikipedia.org/wiki/Scoring_algorithm. If you are using the canonical notion of the Fisher Information Matrix (https://en.wikipedia.org/wiki/Fisher_information#Matrix_form) then it must be positive semi-definite.
-
$\begingroup$ Thanks a lot. At the first iteration in Newton-Raphson method, I see that the observed fisher information matrix (negative Hessian) from (en.wikipedia.org/wiki/Scoring_algorithm ) is positive definite (for maximizing problem) and after the first iteration it becomes indefinite!! In this situation, Is it correct that I say my log likelihood function is not concave? Does it depend on staring values? $\endgroup$ Commented Jun 3, 2021 at 5:47
-
$\begingroup$ Yes it is correct that your log likelihood function is not concave; if it were concave then it would be everywhere negative semi-definite. A function is concave if it is everywhere negative semi-definite (and convex for everywhere positive semi-definite). This does not depend on the starting value for the algorithm. $\endgroup$– JspangCommented Jun 3, 2021 at 15:19
-
$\begingroup$ It's worth noting that since your function is not convex, this algorithm will (almost certainly) not find the global minimum; instead I would recommend some sort of stochastic descent algorithm if a local minimum will not suffice for your use-case. $\endgroup$– JspangCommented Jun 3, 2021 at 15:23
-