I have a highly non-linear (lots of jumps) likelihood function with K parameters (For example, a marked Hawkes Process used in seismology study). I implemented the L-BFGS-B optimization routine and it returns a Hessian matrix that has elements close to zero. As a result, several estimators of this likelihood function have large estimated standard errors.

Maybe the best way to describe the problem is that I have a Hessian that is nearly singular in the sense that its smallest eigenvalue is very small relative to its largest one.

I believe it is a optimization problem rather than a modeling problem and come up with an idea that in the first step, I fix some parameters with initial values, say a1,a2 and a3, and optimization others,say a4~a8. Once I got these estimators, I in turn fix them and estimating those who are fixed at first place (a1~a3). Iterating these two steps until all of them converge to a set of stable estimators.

My questions are 1) Is this idea legitimate ? i.e. Are estimators consistent ? 2) If it is legitimate, should I use 8*8 Hessian matrix to compute the variance-covariance matrix or I use 3*3 Hessian matrix to calculate the variance-covariance matrix for the first estimators and then use 5*5 Hessian matrix to calculate the variance-covariance matrix for the other estimators?

  • 1
    $\begingroup$ Leaving aside the matter of alternately optimizing subsets of parameters (I don't know why you do that, just let BFGS-B do all of them at once), the Hessian approximation produced by L-BFGS-B may bear little resemblance to the actual Hessian - it serves to facilitate the optimization, but may be a terrible estimate of the Hessian at the optimum. You may be better off using a BFGS, not l-BFGS (don't know if it will make a difference in your case). L is only needed if there are a very large number of parameters, which 8 is not. If you can directly compute the Hessian at the optimum, use that. $\endgroup$ – Mark L. Stone Oct 18 '17 at 15:32
  • $\begingroup$ The reason I did this subset optimization is inspired by the grid search argument. Also I tried BFGS, not so many differences with L-BFGS-B, but thanks for the clarification ! $\endgroup$ – skyindeer Oct 18 '17 at 15:35
  • $\begingroup$ If doing (L-)BFGS, optimize all parameters at once. $\endgroup$ – Mark L. Stone Oct 18 '17 at 15:39
  • $\begingroup$ Optimizing all parameters simultaneously leads to some estimators have large estimated standard errors. I suspect the reason for such situation is because I have highly non-linear object function, and there might be bounder issues which lead to unstable results. $\endgroup$ – skyindeer Oct 18 '17 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.