6
$\begingroup$

How minimizing the determinant of the information matrix is equivalent to maximizing the differential Shannon entropy? A similar question was posted in Math SE but hasn't been rigorously answered.

My understanding is: By minimizing the determinant of information matrix, we are trying to minimize the deviation of our estimator from the true distribution because ideally a 0 determinant implies singularity or the 0 deviation of the estimator from the actual distribution. (Is this right understanding? If so then the following question arises)

Question: How does this relate to maximizing differential Shannon entropy? How are we maximizing information?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Information is considered the inverse of uncertainty. If variance is big, information is small. Differential Shannon entropy, aka Kullback-Liebler divergence, is going to be zero when the distros are identical. Similarly the error is going to be zero when they are identical. They are not the same when they are not zero. A 10% error, whatever that means, is not necessarily the same as a 10% KL-divergence. $\endgroup$ – EngrStudent Sep 24 '20 at 13:32
  • $\begingroup$ @EngrStudent This makes sense, but how is determinent of information matrix related to this? So information matrix is inverse of variance matrix, but what exactly does "minimizing determinant of the information matrix" actually mean? $\endgroup$ – GENIVI-LEARNER Sep 24 '20 at 13:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.