I am interested in finding the difference of entropy between two states of the same variable. The probability distribution associated with the variable X changes at each discrete point in time t by an increment in p(i). The current state is therefore always an update of the former state as we advance each step along the message. I am applying the equation below, where t is a multiplier to make the difference of entropy proportional to time (this is part of a study on statistical memory, so such proportionality gives a weight to events happening further along in a message).


My question is: what is I? I am inclined to call it information gain or information divergence, but the literature seems to be highly divergent (no pun intended) on the nomenclature. Of course I could be something different entirely (covariance?).

This is a question from someone with no formal background in statistics or information theory, so please bear with me while I try to make sense of my problem. Any clarification or suggestion will be highly appreciated.

  • $\begingroup$ What are you actually looking for? The evident answer to "what is $I$" is that it's a weighted difference of entropies! What additional, useful information would any other possible answer supply? What actually is the "problem" to which you refer? $\endgroup$ – whuber Jun 27 '17 at 16:56
  • $\begingroup$ Dear @whuber, thanks for your reply. My problem is, well, how to name the thing. I seems to be an instance of a KL divergence or information gain, depending on the source I check. Would either be the case or is this too long a shot? Finding the difference between contiguous entropy states is useful for my practical purposes – but if this is something already backed up by a conceptual framework, I'll have something specific to look up. $\endgroup$ – Chris75 Jun 27 '17 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.