3
$\begingroup$

I have seen formulas that either sum or subtract entropy values (eg Information Gain). However, I have not seen a "multiplication/division of entropy values" anywhere. I would like to know if there is any formula/theorem that multiplies/divides entropy values. Additionally, I would like to know if there is any mathematical reasoning/meaning behind such operation. Thanks.

$\endgroup$
  • $\begingroup$ This sounds like a (speculative, unmotivated) answer in search of a question. What problem are you trying to solve here? $\endgroup$ – whuber Feb 9 '13 at 15:30
  • $\begingroup$ @whuber just curiosity. $\endgroup$ – AM2 Feb 12 '13 at 23:35
5
$\begingroup$

I couldn't think of any situations where I would multiply entropy values, and skimming the beginning of Elements of Information Theory didn't reveal any either. This isn't terribly surprising, as I cannot imagine too many situations where a quantity with units "bits squared" would be useful. (This is actually a pretty common situation--we often add or subtract temperatures, but rarely multiply them by other temperatures).

On the other hand, it is not unheard of to divide entropy values. If H(X) / H(Y) = 2, you could plausibly claim that X carries twice the information of Y. A similar calculation is used when computing the C4.5's information gain ratio, which it uses for building/pruning decision trees.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.