I want to present a report that highlights the variables with lowest information, to emphasize that some action must be taken by the department that controls our data source. I've applied Shannon Entropy to every categorical variable in my dataset, but the problem is that not all the variables have the same number of classes, making me think that direct comparisson would be wrong.

Is there a way to compare the entropy between variables with different number of classes? Which alternatives can I apply to overcome this problem? Maybe some adjusted entropy or another measure of variability within a factor.

  • $\begingroup$ On what grounds are you concluding that low-entropy variables require some sort of action? $\endgroup$ Jun 1 at 23:05
  • $\begingroup$ In my organization, people aren't filling the ERP with proper data, producing homogeneity across some variables, i.e. lower entropy. Is not that lower entropy is the cause, but it's a symptom. $\endgroup$ Jun 4 at 11:59
  • 1
    $\begingroup$ Ah. It sounds like you might want to look at the number of bits below maximal entropy for each variable? In that case you should probably look at $log(n) - H(X)$ where $n$ is the number of possible values that $X$ can take. $\endgroup$ Jun 5 at 13:16
  • $\begingroup$ Can that be interpreted as information loss? $\endgroup$ Jun 6 at 3:22

I did an adjustment to Shannon's formula, where $X$ is the random variable of study:

$$H(X)=-\sum_i p_i \times \log_2 pi \\ H(X)_{adj}= H(X) \times \frac{1}{\log_2n}$$

Where $n \in \mathbb{N} - \{1\}$ is the number of levels of $X$ and $p_i$ represents the probabilities of $X = x_i$, $\forall i =1,\dots,n$.

Please let me know if I'm making a mistake or if there's a formal adjusted Shannon Entropy that I'm missing.

  • 1
    $\begingroup$ You can edit this into your original post. Please reserve "answers" for answers. $\endgroup$
    – Dave
    Jun 1 at 18:37
  • $\begingroup$ This is essentially calculating the fraction of maximal entropy. This may or may not be appropriate - I think you should add some more details to the question to help people determine this. $\endgroup$ Jun 5 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.