# Does comparing Shannon Entropy between categorical variables makes any sense?

I want to present a report that highlights the variables with lowest information, to emphasize that some action must be taken by the department that controls our data source. I've applied Shannon Entropy to every categorical variable in my dataset, but the problem is that not all the variables have the same number of classes, making me think that direct comparisson would be wrong.

Is there a way to compare the entropy between variables with different number of classes? Which alternatives can I apply to overcome this problem? Maybe some adjusted entropy or another measure of variability within a factor.

• On what grounds are you concluding that low-entropy variables require some sort of action? Jun 1 at 23:05
• In my organization, people aren't filling the ERP with proper data, producing homogeneity across some variables, i.e. lower entropy. Is not that lower entropy is the cause, but it's a symptom. Jun 4 at 11:59
• Ah. It sounds like you might want to look at the number of bits below maximal entropy for each variable? In that case you should probably look at $log(n) - H(X)$ where $n$ is the number of possible values that $X$ can take. Jun 5 at 13:16
• Can that be interpreted as information loss? Jun 6 at 3:22

I did an adjustment to Shannon's formula, where $$X$$ is the random variable of study:
$$H(X)=-\sum_i p_i \times \log_2 pi \\ H(X)_{adj}= H(X) \times \frac{1}{\log_2n}$$
Where $$n \in \mathbb{N} - \{1\}$$ is the number of levels of $$X$$ and $$p_i$$ represents the probabilities of $$X = x_i$$, $$\forall i =1,\dots,n$$.