1
$\begingroup$

I want to present a report that highlights the variables with lowest information, to emphasize that some action must be taken by the department that controls our data source. I've applied Shannon Entropy to every categorical variable in my dataset, but the problem is that not all the variables have the same number of classes, making me think that direct comparisson would be wrong.

Is there a way to compare the entropy between variables with different number of classes? Which alternatives can I apply to overcome this problem? Maybe some adjusted entropy or another measure of variability within a factor.

Improve this question
$\endgroup$
4
  • $\begingroup$ On what grounds are you concluding that low-entropy variables require some sort of action? $\endgroup$
    – user3716267
    Commented Jun 1, 2021 at 23:05
  • $\begingroup$ In my organization, people aren't filling the ERP with proper data, producing homogeneity across some variables, i.e. lower entropy. Is not that lower entropy is the cause, but it's a symptom. $\endgroup$
    – dzegpi
    Commented Jun 4, 2021 at 11:59
  • 1
    $\begingroup$ Ah. It sounds like you might want to look at the number of bits below maximal entropy for each variable? In that case you should probably look at $log(n) - H(X)$ where $n$ is the number of possible values that $X$ can take. $\endgroup$
    – user3716267
    Commented Jun 5, 2021 at 13:16
  • $\begingroup$ Can that be interpreted as information loss? $\endgroup$
    – dzegpi
    Commented Jun 6, 2021 at 3:22

1 Answer 1

Reset to default
1
$\begingroup$

I did an adjustment to Shannon's formula, where $X$ is the random variable of study:

$$H(X)=-\sum_i p_i \times \log_2 pi \\ H(X)_{adj}= H(X) \times \frac{1}{\log_2n}$$

Where $n \in \mathbb{N} - \{1\}$ is the number of levels of $X$ and $p_i$ represents the probabilities of $X = x_i$, $\forall i =1,\dots,n$.

Please let me know if I'm making a mistake or if there's a formal adjusted Shannon Entropy that I'm missing.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ You can edit this into your original post. Please reserve "answers" for answers. $\endgroup$
    – Dave
    Commented Jun 1, 2021 at 18:37
  • 1
    $\begingroup$ This is essentially calculating the fraction of maximal entropy. This may or may not be appropriate - I think you should add some more details to the question to help people determine this. $\endgroup$
    – user3716267
    Commented Jun 5, 2021 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.