7
$\begingroup$

I am working on thousands of datasets, many of them are "unbalanced" ; either a multi-class list with highly skewed distribution (For example three categories with ratio of 3500:300:4 samples) or a continues number with skewed distribution. I am looking for some metric that can say "How badly unbalanced" the dataset is. Is there such a meteric?

Eventually I want to score these datasets according to their balanced metric and provide a different balancing/ machine learning solution to each of them. I would prefer a python solution if exist.

Improve this question
$\endgroup$
2
  • $\begingroup$ If you use R, you can use the diversity() function in the 'vegan' package to calculate the Shannon-Weaver index. rdocumentation.org/packages/vegan/versions/2.4-2/topics/… $\endgroup$ – Zilu Liang Feb 1 at 4:42
  • $\begingroup$ It's not clear why you chose the word balance as if imbalance is a bad thing. If you use a method that depends on the amount of balance in levels of $Y$, switch to another method. $\endgroup$ – Frank Harrell Feb 7 at 11:59
15
$\begingroup$

You could use the Shannon entropy as a measure of balance.

On a data set of $n$ instances, if you have $k$ classes of size $c_i$ you can compute entropy as follows: $$ H = -\sum_{ i = 1}^k \frac{c_i}{n} \log{ \frac{c_i}{n}}. $$

This is equal to:

  • $0$ when there is one single class. In other words, it tends to $0$ when your data set is very unbalanced
  • $\log{k}$ when all your classes are balanced of the same size $\frac{n}{k}$

Therefore, you could use the following measure of Balance for a data set: $$ \mbox{Balance} = \frac{H}{\log{k}} = \frac{-\sum_{ i = 1}^k \frac{c_i}{n} \log{ \frac{c_i}{n}}. } {\log{k}} $$ which is equal to:

  • $0$ for a unbalanced data set
  • $1$ for a balanced data set
Improve this answer
$\endgroup$
7
  • 2
    $\begingroup$ Note that any logarithm base is fine because the normalization makes the formula invariant to the base. $\endgroup$ – Simone Oct 13 '16 at 11:36
  • $\begingroup$ @Simone, do you know if this balance equation is a standard balance measure in the literature? $\endgroup$ – felipeduque Feb 16 '17 at 20:44
  • $\begingroup$ Not in particular. I think I saw using entropy as measure of balance in literature. Other times I saw using $\min{\{c_i\}}/\max{\{c_i\}}$. I guess it does not really matter. They are usually just used as guidance to check the degree of balance of a data set. $\endgroup$ – Simone Feb 17 '17 at 9:35
  • 1
    $\begingroup$ @Simone, interesting. Other two options can be using KL distance or cross entrpoy to measure the "distance" of {c_i}/n probabilities to 1/k. For KL distance this should be zero of the data-set is balanced and for cross entropy it should be come the log(k) for balanced dataset $\endgroup$ – oak Oct 29 '18 at 14:19
  • 1
    $\begingroup$ Simone's suggestion is referred to as the "Shannon Diversity Index" and is common in ecology research. Here is some more information: itl.nist.gov/div898/software/dataplot/refman2/auxillar/… $\endgroup$ – James Budarz Nov 5 '20 at 23:21
5
$\begingroup$

Based on the answer of Simone, I wrote this short python code to calculate balance, which works very well for me.

def balance(seq):
    from collections import Counter
    from numpy import log
    
    n = len(seq)
    classes = [(clas,float(count)) for clas,count in Counter(seq).items()]
    k = len(classes)
    
    H = -sum([ (count/n) * log((count/n)) for clas,count in classes]) #shannon entropy
    return H/log(k)

Thank you very much!

Improve this answer
$\endgroup$
1
  • $\begingroup$ for me, I had to change .iteritems() for .items() only, then it worked! thanks! $\endgroup$ – Caio Oliveira Aug 13 '20 at 21:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.