a general measure of data-set imbalance

Question

I am working on thousands of datasets. Many of them are "unbalanced"; either a multi-class list with highly skewed distribution (For example, three categories with the ratio of 3500:300:4 samples) or a continuous number with skewed distribution. I am looking for some metric that can say "How badly unbalanced" the dataset is. Is there such a metric?

Eventually, I want to score these datasets according to their balanced metric and provide a different balancing/ machine learning solution for each of them. I prefer a python solution if it exists.

Answer 1

You could use the Shannon entropy to measure balance.

On a data set of $n$ instances, if you have $k$ classes of size $c_i$ you can compute entropy as follows: $$ H = -\sum_{ i = 1}^k \frac{c_i}{n} \log{ \frac{c_i}{n}}. $$

This is equal to:

• $0$ when there is one single class. In other words, it tends to $0$ when your data set is very unbalanced
• $\log{k}$ when all your classes are balanced of the same size $\frac{n}{k}$

Therefore, you could use the following measure of Balance for a data set: $$ \mbox{Balance} = \frac{H}{\log{k}} = \frac{-\sum_{ i = 1}^k \frac{c_i}{n} \log{ \frac{c_i}{n}}. } {\log{k}} $$ which is equal to:

• $0$ for an unbalanced data set
• $1$ for a balanced data set

Answer 2

Based on the answer of Simone, I wrote this short python code to calculate balance, which works very well for me.

def balance(seq):
    from collections import Counter
    from numpy import log
    
    n = len(seq)
    classes = [(clas,float(count)) for clas,count in Counter(seq).items()]
    k = len(classes)
    
    H = -sum([ (count/n) * log((count/n)) for clas,count in classes]) #shannon entropy
    return H/log(k)

Thank you very much!

Answer 3

I had the same problem and looked for some metrics to measure the degree of unbalance in my datasets, but I did not find any. Then, I created one that varies between 0 (perfectly balanced, the number of samples in all categories is the same) and 1 (extremely badly balanced, when the number of samples in all classes, except for one, is 1 and the rest of samples belong to a single class)

The formula is:

$$imbalance = \frac{Max_{samples} - Min_{samples}}{Total_{samples} - nclass}$$

Examples: For a balanced case $Max_{samples} = Min_{samples}$, then $imbalance =0$

For a three class case ($nclass=3$) having 500, 300 and 100 samples each, we have:

$Max_{samples}=500$, $Min_{samples}=100$, and $Total_{samples} = 900$, then

$imbalance = (500-100)/(900-3) = 0.446$

In an extreme three classes case, we have 500, 1 and 1 samples in each class, then

$Max_{samples}=500$, $Min_{samples}=1$ and $Total_{samples} =502$, then

$imbalance = (500-1)/(502-3) = 1$