13
$\begingroup$

I am working on thousands of datasets. Many of them are "unbalanced"; either a multi-class list with highly skewed distribution (For example, three categories with the ratio of 3500:300:4 samples) or a continuous number with skewed distribution. I am looking for some metric that can say "How badly unbalanced" the dataset is. Is there such a metric?

Eventually, I want to score these datasets according to their balanced metric and provide a different balancing/ machine learning solution for each of them. I prefer a python solution if it exists.

Improve this question
$\endgroup$
5
  • $\begingroup$ If you use R, you can use the diversity() function in the 'vegan' package to calculate the Shannon-Weaver index. rdocumentation.org/packages/vegan/versions/2.4-2/topics/… $\endgroup$
    – Zilu Liang
    Feb 1, 2021 at 4:42
  • 2
    $\begingroup$ It's not clear why you chose the word balance as if imbalance is a bad thing. If you use a method that depends on the amount of balance in levels of $Y$, switch to another method. $\endgroup$
    – Frank Harrell
    Feb 7, 2021 at 11:59
  • $\begingroup$ mild ---> 20-40% of the data set Moderate ---> 1-20% of the data set Extreme ---> <1% of the data set developers.google.com/machine-learning/data-prep/construct/… $\endgroup$
    – Dr Nisha Arora
    Jun 16, 2021 at 1:26
  • $\begingroup$ I opine that many of the answers below are too fancy. If we want a simple scalar to indicate how imbalanced a multi-classed dataset is, just report the percent in the minority class. Do not represent it as a ratio-- this gets highly non-linear and doesn't generalize to multi-class problems. A perfectly balanced binary-class dataset would be 50%. If we have 100 classes that are perfectly balanced, we'd expect 1% for the minority class, which is already hard; but if the minority class in this case were 0.001% we know it's even more imbalanced; also report the number in the minority. $\endgroup$
    – George Forman
    Oct 20, 2021 at 15:31
  • 1
    $\begingroup$ To follow up on @FrankHarrell's point, why do you want to have a balanced machine learning solution? Generally imbalance is not itself a problem, and the machine learning (i.e. statistical) model will be giving a near-optimal solution for the learning task as posed (if applied correctly). If there is a good reason for balancing, it is because the misclassification costs are not equal, and the amount re-weighting/resampling has little or nothing to do with the degree of imbalance. The key is to work out what you really want the model to do. $\endgroup$
    – Dikran Marsupial
    Apr 21 at 11:47

3 Answers 3

Reset to default
25
$\begingroup$

You could use the Shannon entropy to measure balance.

On a data set of $n$ instances, if you have $k$ classes of size $c_i$ you can compute entropy as follows: $$ H = -\sum_{ i = 1}^k \frac{c_i}{n} \log{ \frac{c_i}{n}}. $$

This is equal to:

  • $0$ when there is one single class. In other words, it tends to $0$ when your data set is very unbalanced
  • $\log{k}$ when all your classes are balanced of the same size $\frac{n}{k}$

Therefore, you could use the following measure of Balance for a data set: $$ \mbox{Balance} = \frac{H}{\log{k}} = \frac{-\sum_{ i = 1}^k \frac{c_i}{n} \log{ \frac{c_i}{n}}. } {\log{k}} $$ which is equal to:

  • $0$ for an unbalanced data set
  • $1$ for a balanced data set
Improve this answer
$\endgroup$
7
  • 3
    $\begingroup$ Note that any logarithm base is fine because the normalization makes the formula invariant to the base. $\endgroup$
    – Simone
    Oct 13, 2016 at 11:36
  • $\begingroup$ @Simone, do you know if this balance equation is a standard balance measure in the literature? $\endgroup$
    – felipeduque
    Feb 16, 2017 at 20:44
  • $\begingroup$ Not in particular. I think I saw using entropy as measure of balance in literature. Other times I saw using $\min{\{c_i\}}/\max{\{c_i\}}$. I guess it does not really matter. They are usually just used as guidance to check the degree of balance of a data set. $\endgroup$
    – Simone
    Feb 17, 2017 at 9:35
  • 1
    $\begingroup$ @Simone, interesting. Other two options can be using KL distance or cross entrpoy to measure the "distance" of {c_i}/n probabilities to 1/k. For KL distance this should be zero of the data-set is balanced and for cross entropy it should be come the log(k) for balanced dataset $\endgroup$
    – oak
    Oct 29, 2018 at 14:19
  • 1
    $\begingroup$ Simone's suggestion is referred to as the "Shannon Diversity Index" and is common in ecology research. Here is some more information: itl.nist.gov/div898/software/dataplot/refman2/auxillar/… $\endgroup$
    – James Budarz
    Nov 5, 2020 at 23:21
10
$\begingroup$

Based on the answer of Simone, I wrote this short python code to calculate balance, which works very well for me.

def balance(seq):
    from collections import Counter
    from numpy import log
    
    n = len(seq)
    classes = [(clas,float(count)) for clas,count in Counter(seq).items()]
    k = len(classes)
    
    H = -sum([ (count/n) * log((count/n)) for clas,count in classes]) #shannon entropy
    return H/log(k)

Thank you very much!

Improve this answer
$\endgroup$
2
  • $\begingroup$ for me, I had to change .iteritems() for .items() only, then it worked! thanks! $\endgroup$
    – Caio Oliveira
    Aug 13, 2020 at 21:31
  • $\begingroup$ Python 2 vs 3 I believe. $\endgroup$
    – Asinus Rex
    Oct 8, 2021 at 13:15
0
$\begingroup$

I had the same problem and looked for some metrics to measure the degree of unbalance in my datasets, but I did not find any. Then, I created one that varies between 0 (perfectly balanced, the number of samples in all categories is the same) and 1 (extremely badly balanced, when the number of samples in all classes, except for one, is 1 and the rest of samples belong to a single class)

The formula is:

$$imbalance = \frac{Max_{samples} - Min_{samples}}{Total_{samples} - nclass}$$

Examples: For a balanced case $Max_{samples} = Min_{samples}$, then $imbalance =0$

For a three class case ($nclass=3$) having 500, 300 and 100 samples each, we have:

$Max_{samples}=500$, $Min_{samples}=100$, and $Total_{samples} = 900$, then

$imbalance = (500-100)/(900-3) = 0.446$

In an extreme three classes case, we have 500, 1 and 1 samples in each class, then

$Max_{samples}=500$, $Min_{samples}=1$ and $Total_{samples} =502$, then

$imbalance = (500-1)/(502-3) = 1$

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.