# a general measure of data-set imbalance

I am working on thousands of datasets. Many of them are "unbalanced"; either a multi-class list with highly skewed distribution (For example, three categories with the ratio of 3500:300:4 samples) or a continuous number with skewed distribution. I am looking for some metric that can say "How badly unbalanced" the dataset is. Is there such a metric?

Eventually, I want to score these datasets according to their balanced metric and provide a different balancing/ machine learning solution for each of them. I prefer a python solution if it exists.

• If you use R, you can use the diversity() function in the 'vegan' package to calculate the Shannon-Weaver index. rdocumentation.org/packages/vegan/versions/2.4-2/topics/… Feb 1, 2021 at 4:42
• It's not clear why you chose the word balance as if imbalance is a bad thing. If you use a method that depends on the amount of balance in levels of $Y$, switch to another method. Feb 7, 2021 at 11:59
• mild ---> 20-40% of the data set Moderate ---> 1-20% of the data set Extreme ---> <1% of the data set developers.google.com/machine-learning/data-prep/construct/… Jun 16, 2021 at 1:26
• I opine that many of the answers below are too fancy. If we want a simple scalar to indicate how imbalanced a multi-classed dataset is, just report the percent in the minority class. Do not represent it as a ratio-- this gets highly non-linear and doesn't generalize to multi-class problems. A perfectly balanced binary-class dataset would be 50%. If we have 100 classes that are perfectly balanced, we'd expect 1% for the minority class, which is already hard; but if the minority class in this case were 0.001% we know it's even more imbalanced; also report the number in the minority. Oct 20, 2021 at 15:31
• To follow up on @FrankHarrell's point, why do you want to have a balanced machine learning solution? Generally imbalance is not itself a problem, and the machine learning (i.e. statistical) model will be giving a near-optimal solution for the learning task as posed (if applied correctly). If there is a good reason for balancing, it is because the misclassification costs are not equal, and the amount re-weighting/resampling has little or nothing to do with the degree of imbalance. The key is to work out what you really want the model to do. Apr 21, 2022 at 11:47

You could use the Shannon entropy to measure balance.

On a data set of $$n$$ instances, if you have $$k$$ classes of size $$c_i$$ you can compute entropy as follows: $$H = -\sum_{ i = 1}^k \frac{c_i}{n} \log{ \frac{c_i}{n}}.$$

This is equal to:

• $$0$$ when there is one single class. In other words, it tends to $$0$$ when your data set is very unbalanced
• $$\log{k}$$ when all your classes are balanced of the same size $$\frac{n}{k}$$

Therefore, you could use the following measure of Balance for a data set: $$\mbox{Balance} = \frac{H}{\log{k}} = \frac{-\sum_{ i = 1}^k \frac{c_i}{n} \log{ \frac{c_i}{n}}. } {\log{k}}$$ which is equal to:

• $$0$$ for an unbalanced data set
• $$1$$ for a balanced data set
• Note that any logarithm base is fine because the normalization makes the formula invariant to the base. Oct 13, 2016 at 11:36
• @Simone, do you know if this balance equation is a standard balance measure in the literature? Feb 16, 2017 at 20:44
• Not in particular. I think I saw using entropy as measure of balance in literature. Other times I saw using $\min{\{c_i\}}/\max{\{c_i\}}$. I guess it does not really matter. They are usually just used as guidance to check the degree of balance of a data set. Feb 17, 2017 at 9:35
• @Simone, interesting. Other two options can be using KL distance or cross entrpoy to measure the "distance" of {c_i}/n probabilities to 1/k. For KL distance this should be zero of the data-set is balanced and for cross entropy it should be come the log(k) for balanced dataset
– oak
Oct 29, 2018 at 14:19
• Simone's suggestion is referred to as the "Shannon Diversity Index" and is common in ecology research. Here is some more information: itl.nist.gov/div898/software/dataplot/refman2/auxillar/… Nov 5, 2020 at 23:21

Based on the answer of Simone, I wrote this short python code to calculate balance, which works very well for me.

def balance(seq):
from collections import Counter
from numpy import log

n = len(seq)
classes = [(clas,float(count)) for clas,count in Counter(seq).items()]
k = len(classes)

H = -sum([ (count/n) * log((count/n)) for clas,count in classes]) #shannon entropy
return H/log(k)


Thank you very much!

• for me, I had to change .iteritems() for .items() only, then it worked! thanks! Aug 13, 2020 at 21:31
• Python 2 vs 3 I believe. Oct 8, 2021 at 13:15

I had the same problem and looked for some metrics to measure the degree of unbalance in my datasets, but I did not find any. Then, I created one that varies between 0 (perfectly balanced, the number of samples in all categories is the same) and 1 (extremely badly balanced, when the number of samples in all classes, except for one, is 1 and the rest of samples belong to a single class)

The formula is:

$$imbalance = \frac{Max_{samples} - Min_{samples}}{Total_{samples} - nclass}$$

Examples: For a balanced case $$Max_{samples} = Min_{samples}$$, then $$imbalance =0$$

For a three class case ($$nclass=3$$) having 500, 300 and 100 samples each, we have:

$$Max_{samples}=500$$, $$Min_{samples}=100$$, and $$Total_{samples} = 900$$, then

$$imbalance = (500-100)/(900-3) = 0.446$$

In an extreme three classes case, we have 500, 1 and 1 samples in each class, then

$$Max_{samples}=500$$, $$Min_{samples}=1$$ and $$Total_{samples} =502$$, then

$$imbalance = (500-1)/(502-3) = 1$$