4
$\begingroup$

I am working on thousands of datasets, many of them are "unbalanced" ; either a multi-class list with highly skewed distribution (For example three categories with ratio of 3500:300:4 samples) or a continues number with skewed distribution. I am looking for some metric that can say "How badly unbalanced" the dataset is. Is there such a meteric?

Eventually I want to score these datasets according to their balanced metric and provide a different balancing/ machine learning solution to each of them. I would prefer a python solution if exist.

| cite | improve this question | |
$\endgroup$
9
$\begingroup$

You could use the Shannon entropy as a measure of balance.

On a data set of $n$ instances, if you have $k$ classes of size $c_i$ you can compute entropy as follows: $$ H = -\sum_{ i = 1}^k \frac{c_i}{n} \log{ \frac{c_i}{n}}. $$

This is equal to:

  • $0$ when there is one single class. In other words, it tends to $0$ when your data set is very unbalanced
  • $\log{k}$ when all your classes are balanced of the same size $\frac{n}{k}$

Therefore, you could use the following measure of Balance for a data set: $$ \mbox{Balance} = \frac{H}{\log{k}} = \frac{-\sum_{ i = 1}^k \frac{c_i}{n} \log{ \frac{c_i}{n}}. } {\log{k}} $$ which is equal to:

  • $0$ for a unbalanced data set
  • $1$ for a balanced data set
| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Note that any logarithm base is fine because the normalization makes the formula invariant to the base. $\endgroup$ – Simone Oct 13 '16 at 11:36
  • $\begingroup$ @Simone, do you know if this balance equation is a standard balance measure in the literature? $\endgroup$ – felipeduque Feb 16 '17 at 20:44
  • $\begingroup$ Not in particular. I think I saw using entropy as measure of balance in literature. Other times I saw using $\min{\{c_i\}}/\max{\{c_i\}}$. I guess it does not really matter. They are usually just used as guidance to check the degree of balance of a data set. $\endgroup$ – Simone Feb 17 '17 at 9:35
  • 1
    $\begingroup$ @Simone, interesting. Other two options can be using KL distance or cross entrpoy to measure the "distance" of {c_i}/n probabilities to 1/k. For KL distance this should be zero of the data-set is balanced and for cross entropy it should be come the log(k) for balanced dataset $\endgroup$ – oak Oct 29 '18 at 14:19
  • $\begingroup$ @oak yes all these measures are quite related: if $C$ is the clustering and $Q$ is the balanced clustering: $KL(C||Q) = \log(k) - H(C)$ and the cross-entropy $H(C,Q) = H(P) + KL(C||Q)$. $\endgroup$ – Simone Oct 29 '18 at 16:22
2
$\begingroup$

Based on the answer of Simone, I wrote this short python code to calculate balance, which works very well for me.

def balance(seq):
    from collections import Counter
    from numpy import log

    n = len(seq)
    classes = [(clas,float(count)) for clas,count in Counter(seq).iteritems()]
    k = len(classes)

    H = -sum([ (count/n) * log((count/n)) for clas,count in classes]) #shannon entropy
    return H/log(k)

Thank you very much!

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.