5
$\begingroup$

I attempted a ReLU neural network to classify data sets of 3 classes that are not balanced (in both training and test sets), i.e. 30% of samples are in class A, 10% in class B and 60% in class C. And in particular for this problem, I'm mostly interested in the precision of class C (with reasonable recalls) since that's the only class I can take use of. Currently I artificially clone and add random +/-5% adjustments to each class A and B samples so that each class has roughly 1/3 samples in the training set. And then I choose the winning epoch based on F1 score for class C.

NEW BEST: epoch 1, score: 0.572852844535, F1: 0.58989
5, precision 0.516919, recall 0.686862, accuracy 0.643098 (0.572852844535), learning_rate=1.0 (patience: 320000 / 1599)
F1: 0.589895, precision 0.516919, recall 0.686862, accuracy 0.643098
precisions: [ 0.19046712  0.48642075  0.61648193]
recalls: [ 0.17856346  0.10650572  0.82099259]
class[0] is predicted as class[0]: 40
class[0] is predicted as class[1]: 4
class[0] is predicted as class[2]: 180
class[1] is predicted as class[0]: 54
class[1] is predicted as class[1]: 36
class[1] is predicted as class[2]: 248
class[2] is predicted as class[0]: 116
class[2] is predicted as class[1]: 34
class[2] is predicted as class[2]: 688

NEW BEST epoch 14, score: 0.708267443522, F1: 0.5302
56, precision 0.612621, recall 0.467413, accuracy 0.556719 (0.708267443522), learning_rate=0.974310040474 (patience: 343195 / 22399)
F1: 0.530256, precision 0.612621, recall 0.467413, accuracy 0.556719
precisions: [ 0.22606464  0.33912306  0.82626222]
recalls: [ 0.49551359  0.46152481  0.44271548]
class[0] is predicted as class[0]: 111
class[0] is predicted as class[1]: 89
class[0] is predicted as class[2]: 24
class[1] is predicted as class[0]: 128
class[1] is predicted as class[1]: 156
class[1] is predicted as class[2]: 54
class[2] is predicted as class[0]: 252
class[2] is predicted as class[1]: 215
class[2] is predicted as class[2]: 371

As seen above, at epoch 1, the accuracy looks much better because the network just classified all test samples to class C; in epoch 14, the accuracy looks worse but is in fact better since the network can classify other classes too.

How can I train or test for this unbalanced data set? Should I also artificially balance the test set in addition to the training set?

$\endgroup$
2
  • 6
    $\begingroup$ Imbalance is only a significant problem when you are using a discontinuous improper accuracy scoring rule such as proportion "classified" "correctly". $\endgroup$ Nov 1 '15 at 18:34
  • $\begingroup$ I think this answer is not bad: datascience.stackexchange.com/a/38815/9191 $\endgroup$ May 22 '19 at 8:00
2
$\begingroup$

Jain and Nag suggest a balanced training set and a representational test data set for evaluation.

The balanced training set allows for the model to familiarize itself with less frequent state of interest and helps the model to formulate general rules.

However, as @rep_ho points out you should definitely use a test set that represents the population of your data. Otherwise, you would skew your results.

Note though that relying on accuracy as a performance measure in a highly unbalanced dataset can be a misleading metric. If you have a dataset with two groups with a 90/10 split, then the model might simply 'guess' the first category all the time and nevertheless achieve a 90% accuracy.

$\endgroup$
1
$\begingroup$

For unbalanced sample , you can use oversampling for those which are underrepresented or under sampling which have more representations .

  • But oversampling and under sampling should only be done if you feel that your sample doesn't represent the true population

  • Now the question arises how do we know whether my sample is a correct representation of the population ? It depends on two factors

  • 1) Either you have to consult a subject expert or
  • 2) The results of your test are saying that : For eg Systolic and Diastolic BP of population would certainly lie within confined intervals , but your sample might have a dataset which have people with high BP only .

You can refer www.analyticsvidhya.com to learn how to do over and under sampling in R.

$\endgroup$
0
$\begingroup$

You definitely should not balance your test set. Test set should be independent assessment of your model.

You might should use different scorings then accuracy. E.g. Balanced accuracy (mean of specificity and sensitivity), kappa, F score. Those measures depends on your possibly arbitrary decision of where to put cutout point. You might use area under ROC curve or area under precision/recall curve, which might be of interest of you, since precision with reasonable recall is what you care about.

Other thing you can do, is to move your cutout/cutoff point. So you will not predict class A if your network is confident about A for > 0.333, but for example > 0.1. Other thing you can do is to use synthetic data points, as you already did. There is SMOTE and ROSE algorithm, that might work better than your naive noise imputation. You can also put more weight, in your training to your minority class.

$\endgroup$
2
  • $\begingroup$ I already use F(beta=0.1) as the score function to pick the resulting neural net. My concern is whether I need to use a different loss/regression activation function. My current output activation function is softmax and loss function is negative-log-likelihood. $\endgroup$
    – teddy
    Nov 21 '15 at 6:14
  • 1
    $\begingroup$ Try a different cutoff point and look if you F score tells you that the classification with the new cutoff works better. $\endgroup$ Nov 5 '16 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.