Which balance strategy to learn from my very imbalanced dataset?

Question

I'm using a deep learning approach on a dataset made of ~20 millions of elements, where each element has a TRUE or FALSE label. This dataset unfortunately is veeery imbalanced: I've 98% of falses and only 2% of trues.

My algorithm uses three subsets: a training set, a validation set and a test set. All these sets are independent.

My algorithm runs this way:

(1) it trains an artificial neural network model on the trainining set

(2) it applies the trained model to the validation set, and computes its ROC AUC

(3) if the AUC at (2) is < 80%, it selects a new model and comes back to (1). Otherwise it goes to (4)

(4) it applies the trained model on the test set

To make my algorithm able to recognize trues and false, I'm using an artificially balanced training dataset made of 50% trues and 50% falses; the same for the validation set (50% trues and 50% falses). On the contrary, the test set contains 98% falses and 2% trues.

Is this approach correct?

Do you think I should change the artificial balance of the training set? Do you think I should change the artificial balance of the validation set set?

Answer 1

Any method (classification method or choice of accuracy measure) that requires deleting data is deficient. Once you develop a well-calibrated probability estimation model you can use a proper accuracy score to judge it. Also, a semi-proper scoring rule, the $c$-index (concordance probability; AUROC) is completely unaffected by extreme imbalance. So is the coefficient of discrimination described in Section 10.6 of my Course Notes.

Answer 2

First, you are quite lucky. Your dataset is quite imbalanced but you have quite a few records. If you have ~400K of minority class sample, your imbalance is relative, not absolute.

As for how to validate your results, see here

As for your approach, note that you train and validate on a balanced dataset and you would like to apply the classifier on the imbalanced dataset. You need a way to adapt the classifier you built on the balanced dataset to you regular dataset.

I like to do this adaptation using a technique based on boosting. You can see the details here.