I am currently working on reproducing a paper where the authors (non-statisticians, as far as I know) wanted to solve a classification problem, but had a dataset that was balanced 17:1 in favor of one class (20134 vs 1186 instances).

Their solution was the following (Sect. 5.2):

[...] we generate balanced data for classification by randomly partitioning the non co-presence class into 17 subsets. Each such subset together with the co-presence class constitutes a resampled set for classification. We run experiments with 10 resampled sets, chosen randomly.

I asked for clarification via eMail, and they confirmed the following reading: The bigger class is divided into 17 equally-sized partitions. 10 of these partitions are chosen at random and combined with the (complete) smaller class. They then train and validate 10 independent models (each of them using Multiboost + Random Forest with 10-fold cross validation on their balanced dataset), and sum up the 10 resulting confusion matrices. From the summed-up confusion matrix, they calculate their performance metrics.

Is this (especially the part about using the same small class 10 times) a valid approach to get results for an imbalanced dataset, or does it introduce any problems through the re-use of the entire smaller "copresence" class over all 10 evaluations?

(Note that I am not asking if this is necessarily the best approach, I just want to know if it is valid, i.e. if I can trust the results from the paper, or if the methodology is flawed.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.