I am currently working on reproducing a paper where the authors (non-statisticians, as far as I know) wanted to solve a classification problem, but had a dataset that was balanced 17:1 in favor of one class (20134 vs 1186 instances).
Their solution was the following (Sect. 5.2):
[...] we generate balanced data for classification by randomly partitioning the non co-presence class into 17 subsets. Each such subset together with the co-presence class constitutes a resampled set for classification. We run experiments with 10 resampled sets, chosen randomly.
I asked for clarification via eMail, and they confirmed the following reading: The bigger class is divided into 17 equally-sized partitions. 10 of these partitions are chosen at random and combined with the (complete) smaller class. They then train and validate 10 independent models (each of them using Multiboost + Random Forest with 10-fold cross validation on their balanced dataset), and sum up the 10 resulting confusion matrices. From the summed-up confusion matrix, they calculate their performance metrics.
Is this (especially the part about using the same small class 10 times) a valid approach to get results for an imbalanced dataset, or does it introduce any problems through the re-use of the entire smaller "copresence" class over all 10 evaluations?
(Note that I am not asking if this is necessarily the best approach, I just want to know if it is valid, i.e. if I can trust the results from the paper, or if the methodology is flawed.)