Utilizing cross-validation with up-sampled data

Question

I am creating a support vector machine for extremely unbalanced data in which identifying instances of the rare class is of the highest importance. Since the data is so unbalanced, training and testing a model with no up-sampling results in an extremely accurate model that performs very poorly in terms of its true positive rate.

To ensure that the model is able to appropriately distinguish between the positive and negative class I split the data into a training and test set and performed up-sampling of the rare class within the training data. This allows me to use the test data to estimate the model's performance but it seems that it leaves me unable to utilize k-fold cross validation. Is the method I have utilized an acceptable approach? Is there another methodology that is recommended?

Answer 1

Yes, the CV results are going to be biased. You could still use them to tune the model.

Another option is to use a class weighting scheme that gives asymmetric cost values to different kinds of errors (see the reference below). This is available in some software (e.g. the R kernlab package).

I think this is a better approach since it allows you to dial the cost function to meet your sensitivity or specificity needs. It's another tuning parameter and you don't have that "knob to turn" when up-sampling.

Max

Veropoulos K, Campbell C, Cristianini N (1999). "Controlling the Sensitivity of Support Vector Machines." Proceedings of the International Joint Conference on Artificial Intelligence, 1999, 55–60.

Answer 2

Here is a related answer that might be of interest of this problem. (Sorry for the auto-citation)

Answer 3

I don't use SVM, but in Logistic Regression and ANN's I have sucessfully used k-fold with replicated training data (cut the folds, generate the proper training\test pairs, generate replicas of the less represented class only in training data), sometimes with noise. Biases were removed in the selection of the cutpoints.

A very elementary review to techniques available for unbalanced data if found on this paper. My favorite technique so far (PCA plus noise injection) is found in this paper.