How should I resample the training and testing set with imbalanced data whilst having meaningful performance metrics?

Question

I have an imbalanced dataset of approx. 200 positive and 800 negative examples.

I run nested cross-validation where i=5 and j=5; (i is inner and j is outer). The cross-validation procedure isn't the problem.

For each outer-CV iteration, I randomly split the dataset into 80% training and 20% testing, thus giving 160 positive and 640 negative training examples each time, whilst leaving 40 positive and 160 negative examples in the testing set. However, I then down-sample the training set to balance it to 160 positive and 160 negative samples (remember, it was already in random order). This testing set represents the natural distribution of the positive and negative classes as found in nature. Therefore, in a real world scenario, the classifier will need to correctly predict novel data input with this ratio.

However, with the imbalanced testing sets, I experience poor performance in terms of F1 (e.g. 0.50), MCC (e.g. 0.30), and etc. on the positive class (negative class performance is good, but not useful). On the other hand, if I also balance the testing set (which would not represent nature) then I receive good performance (e.g. F1 0.75, MCC 0.50).

Why do I receive poor performance on the natural test distribution, when clearly the classifier is able to discriminate between the two classes, as I tested by also balancing the testing sets?

What is the appropriate solution to solve this problem? Is it solvable?

Is it a problem of poor features? Is it a problem which SVM cannot solve?

Edit: The first answer below contains useful and interesting information, but it does not actually answer the question: Why are the classification performance metrics generally poor when testing the natural distribution, given that a good decision boundary has clearly been found? This is not a problem of optimizing C or G because I tried to optimize for the positive class but it had almost no effect. Class weights would not work either - classification is fine with a balanced testing set. Also, I am already using probabilistic SVM.

Answer 1

Why do I receive poor performance on the natural test distribution, when clearly the classifier is able to discriminate between the two classes

SVM doesn't try to optimize the performance of any single outcome (i.e. positive or negative). It attempts to produce the 'best' classification overall.

This is done by dividing the dataset into two regions with the region determining how to classify points. In order to do this SVM tries to finds a 'good' hyperplane (boundary) between the two regions. SVM defines 'good' as a balance between a split that accurately classifies data points (minimizes hinge loss) and a split that maximizes the size of the 'margin' (distance between closest correctly classified negative point and the boundary plus the closest correctly classified positive point and the boundary).

Kent Munthe Caspersen gives a great explanation of this in the following thread. "What is the influence of C in SVMs with linear kernel?".

If you add additional training data (of a single class) you will potentially do two things to the costs being minimized by SVM. 1. you could increase the number of points misclassified in that class (increasing ridge cost) 2. you could decrease the margin (also increasing cost as SVM tries to maximize the margin)

in order to compensate for the increased cost the optimal boundary would move to more accurately predict the class of data added. this would naturally come at the cost of the accuracy of the remaining class.

What is the appropriate solution to solve this problem? Is it solvable?

The most appropriate solution depends on your goals. Though my bias is normally to use a probabilistic classifier of some type if possible as there is normally the option to collect more data about an observation before classifying it.

Assuming SVM is the best classifier for your purposes. You can weight the costs of the objective function SVM is minimizing.

If your implementation of SVM supports adding weights to the cost function of the SVM then that can be used otherwise duplicating the positive class would produce an equivalent effect to weighting the cost function. You would want to duplicate 4 times in order to make positive errors 4 times as important as it currently is (and therefore balanced with negative).

Is it a problem of poor features?

No

Is it a problem which SVM cannot solve?

see above

Answer 2

One problem could be using a different metric for evaluating test performance than training. SVM is optimizing empirical missclassification error that tries to minimize the absolute number of correct classifications, regardless what are the proportions in each class, whereas F1 could be penalizing more the errors in the less prevalent class (based on quick look at F1). When optimizing on the natural proportion, the classifier is biased to the prevalent class (because of optimizing the empirical error) and F1 decreases because of that. Your goal seems to be give more importance to less prevalent class. I suggest define the train objective that reflect your goals and evaluate both train and test based on same metric. Adjusting training proportions is one way for achieving that (in this case, you should see improvement in the natural test evaluated with a more balanced metric. Have you checked confusion matrices?). Another is to weight the samples, so that you are able to use all training set.

Answer 3

Why are the classification performance metrics generally poor when testing the natural distribution, given that a good decision boundary has clearly been found?

The classifier has found a good decision boundary for a balanced dataset from training with one.

Surely, balancing training data can improve model performance on the natural, imbalanced scenario, but that's not necessarily going to happen, as it's dependant on the distribution of the balanced set and the classifier's hyperparameters.

What is the appropriate solution to solve this problem? Is it solvable?

From your question, I assume you haven't tried various approaches for balancing your dataset. I'd suggest using SMOTE, Tomek Links and Nearest Neighbours-based methods if you haven't already.