I have an imbalanced dataset of approx. 200 positive and 800 negative examples.
I run nested cross-validation where i=5 and j=5; (i is inner and j is outer). The cross-validation procedure isn't the problem.
For each outer-CV iteration, I randomly split the dataset into 80% training and 20% testing, thus giving 160 positive and 640 negative training examples each time, whilst leaving 40 positive and 160 negative examples in the testing set. However, I then down-sample the training set to balance it to 160 positive and 160 negative samples (remember, it was already in random order). This testing set represents the natural distribution of the positive and negative classes as found in nature. Therefore, in a real world scenario, the classifier will need to correctly predict novel data input with this ratio.
However, with the imbalanced testing sets, I experience poor performance in terms of F1 (e.g. 0.50), MCC (e.g. 0.30), and etc. on the positive class (negative class performance is good, but not useful). On the other hand, if I also balance the testing set (which would not represent nature) then I receive good performance (e.g. F1 0.75, MCC 0.50).
Why do I receive poor performance on the natural test distribution, when clearly the classifier is able to discriminate between the two classes, as I tested by also balancing the testing sets?
What is the appropriate solution to solve this problem? Is it solvable?
Is it a problem of poor features? Is it a problem which SVM cannot solve?
Edit: The first answer below contains useful and interesting information, but it does not actually answer the question: Why are the classification performance metrics generally poor when testing the natural distribution, given that a good decision boundary has clearly been found? This is not a problem of optimizing C or G because I tried to optimize for the positive class but it had almost no effect. Class weights would not work either - classification is fine with a balanced testing set. Also, I am already using probabilistic SVM.