# Down-sampled training set with unbalanced test set

Data: https://www.kaggle.com/c/GiveMeSomeCredit/data (cs-training.csv)

Training Tool: Python, Numpy, Pandas

Balancing data with down sampling is a recommended solution to data imbalance. I balanced a training data set and a cross validation set, but left test data imbalanced. The result was extremely bad. It was worse than when I trained with imbalanced data set. Do I have to balance test set as well? Well, then it won't be helpful in real life.

What exactly is the problem here and how do I have to solve it?

• Training set: 90,000 samples

• Cross validation set: 30,000

• Test set: 30,000

Logistic Regression Result on Test set

parameters optimized with cross validation set:

-Threshold: 0.125

-Polynomial Term: 2

-C: 3

trained with training set & tested with test set:

accuracy 0.910833333333

precision 0.364739183178

recall 0.449651046859

fscore 0.402768475106


Steps Taken:

1. Balancing Data: I balanced training set and cross validation set with down sampling. (leaving test set untouched to compare results)

• Training set: 11,942
• Cross validation set: 4,098
• Test set: 30,000
2. Optimizing Parameters (Threshold, Polynomial Term, C)

Logistic Regression Result on Test set

parameters optimized with cross validation set:

-Threshold: 0.4

-Polynomial Term: 2

-C: 6

trained with training set & tested with test set:

accuracy 0.0691666666667

precision 0.0658022581834

recall 0.979062811565

fscore 0.123316485103

• I think this will be hard for people to answer. Can you say more about your situation & what you did? – gung Jul 10 '15 at 12:06
• @gung I've added original data source and steps taken!:D – Hee Kyung Yoon Jul 13 '15 at 11:38