0
$\begingroup$

Data: https://www.kaggle.com/c/GiveMeSomeCredit/data (cs-training.csv)

Training Tool: Python, Numpy, Pandas

Balancing data with down sampling is a recommended solution to data imbalance. I balanced a training data set and a cross validation set, but left test data imbalanced. The result was extremely bad. It was worse than when I trained with imbalanced data set. Do I have to balance test set as well? Well, then it won't be helpful in real life.

What exactly is the problem here and how do I have to solve it?

Originally, I had three data sets, imbalanced (about 7% : 93%).

  • Training set: 90,000 samples

  • Cross validation set: 30,000

  • Test set: 30,000

Logistic Regression Result on Test set

parameters optimized with cross validation set: 

-Threshold: 0.125

-Polynomial Term: 2

-C: 3


trained with training set & tested with test set:

accuracy 0.910833333333

precision 0.364739183178

recall 0.449651046859

fscore 0.402768475106

Steps Taken:

  1. Balancing Data: I balanced training set and cross validation set with down sampling. (leaving test set untouched to compare results)

    • Training set: 11,942
    • Cross validation set: 4,098
    • Test set: 30,000
  2. Optimizing Parameters (Threshold, Polynomial Term, C)

Logistic Regression Result on Test set

parameters optimized with cross validation set: 

-Threshold: 0.4

-Polynomial Term: 2

-C: 6


trained with training set & tested with test set:    

accuracy 0.0691666666667

precision 0.0658022581834

recall 0.979062811565

fscore 0.123316485103
$\endgroup$
  • $\begingroup$ I think this will be hard for people to answer. Can you say more about your situation & what you did? $\endgroup$ – gung Jul 10 '15 at 12:06
  • $\begingroup$ @gung I've added original data source and steps taken!:D $\endgroup$ – Hee Kyung Yoon Jul 13 '15 at 11:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.