I have a data with size of 1200 rows having binary dependant variable and around 20 independant variables which are categorical as well as continous in nature. I have tried 2 machine learning techniques viz. Random forest and Gradient boosting. I was able to achieve 65-66% accuracy. I realized that the data is biased for 0 outcome and there is 80:20 ratio for 0:1. So I took equal sample of 0 and 1 with total 500 rows of 250 each for 0 and 1s and then again went ahead to make train and test data out of 500 rows. But this time I was able to achieve 70-72% accuracy and my rank order has improved significantly. Is this way of taking equal sampling statistically correct for training the model and doing predictions?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
If you always predict 0, you will get 80% accuracy. That is why it is not a good performance measure for skewed classes. And throwing away your data in order to make classes less skewed is not a good solution. You should better try $F_1$ score. $$F_1 = 2\frac{precision * recall}{precision + recall}$$ Where $$ precision = \frac{truePositives}{truePositives + falsePositives}$$ and $$ recall = \frac{truePositives}{truePositives + falseNegatives}$$
