I'm trying to build a binary 1/0 ML classification algorithm, and was thinking about how to set up the input dataset. If the event I want to predict (the 1's) occur relatively less frequently in the total data than the 0's, does it makes sense to pare the dataset in such a way to get a more equal distribution of 1's and 0's? Would that be falsely representing the data to the algorithm? What are the disadvantages to doing so?
$\begingroup$
$\endgroup$
1
-
2$\begingroup$ You can do that, but it would be better if you didn't have to throw away data. I imagine the reason this question came up is that you're optimizing your solution for accuracy, but the classes are skewed enough that a high-accuracy algorithm has really poor recall. I would recommend optimizing for something other than accuracy (like minimizing a weighted sum of false negatives and false positives, more heavily weighting the false negatives). $\endgroup$– Stumpy Joe PeteCommented Oct 24, 2013 at 0:30
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Define an accuracy metric that reasonably models how you want your algorithm to perform.
Once you have a metric in hand you can cross-validate this question and see if it is an improves the performance.
Some common accuracy metrics that model the problem different: Normalized mutual information, Gini on the labels argsorted by the probabilities, Precision, Recall, AUC.
If the classes are extremely unbalanced and FN are crucial, you'll see an improvement.
answered Oct 24, 2013 at 4:00