Currently, I need to work on a binary classification project (e-commerce), where the buyer class (class 1) is a minority class and is the class of interest. Building a classification model has two types of error;
- predicting as a non-buyer a buyer
- predicting as a buyer a non-buyer.
For the minority class
FP means we have predicted 1, while the actual class is 0, i.e. we classify a non-buyer as a buyer (precision gets lower).
FN means that we have predicted 0, while the actual class is 1, i.e. we miss an actual buyer (recall gets lower).
My question is what metrics I should use to assess different models. Apparently, accuracy is not a very good metric, due to imbalanced classes.
For both classes I will get some metrics, i.e. precision, recall and f1-socre. So where should I focus more?
P.S. Given extra information, that a non-buyer has a small cost and a buyer has some good revenue, would it change the metrics I should focus on?