I am developing a fraud detection system for online purchases using random forests. My first concern was to optimize for recall, as the dataset was originally unbalanced (98% no fraud events and 2% fraud events). After resampling my dataset and forcing it to be "balanced" (40% fraud and 60% no fraud) it is now performing well in terms of recall (40%). The issue is that I now want to optimize for specificity. Why? Because the tradeoff of optimizing for recall is that I am declining a lot of no fraud transactions (i.e. sacrificing my acceptance rate).
I am using sklearn to implement my algorithm. In particular, I am using the GridSearchCV to optimize hyperparameter (for now, max_feautures and n_estimators), but GridSearchCV doesn't has a built in method to optimize for specificity. Further, specificity is a measure of statistical precision, and I would like to optimize for the value at risk in each purchase. I am considering using a cost matrix, but I am unsure on how to do this. Any advice welcome.