I have implemented a machine learning algorithm (say, algorithm A) to perform binary classifications of fraudulent and non fraudulent clients. I am already using other algorithms (say, B and C) to perform this classification and I want to see which of the three is the best one. What would be the appropriate experimental setting to do so?
So far, I have been using accuracy and recall as my main measures of model performance. In order to do this, I have needed to know which are false positives (clients that were labeled by the model as fraudulent when they actually were good clients). When the model is actually implemented, however, I won't be able to identify false positives unless I have a control group. Isn't this right?
How can I know which of the three algorithms is working, if any? Should I used ANOVA? If so, what variable should I perform ANOVA in (maybe the mean amount of money that was "rescued" from fraudsters)? What are other alternatives?