I have implemented a machine learning algorithm (say, algorithm A) to perform binary classifications of fraudulent and non fraudulent clients. I am already using other algorithms (say, B and C) to perform this classification and I want to see which of the three is the best one. What would be the appropriate experimental setting to do so?

So far, I have been using accuracy and recall as my main measures of model performance. In order to do this, I have needed to know which are false positives (clients that were labeled by the model as fraudulent when they actually were good clients). When the model is actually implemented, however, I won't be able to identify false positives unless I have a control group. Isn't this right?

How can I know which of the three algorithms is working, if any? Should I used ANOVA? If so, what variable should I perform ANOVA in (maybe the mean amount of money that was "rescued" from fraudsters)? What are other alternatives?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.