I have recently implemented a machine learning algorithm as a part of a new credit risk scoring system. I would now like to evaluate the accuracy/performance of the algorithm when used in a "real world setting".
In order to do the evaluation, there is a need for manual data gathering, since this is labor intensive I would like to keep evaluation sample as small as possible, while still keeping it large enough so that it can yield significant results.
I have not been able to find much information online regarding how to formally evaluate the performance of a machine learning algorithm when it is used in practice.
Are there any guidelines/suggestion regarding the minimum sample size needed to evaluate the performance of an ML algorithm?
I understand that the sample I draw would need to be random and undergo a significance test to ensure the sample conforms with the trends of the total population, but outside of that what other consideration need to be made?