Machine learning algorithm test/evaluation sample size

Question

I have recently implemented a machine learning algorithm as a part of a new credit risk scoring system. I would now like to evaluate the accuracy/performance of the algorithm when used in a "real world setting".

In order to do the evaluation, there is a need for manual data gathering, since this is labor intensive I would like to keep evaluation sample as small as possible, while still keeping it large enough so that it can yield significant results.

I have not been able to find much information online regarding how to formally evaluate the performance of a machine learning algorithm when it is used in practice.

Are there any guidelines/suggestion regarding the minimum sample size needed to evaluate the performance of an ML algorithm?

I understand that the sample I draw would need to be random and undergo a significance test to ensure the sample conforms with the trends of the total population, but outside of that what other consideration need to be made?

Answer 1

I do not have a definitive answer for your scenario, however I may suggest the following, considering documentation resources, accuracy confidence interval and synthetic data generation.

1. Resources

https://epub.ub.uni-muenchen.de/26870/1/TR.pdf

https://pdfs.semanticscholar.org/452e/6c05d46e061290fefff8b46d0ff161998677.pdf

2. Accuracy confidence interval

Based on R caret package ConfusionMatrix.R source code:

https://github.com/topepo/caret/blob/master/pkg/caret/R/confusionMatrix.R

line 218: binom.test(sum(diag(x)), sum(x))$conf.int

where x is confusion matrix, we can read how the accuracy confidence interval is computed. The lower test size is, the wider such interval is. Examples:

binom.test(500,600)$conf.int

[1] 0.8010559 0.8622856

binom.test(50,60)$conf.int

[1] 0.7147807 0.9170712

Hence, one more criteria is to ensure you have test data size large enough to ensure required accuracy confidence interval size. However, that is not the only criteria for your purpose (see point 4).

3. Synthetic data generation

If your training/validation set gathers "real data", based on it there is chance of generating synthetic data by using the R synthpop package

https://cran.r-project.org/web/packages/synthpop/vignettes/synthpop.pdf

4. Understanding spatial distribution of your covariates

Generally speaking, you have to make sure your test data exhaustively exploits the statistical distribution of your covariates.

Answer 2

There is no one ultimate solution in regards. Generally speaking, it depends on the size of the whole dataset, in which, the splitting should make sure that each set (training and testing) covers the distribution among the features and stratified for the label variable.

It could also be recommended to use different splits and make comparisons of the score results for each split. It could be a good indicator if all results were quite close. For a practical point of view, I, sometimes, use the split that achieves a score in the middle.