What is the best method to determine the minimum number of training samples required for a classifier?
I am only comparing one classifier (four class problem), discriminant function analysis (DFA) with different training sample sizes from different data sets. However, I could be considered to be comparing more than one classifier as I am also comparing DFA using the within matrix and separate matrix methods. The data is accelerometer data used to classify animal behaviours. The different data sets are all from the same original data but have had treatments applied, eg smoothed, outlier removed.
I have run 100 interations of randomly selected increasing subsample sizes (10 - 2550) from each data set through each classifier. Averaging the classification accuracy or error rate for each subsample size separately for each data set gives me a learning curve for the classifier on each of the four classes.
Using the area at which the learning curve levels out, I would like to suggest a possible minimum training sample size for each class required for a future study. Rather than visually looking at the learning curve and making a recommendation, is there an appropriate statistical test. Am I talking about data-driven feature selection? I dont need to extrapolate as it looks like I have enough training samples. I dont think ROC curves are applicable as I only have the classification rates from each run of each sub sample sizes rather than predicted membership probabilities which is what I think i need for ROC curves.
I have looked over several posts here which have been useful in advancing my thinking. This post (How large a training set is needed?) relates to my requirement but I dont see it suggesting an appropriate test.
This paper (http://jmlr.org/papers/volume7/demsar06a/demsar06a.pdf) suggests the Friedman test with the corresponding post-hoc tests for comparison of more classifiers over multiple data sets. However, it doesn't appear to take into account the increasing sub sample size from the same training data set, rather it uses completely independent data sets.