The search term you are looking for is "learning curve", which gives the (average) model performance as function of the training sample size.
Learning curves depend on a lot of things, e.g.
- classification method
- complexity of the classifier
- how well the classes are separated.
(I think for two-class LDA you may be able to derive some theoretical power calculations, but the crucial fact is always whether your data actually meets the "equal COV multivariate normal" assumption. I'd go for some simulation on for both LDA assumptions and resampling of your already existing data).
There are two aspects of the performance of a classifier trained on a finite sample size $n$ (as usual),
- bias, i.e. on average a classifier trained on $n$ training samples is worse than the classifier trained on $n = \infty$ training cases (this is usually meant by learning curve), and
- variance: a given training set of $n$ cases may lead to quite different model performance.
Even with few cases, you may be lucky and get good results. Or you have bad luck and get a really bad classifier.
As usual, this variance decreases with incresing training sample size $n$.
Another aspect that you may need to take into account is that it is usually not enough to train a good classifier, but you also need to prove that the classifier is good (or good enough). So you need to plan also the sample size needed for validation with a given precision. If you need to give these results as fraction of successes among so many test cases (e.g. producer's or consumer's accuracy / precision / sensitivity /
positive predictive value), and the underlying classification task is rather easy, this can need more independent cases than training of a good model.
As a rule of thumb, for training, the sample size is usually discussed in relation to model complexity (number of cases : number of variates), whereas absolute bounds on the test sample size can be given for a required precision of the performance measurement.
Here's a paper, where we explained these things in more detail, and also discuss how to constuct learning curves:
Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33.
accepted manuscript on arXiv: 1211.1323
This is the "teaser", showing an easy classification problem (we actually have one easy distinction like this in our classification problem, but other classes are far more difficult to distinguish):
We did not try to extrapolate to larger training sample sizes to determine how much more training cases are needed, because the test sample sizes are our bottleneck, and larger training sample sizes would let us construct more complex models, so extrapolation is questionable. For the kind of data sets I have, I'd approach this iteratively, measuring a bunch of new cases, showing how much things improved, measure more cases, and so on.
This may be different for you, but the paper contains literature references to papers using extrapolation to higher sample sizes in order to estimate the required number of samples.