Is there a concept of “enough” data for training statistical models?

Question

I work on quite a lot of statistical modelling, such as Hidden Markov Models and Gaussian Mixture Models. I see that training good models in each of these cases requires a large (> 20000 sentences for HMMs) amount of data that is taken from similar environments as the final use. My question is:

1. Is there a concept of "enough" training data in the literature? How much training data is "good enough"?
2. How can I compute how many sentences are needed for "good" (that give a good recognition accuracy (> 80%)) models to be trained?
3. How do I know if a model has been trained properly? Will the coefficients in the model start to exhibit random fluctuations? If so, how do I distinguish random fluctuations and real changes due to model update?

Please feel free to retag this question in case it needs more tags.

Answer 1

You can slice your dataset into consecutive subsets with 10%, 20%, 30%, ... , 100% of your data and for each subset estimate the variance of your estimator accuracy using k-fold cross validation or bootstrapping. If you have "enough" data, plotting the variances should display a decreasing monotonic line that should reach a plateau before 100%: adding more data does not decrease the variance of the accuracy of the estimator in any significant way.