10
$\begingroup$

I work on quite a lot of statistical modelling, such as Hidden Markov Models and Gaussian Mixture Models. I see that training good models in each of these cases requires a large (> 20000 sentences for HMMs) amount of data that is taken from similar environments as the final use. My question is:

  1. Is there a concept of "enough" training data in the literature? How much training data is "good enough"?
  2. How can I compute how many sentences are needed for "good" (that give a good recognition accuracy (> 80%)) models to be trained?
  3. How do I know if a model has been trained properly? Will the coefficients in the model start to exhibit random fluctuations? If so, how do I distinguish random fluctuations and real changes due to model update?

Please feel free to retag this question in case it needs more tags.

$\endgroup$
10
$\begingroup$

You can slice your dataset into consecutive subsets with 10%, 20%, 30%, ... , 100% of your data and for each subset estimate the variance of your estimator accuracy using k-fold cross validation or bootstrapping. If you have "enough" data, plotting the variances should display a decreasing monotonic line that should reach a plateau before 100%: adding more data does not decrease the variance of the accuracy of the estimator in any significant way.

$\endgroup$
1
  • $\begingroup$ I will have to try that. Sounds interesting. Thanks! $\endgroup$ – Sriram Aug 24 '11 at 9:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.