I have a question in mind about machine learning systems. Say, I have a named entity extraction systems which give me an accuracy of 90%(precision 90, and recall 90) during training and testing. How can I assure that the same system will perform with 90% precision and recall in runtime? In the case of a completely new data set of the same kind. How are these statistical tests significant?
1 Answer
$\begingroup$
$\endgroup$
Short answer is you can never be sure on how your system will perform in the real world because the unseen data is actually never sampled for the same distribution as the training data.
What you can do is use K-fold cross validation and calculate the mean and standard deviation of your performance metrics, this will give you a general sense on how your system will perform on unseen data.
K-fold cross validation is when you divide your training data into K number of sets (folds) then use K-1 of those sets to train the algorithm and 1 to validate it. You then keep track of the metrics you want (precision, recall, accuracy) for the different validation sets to calculate the mean and standard deviation across folds (i.e precision = 0.9 +/- 0.05).
Since in each fold the algorithm never sees the validation data, the final result can be seen as an approximation of your performance on unseen data, however, since you (the person) do see the validation results you must be careful not to overfit your choice of parameters to what you are seeing.
