I am looking mainly for ideas and approaches which I could not find by just Googling.
I created a classification model to predict about 175 unique classes from text features. I trained the model on about 8M examples with a 80-20 test train split. The accuracy on the test set was consistently > 99% , I used k-fold validation while building the model. The model was 'deployed' meaning the it's batch predicting every day and now has made over 31M predictions.
I want to estimate the prediction error rate on the 20M rows using inferential tests. The challenge is that 20M rows is a lot. I was thinking of plotting a sampling distribution by drawing multiple random sample sets of a specific size and calculating the classification error rate of each sample set (let's say calculating an error using 30 samples drawn 30 times with replacement).
The problem is that I'll have to manually verify 900 predicted examples if I want to do that. Also how big a sample do I need to draw and how many times?
Any suggestions on best approach?