I am looking mainly for ideas and approaches which I could not find by just Googling.

I created a classification model to predict about 175 unique classes from text features. I trained the model on about 8M examples with a 80-20 test train split. The accuracy on the test set was consistently > 99% , I used k-fold validation while building the model. The model was 'deployed' meaning the it's batch predicting every day and now has made over 31M predictions.

I want to estimate the prediction error rate on the 20M rows using inferential tests. The challenge is that 20M rows is a lot. I was thinking of plotting a sampling distribution by drawing multiple random sample sets of a specific size and calculating the classification error rate of each sample set (let's say calculating an error using 30 samples drawn 30 times with replacement).

The problem is that I'll have to manually verify 900 predicted examples if I want to do that. Also how big a sample do I need to draw and how many times?

Any suggestions on best approach?

  • $\begingroup$ Why do you want to do inference? What is the goal? With the amount of data you have you are almost guaranteed to reject anything. $\endgroup$ – user2974951 Aug 13 at 8:28
  • $\begingroup$ "The problem is that I'll have to manually verify 900 predicted examples" What do you mean by that? It's not clear to me why you don't simply compute the error over the new data the model is currently used to make predictions for, data that was not used in training or model selection. $\endgroup$ – CloseToC Aug 13 at 11:08
  • $\begingroup$ The new data is unlabeled. The training and test data were labeled so I could calculate the model accuracy. The model has made some predictions on the new data. How do I know these predictions are accurate? I want to calculate a margin of error for the prediction accuracy. $\endgroup$ – vagabond Aug 13 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.