How to extrapolate training metrics to test dataset? i have split my training data 30/70 and trained models, my models are performing really well on the training set but i have a large unlabelled dataset where i want to do inference, how could i measure accuracy on this dataset for which i don't have the correct answer.
I want to know how well my model does in production. how would you validate a large dataset say 600k examples you are doing inference on, i have around 40 classes which i am predicting. If i validate a sample of this by hand can i then extrapolate those metrics to the whole population. Have you come across similar examples ? My models are performing really well on the training data 98% F1 scores. I was thinking of using a bayesian estimation to find out how well my does in production. I don't have feedback loop so i need to check the predictions manually.
 A: When you split your dataset into training, validation, and test subsets, you treat the test set as your unseen data after training your models on the training subset.
You can then use your accuracy metrics from having your model estimate on the test set to infer how your model will predict on your unlabeled dataset.
You mentioned that your models perform well on training data; after validation you should have your model classify on your test set, treating it like unseen, unlabeled data. After you verify your model's predictions, calculate F1 score (since you used that to measure accuracy for your training data) for the predicted values on the test data, which you still have labels for. This new F1 score can be the metric you use to infer your model's accuracy for the data you actually don't have labels for.
This link might clear up some of your confusion on how to make sure your model is ready for your original unlabeled data. Final Model
A: You should be splitting up your data into training and test sets. Train on your training set and then evaluate performance on your test set.
