1
$\begingroup$

i have split my training data 30/70 and trained models, my models are performing really well on the training set but i have a large unlabelled dataset where i want to do inference, how could i measure accuracy on this dataset for which i don't have the correct answer.

I want to know how well my model does in production. how would you validate a large dataset say 600k examples you are doing inference on, i have around 40 classes which i am predicting. If i validate a sample of this by hand can i then extrapolate those metrics to the whole population. Have you come across similar examples ? My models are performing really well on the training data 98% F1 scores. I was thinking of using a bayesian estimation to find out how well my does in production. I don't have feedback loop so i need to check the predictions manually.

$\endgroup$
0
$\begingroup$

You should be splitting up your data into training and test sets. Train on your training set and then evaluate performance on your test set.

$\endgroup$
  • $\begingroup$ hi wes, yes I have done that but then i have a large unlabelled dataset and i want to be able to estimate my metrics for that. $\endgroup$ – Janos Neumann Feb 11 at 17:18
  • $\begingroup$ The results you got on your test set ARE your estimate for performance on your unlabelled set.... $\endgroup$ – astel Feb 12 at 0:27
  • $\begingroup$ well not always have you heard of over fitting your training data especially if you train many times with the same labelled dataset $\endgroup$ – Janos Neumann Feb 12 at 13:55
  • $\begingroup$ If you're overfitting that should be clear in your test set performance. $\endgroup$ – Wes Feb 12 at 15:28
0
$\begingroup$

When you split your dataset into training, validation, and test subsets, you treat the test set as your unseen data after training your models on the training subset.

You can then use your accuracy metrics from having your model estimate on the test set to infer how your model will predict on your unlabeled dataset.

You mentioned that your models perform well on training data; after validation you should have your model classify on your test set, treating it like unseen, unlabeled data. After you verify your model's predictions, calculate F1 score (since you used that to measure accuracy for your training data) for the predicted values on the test data, which you still have labels for. This new F1 score can be the metric you use to infer your model's accuracy for the data you actually don't have labels for.

This link might clear up some of your confusion on how to make sure your model is ready for your original unlabeled data. Final Model

$\endgroup$
  • $\begingroup$ thanks tbd, but thats my question, how can i infer metrics from the validation set to the unlabeled dataset ? Do you have an example, a link ? $\endgroup$ – Janos Neumann Feb 12 at 11:30
  • $\begingroup$ I just updated my answer, if you have more questions about the train/test split and using models on unlabeled data, this article may help final ML model $\endgroup$ – tbd Feb 12 at 19:43

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.