Say I have a dataset of labeled elements and an unlabeled dataset that I would like to apply my machine learning model to after training. How would I go about reporting accuracy/ROC AUC/etc. in this scenario. For example:

Labeled Train Dataset:
[0, 1, 2] => 3
[1, 3, 5] => 9

Labeled Test Dataset:
[2, 5, 1] => 8

Unlabeled Datset:
[6, 7, 8]

Even though there is no known ground truth for the element in the Unlabeled Dataset, I would still like to report it.

Would I train the model on Labeled Train Dataset and report accuracy/loss/etc. based on the Labeled Test Dataset, then retrain the model using using both the Labeled Train Dataset and Labeled Test Dataset, and report the output of the Unlabeled Dataset based on this new retrained model? This would mean that the two things reported are not from the "same" model, but I am not sure of any other way to utilize all of the data for predicting on the Unlabeled Dataset while still providing evaluation metrics. Does any have suggestions?

  • 1
    $\begingroup$ The field of semi-supervised learning concerns how to use labeled and unlabeled data together to improve supervised models. I think you could make headway by reading review articles about this topic. $\endgroup$
    – Sycorax
    Aug 12 '20 at 2:50

Short answer: you can't report results on data you don't have a ground truth for. You can obviously report the predictions for this data, but you cannot say if they are right or wrong. Training on the Test set is a bad idea, this data should be reserved for a final evaluation at the end (You may want to look into Train / Validate / Test split if you are doing any hyperparameter tuning).

-- However --

Your unlabeled data can still be useful. If you want to take advantage of it, you should investigate self-supervised pretraining. The actual implementation will depend on your application, but the general idea of it is:

  1. Use a self-supervised model which does not require labels. The model type will depend on your application, but for inspiration I would look into Autoencoders, Word2Vec.

  2. All or part of the model used in step 1 can now be used as a component of your classification model. The idea is that the self-supervised model will have learnt useful features which can aid with classification.

  • $\begingroup$ +1 As you said, I meant reporting the prediction for the unlabeled data, not if it was correct or not. Sorry if I didn't make that clear enough. $\endgroup$
    – 3michelin
    Aug 12 '20 at 1:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.