I am working on active learning and I was wondering about the difference if we split the dataset into training and testing or collecting and labeling the training and testing datasets separately. Either way, the ratio between training and testing will be maintained (70%,30%).

I want to select samples to label them and train the model to boost its performance and then I come back again to select new samples to label for testing because the model will be well trained and select hard and new examples to the model. Doing such that, is it appropriate?

Thank you for your answers.

  • $\begingroup$ Do you have the same distribution (population, not empirical) in both sets? It looks like the answer is that you do not, but I would like to hear your take. $\endgroup$
    – Dave
    Jun 24 '21 at 10:36
  • $\begingroup$ Yes, they should have the same distributions. $\endgroup$ Jun 24 '21 at 10:43
  • $\begingroup$ Are you sure? You make it sound like your test set is going to be especially hard. $\endgroup$
    – Dave
    Jun 24 '21 at 10:46
  • $\begingroup$ Yes, my test dataset should contain diverse, representative, and hard examples. Hard samples should be from all the classes. $\endgroup$ Jun 24 '21 at 10:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.