What is the difference between spliting the dataset into training and testing or collecting the training and testing data seperately?

Question

I am working on active learning and I was wondering about the difference if we split the dataset into training and testing or collecting and labeling the training and testing datasets separately. Either way, the ratio between training and testing will be maintained (70%,30%).

I want to select samples to label them and train the model to boost its performance and then I come back again to select new samples to label for testing because the model will be well trained and select hard and new examples to the model. Doing such that, is it appropriate?

Thank you for your answers.

Answer 1

The key difference between splitting a dataset and collecting separate training and testing data lies in the distributions. When splitting, both sets are drawn from the same underlying distribution, ensuring reliable generalisation testing. Collecting data separately can introduce a distribution shift, especially if the test set is made intentionally harder, which could distort model generalisation.

For active learning, selecting harder examples from the test set for training may skew the model toward outliers or complex cases, reducing its ability to generalise to typical examples. It is important to maintain a representative test set that reflects the overall data distribution, ensuring proper evaluation. The goal in active learning is to enhance performance across all data, not just on the most difficult instances.