I'm working on a machine learning project to find particular key points in images. To do this, I'm using a U-net like architecture and treating it as a regression problem to produce a heat-map of centroids. This generally works well, and this model is capable of fitting my training data.

I'm trying to be disciplined and use best practices here, so I began by curating a dataset of 100 random images, and then split 15 out for final model evaluation. The other 85 images I use for training (and further split into train/dev to evaluate hyperparameters). I'm now at the point where I have the best model I have according to train/dev cross-validation, so I now evaluate it on my final test set. The performance is OK - when I train and take the best epoch, I have a recall of 100% on my training set, but only about 82% on my validation set (using early stopping).

OK, but not great, and obviously I'd like my model to generalise better. At this point my feeling is training on just 85 images isn't enough, and this model would perform better with more data. I've got tens of thousands of images, but I'm only working with 100 as labelling is fairly labour intensive. I don't plan to cherry-pick the images or inspect what's in my test set, but I would be continuing to add images until my test recall approaches 100%.

My question is: is it sound to simply add more images to my training set, or would this count as fitting to my test set? If it's unsound, what's the better option - should I grow my set of images (e.g., add another 100 images) and then attain another random split with a different seed?


1 Answer 1


As long as you test your model on unseen data, you are free to add as much training data as you like.

Keep in mind that random chance might lead to good test performance if you often train your model and reevaluate. So you might want to evaluate your final model on an additonal different test set and check if the performance does not drop too much in comparison to the other test set. Or you might choose a test set that is large enough such that good test performance by random chance is higly unlikely. The size if this set depends on how often you evaluate your model on this data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.