I'm doing a regression task using machine learning on some small data sets (less than 100) with numeric features. Before training the model, I would like to take 20% of the data as a holdout test set to be excluded from the training process. Currently, I'm choosing the holdout set randomly and I find the choice of the holdout set would significantly influence the performance of the final model (which is as expected).

Therefore, I'm wondering what should I do to develop a model which is robust to the change of data splitting method given such small data sets?


Since you have really small data-set, you can use leave-one-out cross validation.

  • $\begingroup$ Seconded -- leave-one-out CV is an excellent way to deal with this issue. In addition, you could try some data augmentation techniques to artificially expand the functional size of your dataset, but those are more likely to be biased because of the small dataset size. $\endgroup$ – tchainzzz Dec 25 '20 at 8:05
  • $\begingroup$ Thanks for the reply! Do you suggest I should run LOO-CV on the whole data set or the randomly selected training set? $\endgroup$ – dyluns Dec 25 '20 at 9:50
  • $\begingroup$ Whole data-set. You are already splitting your data with leave-one-out CV, you do not need to do any split beforehand. $\endgroup$ – CheeseBurger Dec 25 '20 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.