0
$\begingroup$

I'm doing a regression task using machine learning on some small data sets (less than 100) with numeric features. Before training the model, I would like to take 20% of the data as a holdout test set to be excluded from the training process. Currently, I'm choosing the holdout set randomly and I find the choice of the holdout set would significantly influence the performance of the final model (which is as expected).

Therefore, I'm wondering what should I do to develop a model which is robust to the change of data splitting method given such small data sets?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Since you have really small data-set, you can use leave-one-out cross validation.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Seconded -- leave-one-out CV is an excellent way to deal with this issue. In addition, you could try some data augmentation techniques to artificially expand the functional size of your dataset, but those are more likely to be biased because of the small dataset size. $\endgroup$
    – chang_trenton
    Commented Dec 25, 2020 at 8:05
  • $\begingroup$ Thanks for the reply! Do you suggest I should run LOO-CV on the whole data set or the randomly selected training set? $\endgroup$
    – dyluns
    Commented Dec 25, 2020 at 9:50
  • $\begingroup$ Whole data-set. You are already splitting your data with leave-one-out CV, you do not need to do any split beforehand. $\endgroup$
    – CheeseBurger
    Commented Dec 25, 2020 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.