Determining suitable seed number in R to randomly split data into training and test sets Good day. I have sample size of 160 and I am randomly splitting them into 70% train and 30% test data sets. My question is about the set.seed() value which returns different random samples in each data sets every time and it has seemed to effect greatly my modelling results in terms of accuracy. So how can we determine the suitable number of seed?
 A: Welcome to CV!
This is a well known issue and data splitting is for that reason only considered a good option when the number of observations $n$ is sufficiently large. There are several things you could consider though:


*

*Repeated splitting, averaging results: Simply repeat your analysis using multiple different seeds (as many as you can compute within a reasonable time). Then report for example $\text{mean} \pm \text{SD}$ of all seeds;

*Cross-validation: If you have no hyperparameters or other model selection, you can perform $k$-fold cross-validation, separating your data into a train set $n_\text{train}=\frac{n(k-1)}{k}$ and a test set $n_\text{test}=\frac{n}{k}$. You repeat this process $k$ times, until every observation has been used once for testing. Common variants are 10-fold cross-validation or $n$-fold cross-validation (also called leave-one-out). Again, use as many as time allows;

*Nested cross-validation: If you also want to do model selection or hyperparameter tuning, then cross-validate within your main cross-validation. You do this by repeatedly partitioning your training set into a training set and validation set. You then tune your model by minimizing the average loss on the validation sets and report its average performance on the test sets. This is computationally intensive, so you might not be able to use leave-one-out.

*Bootstrapping: Repeatedly sample with replacement from your original sample to create 'new' data. Then run your model on each bootstrap sample and report estimates of bias and variance.


You can also combine these methods. For example, you could do multiple runs of cross-validation, using a different seed every time. This is even good practice if your data are really large, but the benefits will be smaller, and the computational cost can become prohibitively large.
