Good day. I have sample size of 160 and I am randomly splitting them into 70% train and 30% test data sets. My question is about the set.seed() value which returns different random samples in each data sets every time and it has seemed to effect greatly my modelling results in terms of accuracy. So how can we determine the suitable number of seed?

  • 3
    $\begingroup$ set.seed() just makes a 'random' sample reproducible. It doesn't matter what value you use - literally choose any number. See this link: stackoverflow.com/questions/13605271/… $\endgroup$ – André.B May 8 at 3:58
  • $\begingroup$ André is right, but you could for example use the current time (13:56 --> set.seed(1356)) to avoid overusing the same seed. $\endgroup$ – Frans Rodenburg May 8 at 4:57
  • 1
    $\begingroup$ I am voting to leave this open. It is only incidentally about R and set.seed() and is really a statistics question (as shown in @FransRodenburg 's answer). $\endgroup$ – Peter Flom - Reinstate Monica May 8 at 12:22

Welcome to CV!

This is a well known issue and data splitting is for that reason only considered a good option when the number of observations $n$ is sufficiently large. There are several things you could consider though:

  • Repeated splitting, averaging results: Simply repeat your analysis using multiple different seeds (as many as you can compute within a reasonable time). Then report for example $\text{mean} \pm \text{SD}$ of all seeds;
  • Cross-validation: If you have no hyperparameters or other model selection, you can perform $k$-fold cross-validation, separating your data into a train set $n_\text{train}=\frac{n(k-1)}{k}$ and a test set $n_\text{test}=\frac{n}{k}$. You repeat this process $k$ times, until every observation has been used once for testing. Common variants are 10-fold cross-validation or $n$-fold cross-validation (also called leave-one-out). Again, use as many as time allows;
  • Nested cross-validation: If you also want to do model selection or hyperparameter tuning, then cross-validate within your main cross-validation. You do this by repeatedly partitioning your training set into a training set and validation set. You then tune your model by minimizing the average loss on the validation sets and report its average performance on the test sets. This is computationally intensive, so you might not be able to use leave-one-out.
  • Bootstrapping: Repeatedly sample with replacement from your original sample to create 'new' data. Then run your model on each bootstrap sample and report estimates of bias and variance.

You can also combine these methods. For example, you could do multiple runs of cross-validation, using a different seed every time. This is even good practice if your data are really large, but the benefits will be smaller, and the computational cost can become prohibitively large.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.