Should I re-shuffle my data?

Question

We have a set biological samples that was pretty expensive to obtain. We put these samples through a series of tests to generate data that are used for building a predictive model. For this purpose we have divided the samples to training (70%) and testing (30%) sets. We have successfully created a model and applied it on the testing set to discover that the performance was "less than optimal". The experimentalists now want to improve the biological tests in order to create a better model. Provided that we cannot obtain new samples, would you suggest us to re-shuffle the samples to create new training and validation sets or to stick with the original division. (We don't have any indication that the division was a problematic one).

Answer 1

As you already use a holdout sample, I would say you should keep it and build your new models on the same training sample so that all models will consider the same relationships between features. In addition, if you perform feature selection, the samples must be left out before any of these filtering stages; that is, feature selection must be included in the cross-validation loop.

Of note, there are more powerful methods than a 0.67/0.33 split for model selection, namely k-fold cross-validation or leave-one-out. See e.g. The Elements of Statistical Learning (§7.10, pp. 241-248), www.modelselection.org or A survey of cross-validation procedures for model selection by Arlot and Celisse (more advanced mathematical background required).