Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

We have a set biological samples that was pretty expensive to obtain. We put these samples through a series of tests to generate data that are used for building a predictive model. For this purpose we have divided the samples to training (70%) and testing (30%) sets. We have successfully created a model and applied it on the testing set to discover that the performance was "less than optimal". The experimentalists now want to improve the biological tests in order to create a better model. Provided that we cannot obtain new samples, would you suggest us to re-shuffle the samples to create new training and validation sets or to stick with the original division. (We don't have any indication that the division was a problematic one).

share|improve this question
1  
How did you divide the data? Randomly, by hand, or some other method? Though, in truth, the part about "successfully created a model" is a MUCH larger part of the issue. Before doing expensive things, you should see if you're using the appropriate type of model, if you've overfit your training data, and if you have the appropriate data for what you are trying to predict. –  Wayne Jan 2 '11 at 15:06
    
BTW, I forgot to turn on the cynicism mode before the "successfully created a model" –  DavidDong Jan 3 '11 at 7:28

1 Answer 1

up vote 12 down vote accepted

As you already use a holdout sample, I would say you should keep it and build your new models on the same training sample so that all models will consider the same relationships between features. In addition, if you perform feature selection, the samples must be left out before any of these filtering stages; that is, feature selection must be included in the cross-validation loop.

Of note, there are more powerful methods than a 0.67/0.33 split for model selection, namely k-fold cross-validation or leave-one-out. See e.g. The Elements of Statistical Learning (§7.10, pp. 241-248), www.modelselection.org or A survey of cross-validation procedures for model selection by Arlot and Celisse (more advanced mathematical background required).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.