How to prevent the appearance of overfitting due to firing many models on a small data set? I applied support vector machines to a relatively small dataset. I used relatively simple techniques, and achieved publishable results. Now, when already writing the paper, I got an idea, which would improve the classification results considerably.
Although I use the relatively simple techniques, I am fearing the over-fitting problem: presenting a large number of models for a small dataset makes it look like attempted million things and published only the best classifier.
What would be the best to add the results? Should I write something like that 

after the initial results were obtained, we got another idea..

or should I just omit the best result?
 A: There is a neat way to address this problem. Use validation tecniques. If your method has hyper parameter, SVM has, split your data into 3 groups; training, validation and test. Train your algorithm on training set, find best hyper-params on validation set, and publish the results on test set. You may also consider to use k-fold cross validation. That shows mean of different training settings and variation between them. You may add the method that you mention about in a separate trial and compare results with your previous method. That explicitly shows how each method affects the overall result. 
Also be careful about methodology. If the algorithm is not specific to the current dataset, use it on another dataset to be sure that nothing is wrong.
A: This is really a matter of opinion and style rather than of right or wrong answers. However, thinking through the matter in terms of the overall goal of preserving scientific integrity, it seems like best practices would be to fully disclose the order in which analytic techniques were conceived and applied relative to the rest of the paper.  
Thus, I'd say something like "the analytic design was as follows [...], using this design, the results obtained were [...].  Thereafter, one additional model was fit to the data, as follows [...], the results obtained were [...].  Although the original design did not include MODEL2, it was computed after the fact for the following reasons [...].  No other models were attempted."  
