Best practices for measuring and avoiding overfitting? I am developing automated trading systems for the stock market. The big challenge has been overfitting. Can your recommend some resources describing methods for measuring and avoiding overfitting?
I started with training/validation sets, but the validation set always gets tainted.
Also, the time series data is always changing because the market is always changing. How do you measure this and determine the likelihood of consistent results on unseen data?
Thanks.
 A: For over-fitting in model selection, then a paper worth reading is 
C. Ambroise and G. J. McLachlan, "Selection bias in gene extraction on the basis of microarray gene-expression data", PNAS, vol. 99 no. 10 6562-6566, May 2002. http://dx.doi.org/10.1073/pnas.102102699 
For a discussion of the same sort of problem that arises in model selection, see
G. C. Cawley, N. L. C. Talbot, "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation", Journal of Machine Learning Research, 11(Jul):2079−2107, 2010. http://jmlr.csail.mit.edu/papers/v11/cawley10a.html
The way to solve the problem of the validation set becoming tainted is to use nested cross-validation, so the method used to make choices about the model is performed independently in each fold of the cross-validation used for performance estimation.  Essentially the performance estimation must estimate the performance of the whole model fitting procedure (fitting the model, feature selection, model selection, everything).
The other approach is to be a Bayesian.  The risk of over-fitting is introduced whenever you optimise a criterion based on a finite sample of data, so if you marginalise (integrate out) rather than optimise then classical over-fitting is impossible.  You do however have the problem of specifying the priors.
