11
$\begingroup$

I am developing automated trading systems for the stock market. The big challenge has been overfitting. Can your recommend some resources describing methods for measuring and avoiding overfitting?

I started with training/validation sets, but the validation set always gets tainted.

Also, the time series data is always changing because the market is always changing. How do you measure this and determine the likelihood of consistent results on unseen data?

Thanks.

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ B Seven, your question is much too high level and not very specific. Basically the whole field of machine learning can be boiled down to the question of how to avoid overfitting. There are several strategies like cross-validation, regularization or using a proper prior. Every good machine learning book can help you with that (e.g. the Duda/Hart/Stork or the one by Bishop). It is also not clear what you mean by a "tainted validation set". If your model cannot cope with changing time series data, it means that it is probably too simple. But more complex models will need even more regularization. $\endgroup$
    – fabee
    Commented Sep 15, 2011 at 12:31
  • $\begingroup$ @ B Seven - if your validation set gets tainted (i assume by fitting models to it) then perhaps dividing your data into a training, test and validation set may be more appropriate? $\endgroup$
    – richiemorrisroe
    Commented Sep 15, 2011 at 12:35
  • $\begingroup$ OK, that makes sense. So different approaches to avoid overfitting work in different domains. $\endgroup$
    – B Seven
    Commented Sep 15, 2011 at 19:07

1 Answer 1

Reset to default
6
$\begingroup$

For over-fitting in model selection, then a paper worth reading is

C. Ambroise and G. J. McLachlan, "Selection bias in gene extraction on the basis of microarray gene-expression data", PNAS, vol. 99 no. 10 6562-6566, May 2002. http://dx.doi.org/10.1073/pnas.102102699

For a discussion of the same sort of problem that arises in model selection, see

G. C. Cawley, N. L. C. Talbot, "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation", Journal of Machine Learning Research, 11(Jul):2079−2107, 2010. http://jmlr.csail.mit.edu/papers/v11/cawley10a.html

The way to solve the problem of the validation set becoming tainted is to use nested cross-validation, so the method used to make choices about the model is performed independently in each fold of the cross-validation used for performance estimation. Essentially the performance estimation must estimate the performance of the whole model fitting procedure (fitting the model, feature selection, model selection, everything).

The other approach is to be a Bayesian. The risk of over-fitting is introduced whenever you optimise a criterion based on a finite sample of data, so if you marginalise (integrate out) rather than optimise then classical over-fitting is impossible. You do however have the problem of specifying the priors.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I did implement Cross Validation and Leave One Out Cross Validation, but didn't quite figure out how to measure and mitigate overfitting. I looked at those papers, but they are over my head. Can you recommend any more introductory resources? $\endgroup$
    – B Seven
    Commented Sep 15, 2011 at 19:15
  • 2
    $\begingroup$ for measuring the overfitting, you just need to nest the cross-validation. The outer cross-validation is used for performance assessment, and withing each fold of the outer cross-validation an "inner" cross-validation is used for feature selection and model selection etc. That will give you an unbiased performance estimate. $\endgroup$
    – Dikran Marsupial
    Commented Sep 16, 2011 at 7:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.