4
$\begingroup$

If you train a lot of machine learning algorithms on a problem, svm, nn, rf,...via i.e. caret, and I mean a lot of it,like hundreds or thousands (not that hard,considering all parameter tuning in validation), eventually you will find one that will work.

But that is data snooping. As you have trained so much models, I think that you have to test the hypothesis that you had been lucky, maybe with White Reallity Check test or Hansen Superior Predictive Ability test.

Around 99.9% of the papers, articles, posts,.... didn't use White's or Hansen's tests (0.1% are papers regarding stock trading via ML). I suppose this is because, on a normal basis, we use to train a few models(really?).

The question is:

Do you have any idea about the number of models required to consider data snooping terrible effects?

This is, if i'm choosing between 3 models, I think that the possiblity of getting good results by chance is low. But in choosing between 30? 300? 3000?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

A lot of effort is underway to address this issue of data dredging with the techniques from differential privacy. The algorithm allows repeated reuse of the test set in evaluating different models, and in the process keeps enough details about the test set hidden that the final model does not overfit.

See the tutorial "Rigorous Data Dredging: Theory and Tools for Adaptive Data Analysis" in this yr's ICML

http://icml.cc/2016/?page_id=97

Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. L. (2015, June). Preserving statistical validity in adaptive data analysis. In Proceedings of the Forty-Seventh Annual ACM on Symposium on Theory of Computing (pp. 117-126). ACM.

Improve this answer
$\endgroup$
12
  • $\begingroup$ Yes, and if you can explain why their method works, please answer the following question. I'm not saying there's nothing to this paper, but I am saying it doesn't make sense to me. I am very interested if someone can explain it a clear manner. $\endgroup$
    – Cliff AB
    Commented Sep 14, 2016 at 20:30
  • $\begingroup$ @Cliff AB I think we crossed path before. Let me make it clear, I'm not saying differential privacy is the (only) holy grail in machine learning that completely solves the data snooping problem. But, this is a pretty neat trick on an otherwise intractable problem, isn't it. Before embarking on a 1000 model searches, this algorithm provides at least a sort of protective shield against overfitting. $\endgroup$
    – horaceT
    Commented Sep 14, 2016 at 20:54
  • 1
    $\begingroup$ To be clear on my side, I'm not saying there's nothing to it, but that I don't get it yet. There's a lot of things that work great that I don't get! I have it in my mind to thoroughly investigate this some day, but given that I haven't had the time, if someone else has I'm very interested. $\endgroup$
    – Cliff AB
    Commented Sep 14, 2016 at 20:58
  • 1
    $\begingroup$ @Cliff AB I hear you and I'll add an example to show how I think it should work. But your skepticism is justified, given what little else is out there right now. $\endgroup$
    – horaceT
    Commented Sep 14, 2016 at 21:59
  • $\begingroup$ If I understood it, the idea is: add noise to the test set performance and use this as the real test set performance for that model. You've just made it re-usable. Depending on the similarity of train and test performances, you add noise in one way or another. I don't know if it works, but sounds good. Is this right? $\endgroup$
    – PeterTschuschke
    Commented Sep 15, 2016 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.