18
$\begingroup$

While the results of the private test set can not be used to refine the model further, isn't model selection out of a huge number of models being performed based on the private test set results? Would you not, through that process alone, end up overfitting to the private test set?

According to "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance" by Bailey et.al. it is relatively easy to "overfit" when selecting the best out of a large number of models evaluated on the same dataset. Is that not happening with Kaggle's private leaderboard?

  • What are the statistical justifications for the best performing models on the private leaderboard being the models that generalize the best to out-of-sample data?
  • Do companies actually end up using the winning models, or is the private leaderboard there just to provide the "rules of the game", and the companies are actually more interested in the insight that arises from the discussion of the problem?
Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Somewhat related: stats.stackexchange.com/q/235591 $\endgroup$
    – Kodiologist
    Commented Jun 8, 2017 at 16:27
  • 2
    $\begingroup$ You could look at the difference between private and public scores. One could argue that a non-overfitted model should achieve similar performance on both data sets. $\endgroup$
    – shadowtalker
    Commented Jun 8, 2017 at 16:28
  • 2
    $\begingroup$ @shadowtalker That indeed would be a good way to detect overfitting, but what we are actually interested in is the out-of-sample predictive power of the model, not the degree of overfitting. An overfit model - i.e. one that works much better in-sample than out-of-sample - may have better out-of-sample performance than a model that is not overfit. I do not have a reference on hand, but I believe that is often the case in complex domains, e.g. computer vision, when using complex models, e.g. CNNs. $\endgroup$
    – rinspy
    Commented Feb 1, 2018 at 10:00

1 Answer 1

Reset to default
9
$\begingroup$

Well the points you present are fair, however I think that there is a far more real issue with people overfitting on the public leaderboard.

This may happen when you do 100 or so submissions, the public test set will eventually bleed out on to your hyperparameter selection and thus overfit. I think that the private leaderboard is necessary in that respect.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.