14
$\begingroup$

Kaggle competitions determine final rankings based on a held-out test set.

A held-out test set is a sample; it may not be representative of the population being modeled. Since each submission is like a hypothesis, the algorithm that won the competition may just, by total chance, have ended up matching the test set better than the others. In other words, if a different test set were selected and the competition repeated, would the rankings remain the same?

For the sponsoring corporation, this doesn't really matter (probably the top 20 submissions would improve their baseline). Although, ironically, they might end up using a first-ranked model that is worse than the other top five. But, for the competition participants, it seems that Kaggle is ultimately a game of chance--luck isn't needed to stumble on the right solution, it's needed to stumble on the one that that matches the test set!

Is it possible to change the competition so that all the top teams who can't be statistically distinguished win? Or, in this group, could the most parsimonious or computationally cheap model win?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Some folks use the testing on the hidden set to back out actual test values. This allows them to nearly perfectly fit the results. The holdout prevents this. My personal opinion is that the difference between holdout and non-holdout are about getting rid of cheaters. $\endgroup$
    – EngrStudent
    Commented Jul 19, 2017 at 21:04
  • 2
    $\begingroup$ Of course test data should be held out from the participants, but I am wondering whether having a single hold out test set makes the competition results (for the top $X$ teams) essentially depend mostly on chance. $\endgroup$
    – Sam Weisenthal
    Commented Jul 19, 2017 at 21:09
  • $\begingroup$ The scores are weighted. A great system is going to outperform a junk one nearly every time. It takes a truckload of work to fail so bad that the last becomes first. Local order, perhaps 10 steps in rank or less, when there are thousands of participants, would change if the holdout was resampled. You could make a numeric experiment to show this. $\endgroup$
    – EngrStudent
    Commented Jul 20, 2017 at 12:32
  • 2
    $\begingroup$ From the perspective of the sponsoring corporation, they are not forced to actually implement the winning model. If I remember correctly the model that won the netflix challenge was never implemented. They can take a few credible candidates among the best models and test them further. $\endgroup$
    – David Ernst
    Commented Sep 14, 2017 at 22:19

2 Answers 2

Reset to default
15
$\begingroup$

Yes, your reasoning is correct. If a different test set was selected and the competition repeated, rankings would indeed change. Consider the following example. All entries to a Kaggle competition with binary labels just guess randomly (and, say, independently) to predict their output. By chance, one of them will agree with the holdout more than others, even though no prediction is going on.

While this is a bit contrived, we can see that variance in each of the submission's models would mean that applying many such entries would indeed just be fitting to the noise of the holdout set. This tells us that (depending on the individual model variances), the top-N models probably generalize the same. This is the garden of forking paths, except the "researchers" aren't the same (but that doesn't matter).

Is it possible to change the competition so that all the teams who can't be statistically distinguished from the top performance on the test set win?

Indeed.

  • One approach (impractical as it is) would be to explicitly work out the variance of a given model in each entry, which would give us a CI on their holdout performance.
  • Another approach, which might take a lot of computation, is to bootstrap a CI on holdout performance, by exposing a training and testing API to all of the models.
Improve this answer
$\endgroup$
6
  • $\begingroup$ Great answer. Can you elaborate on how the two methods might be implemented? $\endgroup$
    – Sam Weisenthal
    Commented Jul 19, 2017 at 23:05
  • 1
    $\begingroup$ It's interesting: the best model might in fact not be the winning team's. $\endgroup$
    – Sam Weisenthal
    Commented Jul 19, 2017 at 23:06
  • 2
    $\begingroup$ Explicitly working out the variance can't be done without the data distribution (I only mention it since it explains the theory). This paper describes several methods (and where they fail) for accuracy estimation, including bootstrap and cross-validation. Unlike the paper, though, in this context, we're not doing CV for model selection on a training set, but rather for a robust "score" on the combined training and test data set. $\endgroup$
    – VF1
    Commented Jul 20, 2017 at 2:02
  • $\begingroup$ Perhaps two rounds is better for robust winner estimation. The first removes the 99% worst, and the second round re-estimates the rankings to "polish" the order. $\endgroup$
    – EngrStudent
    Commented Jul 20, 2017 at 12:50
  • 2
    $\begingroup$ To add on to ideas here, check out this paper by the winner of the NCAA March Madness Kaggle competition in 2014. Scroll down to section 4, "Simulation Study". According to their sim, if their model's predicted probabilities for each matchup were in fact the true state of nature, their median placing would be 11th place. $\endgroup$
    – klumbard
    Commented Jul 25, 2017 at 14:58
3
$\begingroup$

There are other types of competitions in Kaggle have no chance elements. For example, this one Stanta's Stolen Sleigh.

It is a discrete optimization problem and it even does not have private leader board. What you see in the public leader board is the final results.

Comparing to supervised learning, that has an easy start for many people, this types of competition is more "hard" in nature.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.