The Netflix competition kicked off a plethora of competitions to develop the best methods to learn information from given datasets. These include predicting traffic flow, predicting the success of grant applications, estimating the time patients spend in the hospital, and constructing good representations of data in an unsupervised manner. Some are run independently, others are hosted by platforms such as Kaggle and TunedIT.
Clearly, these competitions can spur interest in the field and some specific applications. However, they may also cause a large amount of resources to be devoted to minutely optimizing a narrow application. To what extent are these competitions beneficial, and to what extent are they detrimental? How can competition organizers structure these competitions to maximize the potential academic and commercial benefits, and what common pitfalls should they avoid?