Do Kaggle's evaluation criteria make sense?

Question

On Kaggle, the submitted models are different and better models are expected to yield higher scores. However, these scores are calculated on a blinded test set; they are random variables and their realized values contain errors. Thus, when we rank models based on having higher (or lower) realized values of the scores, they may be such because of their underlying true value, error, or both.

It means that the first ranked model is not necessary better than the second model given the observed fact that the difference between the first models is usually 0.01. This difference is too small to say that one model is better than the others, and it can completely due to randomness.

I see no better way to rank models on Kaggle. Of course it is relevant in the sense that we need a precise measurement to rank players, but does it make sense to say that the first model is better than the second one?

Answer 1

This boils down to using sample means instead of true means to rank models.

"Does it make sense" depends a lot on what Kaggle's aim is. Is their aim to rank people by their expected true means? Then ranking by models by their sample means make sense.

If their aim is to set a rigid ranking system so that any model ranked above another model is significantly better than the lower ranked model, then their approach doesn't make sense.

One solution could be to create a confidence interval for a model's rank to give an indication of the level of uncertainty involved in the rank. But although this approach indicates the uncertainty in the model rankings it might also not make sense if Kaggle's aim isn't to quantify ranking errors.

Answer 2

Kaggle runs competitions, and competitions need a way of figuring out who wins. Given the nature of what Kaggle does, a statistical accuracy measure on a test dataset is as good a way of doing it as any.

You can argue about whether the winning model is the "real" best model, but this kind of question applies to any competitive pursuit. If team A wins a soccer game 1-0, or a basketball game 99-98, is it really better than team B? What if team B lost a bunch of players to injury? Or if team B won 5 games in a row prior to losing this one, while team A lost 5 in a row? And so on.

A more interesting question, IMO, is the degree to which such statistical measures of accuracy are particularly relevant, given how models are used in the real world, and the part that other, non-statistical, measures might play. In particular, the models that do well in Kaggle tend to be highly complex, maybe using multiple sub-models and combining them in various ways. If you were to implement such a model, you would also have to ask questions like:

• How much of a maintenance burden does the model impose? If it contains several different sub-models, do they each require installing (and maintaining) a different software package? What if I use Windows (Server) and the software is only available for Unix/Linux (or vice-versa)?
• What kind of computational performance can I expect, both when fitting and when scoring? Maybe there is a requirement that the model must be able to score a minimum X rows of data in a given amount of time. Is the model so computationally intensive that it's unacceptably slow on my hardware?

Given these types of questions, you might well conclude that the highly complex model, despite its predictive accuracy, doesn't actually suit your needs. Indeed, you might find that a simpler model is 99% as accurate as the complex one, but is also easier to install and maintain, and runs much faster when scoring.

Of course, the answers to these questions will be highly dependent on your own circumstances, so it would be unreasonable for Kaggle to take them into account for competitions. But that doesn't make them any less important outside that narrow context.