Quite a few Kaggle competitions have used or are using the Logarithmic Loss metric as the quality measure of a submission.
I'm wondering if there are other ways besides N-fold cross-validation to calculate confidence intervals for this metric. If model X has a log loss of 0.123456 on the test set and model Y has a log loss of 0.123457, I'm sure you'll agree that model X is not significantly better than model Y, unless we're talking about a gazillion data points.
Why something else than N-fold cross-validation? Simple answer: performance. For a certain application I need to know whether model X is significantly better than model Y (when looking at Log Loss). In other words, I need to know whether the Log Loss for model X falls outside the 95% confidence interval for the Log Loss of model Y.
I need to do this comparison many, many times with different models and datasets that are coming in every day. Performance is crucial, so doing 10-fold cross-validation a 1,000 times to get a rough estimate of the confidence intervals is not going to cut it, I'm afraid. The datasets for which I have to calculate the log loss are usually in the range of say 50 positives and 10,000 negatives to 20,000 positives and 1 million negatives.
What would you advise?