I've been using BART (Bayesian Additive Regression Trees) for both regression and classification problems. BART, unlike many other tree based models, provides you with uncertainties on its predictions. So for a regression problem it will for example provide:

y_estimated = 100 +- 10

for a a classification problem:

y_prob = 0.6 +- 0.1

For the regression, it is reasonably intuitive to check the intervals provided. E.g., we can test that 90% of our test samples fall within their 90% confidence interval provided by the model.

For classification, I don't seem to find a similar approach. I found multiple papers [1][2] that assess calibration for both regression and classification models, but they seem to do this fundamentally different for regression and classification models. For regression problems, the provided uncertainty (+- 10 in our example, but in general a Gaussian distribution e.g.) is assessed, while for classification problems not the provided error but the estimated point estimate of probability is assessed (with a scoring rule).

I suspect I'm missing something here? Are there ways to assess the quality of provided uncertainties on the probability estimate in classification problems? And if not, why?


[1] http://www.gatsby.ucl.ac.uk/~balaji/ensembles_nipsbdl16.pdf

[2] http://mlg.eng.cam.ac.uk/pub/pdf/QuiRasSinetal06.pdf


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.