I've been using BART (Bayesian Additive Regression Trees) for both regression and classification problems. BART, unlike many other tree based models, provides you with uncertainties on its predictions. So for a regression problem it will for example provide:
y_estimated = 100 +- 10
for a a classification problem:
y_prob = 0.6 +- 0.1
For the regression, it is reasonably intuitive to check the intervals provided. E.g., we can test that 90% of our test samples fall within their 90% confidence interval provided by the model.
For classification, I don't seem to find a similar approach. I found multiple papers  that assess calibration for both regression and classification models, but they seem to do this fundamentally different for regression and classification models. For regression problems, the provided uncertainty (+- 10 in our example, but in general a Gaussian distribution e.g.) is assessed, while for classification problems not the provided error but the estimated point estimate of probability is assessed (with a scoring rule).
I suspect I'm missing something here? Are there ways to assess the quality of provided uncertainties on the probability estimate in classification problems? And if not, why?