10
$\begingroup$

Previous questions have asked about creating prediction intervals for estimates derived from random forests or boosted regression trees, in a similar way to is easily achieved with linear regression models.

A comment on this question described the RMSE as an estimate of the standard deviation of the residual error, supporting use of the RMSE in construction of an interval around a prediction (or estimate) from a BRT or RF.

Am I right in thinking that CART methods relax the requirement for homoscedasticity? If so, it would seem that using the RMSE calculated across the full range of residuals would lead to inappropriately wide intervals in some regions, and too narrow intervals in others. It would then seem the only way to estimate an interval would be through bootstrapping (BRT) or accessing trees' individual predictions (RF).

[That same question] (Confidence interval of RMSE) attracted advice on the construction of a confidence interval for the standard deviation of residuals, assuming mean residual is zero, with normal distribution, based on a chi squared statistic around the RMSE.

How would such an interval on the SD be used? Would using the high end of the CI for SD as the value in a CI such as $\hat{x} \pm z\hat{SD}_u$ be a valid, if conservative, interval? Could you still attribute a specific 'confidence' value (e.g. 95%) to such an interval, given that it has 'nested' confidences?

$\endgroup$
4
  • 1
    $\begingroup$ I'm interested to know if you found a resolution to this question. $\endgroup$ Jan 8, 2016 at 15:56
  • 1
    $\begingroup$ Another good question left unanswered on CV.... $\endgroup$ Oct 8, 2016 at 17:47
  • $\begingroup$ Wish I could help, but I'm curious about the answer myself! $\endgroup$ Oct 8, 2016 at 17:47
  • $\begingroup$ I asked a question that is related to this: stats.stackexchange.com/questions/524863/…. Your point that "using the RMSE calculated across the full range of residuals would lead to inappropriately wide intervals in some regions, and too narrow intervals in others" reminds me of the note at the end of my question, where I wrote "this is true [the intervals have the desired coverage] only on average: if we condition on X, we can find subsets of the predictor space where the intervals have lower coverage." $\endgroup$
    – Adrian
    Jun 16, 2022 at 0:08

1 Answer 1

2
$\begingroup$

The CART, as I understand it, does not have homoscedasticity assumptions. If anything it presumes that the variance of each component is independent from the variance of all other components. It doesn't account for correlations in variables either.

The normality assumption is problematic. It is convenient but not necessarily true. There is often hand-waving about "law of large numbers" but the real world, impo, likes to frustrate such things.

Have you considered using quantile regression forests for your estimate, or is that part of the problem?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.