0
$\begingroup$

Say I have a trained Random Forest (RF) consisted of $m$ decision trees and I am interested to estimate $y$ from $t_1$ to $t_n$. The good thing about RF is that I have an ensemble of estimators and a deterministic estimator in one place.

After using RF for estimation, I calculate the MAE for RF, and I calculate the CRPS for ensemble members of RF: $m$ decision tree regressors.

CRPS is a probabilistic measure that seeks to evaluate the accuracy of the ensemble, and MAE targets the same quality but for the deterministic model.

Is there any point in comparing these two metrics? What could be the intuition behind say: my CRPS is slightly smaller than MAE in this case.

I think this is a vague comparison: Accuracy of an ensemble vs accuracy of single model. They are not meant to compete; they don't belong to the same world. Ensemble is targeting to resemble the distribution of $y$, while the deterministic model is after the average of $y$.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

You are mistaken. The CRPS ("Continuous Ranked Probability Score") is an example of proper . It evaluates a full predictive density.

But your $m$ trees do not output a full density! Instead, each separate tree gives you its best forecast for a single point prediction (which will depend on the loss function it minimizes, so it will probably be an estimate of the future expected value, cf. Kolassa. 2020, IJF). So the variation you see in your $m$ tree forecasts is not an estimation of the future variability in the observables, but the variability in the (unobservable!) mean!

Thus, it doesn't make sense to apply the CRPS to the ensemble point predictions of your $m$ trees in the first place, because these do not form a density forecast for the observables!

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you Stephan for the answer and the reference. I don't quite grasp it though. At each time step $t$ we have $m$ different estimates. Why cant they be seen as a PDF/CDF for $y$? I mean I thought that's the whole point of ensemble modelling. To have a distribution instead of a single deterministic estimate. And second thing, CRPS is calculated for each time step, to compare this distribution with the one constructed from the single observation. And then it is averaged over time. Do you think it is flawed this way? $\endgroup$
    – Alireza Amani
    Commented Feb 11, 2021 at 16:34
  • $\begingroup$ The key question is: a distribution of what? Your ensemble gives (an estimate of) a distribution for the unobservable mean $\mu=E(y)$. The CRPS evaluates a density prediction of $y$. The difference is precisely the same as between a confidence interval (which your ensemble gives, for the mean), and a prediction interval (which can be derived from a true density forecast, i.e., one for the observations). See here. $\endgroup$
    – Stephan Kolassa
    Commented Feb 11, 2021 at 18:38
  • $\begingroup$ Lets say $y$ is the air temperature and we use humidity ($x_1$) and wind speed ($x_2$) as two features to predict $y$. We are not trying to forecast a time series. So $t$ is merely an index here. To answer your question: at index $t_1$, $x_1$ and $x_2$ are within some range. The $m$ estimates for this situation (time index) are trying to resemble the distribution of $y$ when $x_1$ and $x_2$ have some specific values or within some specific range. $\endgroup$
    – Alireza Amani
    Commented Feb 12, 2021 at 2:52
  • $\begingroup$ This discussion has nothing at all to do with whether we are forecasting a time series. Everything applies equally whether we have a time series or some other prediction problem. Since you are writing about the distribution of $y$, you are looking for prediction intervals, or a density prediction. As above, collecting the point predictions from the $m$ trees won't help you, since each one will have abstracted away the residual noise. I recommend How do I calculate prediction intervals for random forest predictions? Does that help? $\endgroup$
    – Stephan Kolassa
    Commented Feb 12, 2021 at 7:15
  • $\begingroup$ It most definitely help! thanks a lot for the help. In my field of study and the peer-reviewed papers, I see CRPS used in multi-model forecasting a lot. The difference being they use dynamical models and I want to use Random Forest. Exactly as i described, thinking of point forecasts of each model as an empirical way to construct a PDF for $y$ at each time step. Now this discussion helps me to be skeptic and question. $\endgroup$
    – Alireza Amani
    Commented Feb 12, 2021 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.