5
$\begingroup$

I'm doing time series forecasting using Prophet. My forecasting model returns not only a prediction y_hat, but also an uncertainty interval y_hat_lower and y_hat_upper.

I wanted to calculate some metrics, like RMSE, MAE, etc. to evaluate the model's performance but don't know how to incorporate the interval in the calculation.

Should I be trying to do that? Normally, how do people evaluate these kinds of predictions?

$\endgroup$
3
  • 2
    $\begingroup$ Hi @CarlosGDCJ, welcome to CV. What is y_hat_lower/upper? Is it a 95% confidence interval? Prediction interval? Mean +/- one standard error? $\endgroup$
    – Alex J
    Commented Jun 27, 2023 at 2:01
  • $\begingroup$ @AlexJ I edited the question adding more context. $\endgroup$
    – CarlosGDCJ
    Commented Jun 27, 2023 at 3:07
  • $\begingroup$ Hm. I'm not sure. I am not familiar with an "uncertainty interval" - it appears to be something Bayesian but I don't know Prophet so can't say for sure. Usually, to calculate RMSE/MAE, you would calculate it with the usual formula, which should be available if you have the raw data and the prediction y_hat from the model. $\endgroup$
    – Alex J
    Commented Jun 27, 2023 at 3:59

1 Answer 1

6
$\begingroup$

If these are prediction intervals (and thus not confidence intervals, there is a difference!) for a prespecified coverage probability, then one can use interval scores. These can be scaled to be comparable between different time series (if this is what you are working with).

If these are confidence intervals, then you are probably out of luck, because CIs by definition pertain to unobservable parameters.

$\endgroup$
3
  • 1
    $\begingroup$ My background in statistics is too weak for me to say I understood every word in your answer, but it was helpful nonetheless. I'm doing time series forecasting using Prophet, and my "interval" is Prophet's uncertainty interval. I initially thought it was a confidence interval, but after reading some of your references I'm leaning towards a prediction interval. If it isn't too much trouble, can you help me find out? This is a short explanation about Prophet's interval $\endgroup$
    – CarlosGDCJ
    Commented Jun 27, 2023 at 2:59
  • $\begingroup$ That is a wonderful question, which I don't know the answer to (although it does sound a little more like it's PIs), so I posted it as such: Are Prophet's "uncertainty intervals" confidence intervals or prediction intervals? $\endgroup$ Commented Jun 27, 2023 at 10:37
  • $\begingroup$ Thank you! It seems I'm dealing with prediction intervals. I'll look into interval scores then. $\endgroup$
    – CarlosGDCJ
    Commented Jun 27, 2023 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.