1
$\begingroup$

I'm trying to compare time series forecast models. Among my 4 ts models, Facebook Prophet is showing the lowest MAE metric at 38.3 units (website visits). However, when I check R Sqrd for the same model, it has only 0.151 or 15.1% of the variance explained while other models such as ARIMA or ETS have a > 70% R Sqrd metric. If MAE is the lowest, should R2 be the highest for Prophet model? Can anyone pls help with this? Thanks.

models

time series plots

model fit

model forecast = 3 months

$\endgroup$
1
$\begingroup$

While counter-intuitive, the phenomenon of two metrics (here, MAE and $R^2$) favouring different models is not uncommon. There is no way of "fixing this" because there is nothing "to be fixed". Two metrics might quantify structurally different patterns.

The real solution is to correctly choose what metric better reflects your losses. This does require domain expertise as well as consideration of what is the behaviour you are trying to model. $R^2$ for example will penalise rare "large errors" more harshly than MAE. Similarly $R^2$ be more prone to love/hate a model that gets the directionality of forecasts right/wrong. This later point is potentially the reason that Prophet's $R^2$ seems to suffer in our application: Prophet predictions suggest an up-tick from May onwards while the real data do not really present that; ARIMA predictions seems to resistant to this as a whole.

Finally: You seem to have a rather long forecast horizon. The pre-pandemic data are obviously not very representative of the post pandemic so you are using less than 12 months of data to predict 6 months. That's a pretty tall order... Maybe consider using a GARCH model to model the volatility of the series too? (See: Why is a GARCH model useful?) And a more realistic horizon. For example, exact predictions 134 and 135 days from now for something reasonably dependent to exogenous factors are often useless (e.g. tourist arrivals), their approximate level though is probably useful.

(Side-comment: I do find it a bit unusual that RMSE and $R^2$ strongly disagree but again, that is not implausible.)

$\endgroup$
1
  • $\begingroup$ image 3 in my post was actually 6 month test period while remaining data from Jan 2019 was training. I have now updated the post with another img, showing the forecast for a time period of 3months. Thanks for explaining about R2 vs MAE. I'll do some more reading here. In terms of data, these are visits to the site and my starting data point is Jan 01, 2019 till 5 Jul 2021. $\endgroup$
    – AdilK
    Jul 7 at 14:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.