Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm trying to compare time series forecast models. Among my 4 ts models, Facebook Prophet is showing the lowest MAE metric at 38.3 units (website visits). However, when I check R Sqrd for the same model, it has only 0.151 or 15.1% of the variance explained while other models such as ARIMA or ETS have a > 70% R Sqrd metric. If MAE is the lowest, should R2 be the highest for Prophet model? Can anyone pls help with this? Thanks.
While counter-intuitive, the phenomenon of two metrics (here, MAE and $R^2$) favouring different models is not uncommon. There is no way of "fixing this" because there is nothing "to be fixed". Two metrics might quantify structurally different patterns.
The real solution is to correctly choose what metric better reflects your losses. This does require domain expertise as well as consideration of what is the behaviour you are trying to model. $R^2$ for example will penalise rare "large errors" more harshly than MAE. Similarly $R^2$ be more prone to love/hate a model that gets the directionality of forecasts right/wrong. This later point is potentially the reason that Prophet's $R^2$ seems to suffer in our application: Prophet predictions suggest an up-tick from May onwards while the real data do not really present that; ARIMA predictions seems to resistant to this as a whole.
Finally: You seem to have a rather long forecast horizon. The pre-pandemic data are obviously not very representative of the post pandemic so you are using less than 12 months of data to predict 6 months. That's a pretty tall order... Maybe consider using a GARCH model to model the volatility of the series too? (See: Why is a GARCH model useful?) And a more realistic horizon. For example, exact predictions 134 and 135 days from now for something reasonably dependent to exogenous factors are often useless (e.g. tourist arrivals), their approximate level though is probably useful.
(Side-comment: I do find it a bit unusual that RMSE and $R^2$ strongly disagree but again, that is not implausible.)
