0
$\begingroup$

I am trying to produce the Mean Absolute Scaled Error (MASE) for a custom time series model to compare its performance in forecasting several indicators with different units. I wanted to check my work by also calculating the MASE of the simple naive (one-step ahead) estimator or mean estimator both manually and with the Fable package. However I am stuck reproducing even the following simple example using the aus_production dataset from the tsibbledata package using the Fable package. I adapted the example from Hyndman & Athanasopoulos:

require(fable)
require(lubridate)
require(tsibbledata)
require(dplyr)

# This master dataset contains both the train and test sets
recent_production <- aus_production %>% 
  select(Quarter, Beer) %>%
  filter(year(Quarter) >= 1992)
  
# Training set
beer_train <- recent_production %>%
  filter(year(Quarter) <= 2007)

# Fit 2 models with training set
beer_fit <- beer_train %>%
  model(
    Mean = MEAN(Beer),
    `Naïve` = NAIVE(Beer)
  )

# We see the training MAE of naive estimator from Fable is rounded to 55.1
accuracy(beer_fit)
# A tibble: 2 × 10
  .model .type          ME  RMSE   MAE    MPE  MAPE  MASE RMSSE   ACF1
  <chr>  <chr>       <dbl> <dbl> <dbl>  <dbl> <dbl> <dbl> <dbl>  <dbl>
1 Mean   Training 1.14e-14  43.9  35.6 -0.945  7.95  2.45  2.57 -0.120
2 Naïve  Training 8.14e- 1  65.9  55.1 -0.860 12.2   3.78  3.86 -0.244  

# Verify this with my manual training MAE
my_trainMAE <- beer_train %>%
  mutate(naive = lag(Beer,1),
         `abs naive error` = abs(Beer - naive)) %>%
  as.data.frame %>%
  summarise(mean(`abs naive error`, na.rm = TRUE)) %>%
  as.numeric()
my_trainMAE
[1] 55.08475

# Calculate MASE for the "mean" estimator manually
recent_production %>% 
  filter(Quarter == max(beer_train$Quarter)+1) %>%
  mutate(mean_forecast = mean(beer_train$Beer),
         `abs error for mean` = abs(Beer - mean_forecast),
         `MASE for mean` = mean(`abs error for mean`/my_trainMAE))
# A tsibble: 1 x 5 [1Q]
  Quarter  Beer mean_forecast `abs error for mean` `MASE for mean`
    <qtr> <dbl>         <dbl>                <dbl>           <dbl>
1 2007 Q1   427          436.                 9.45           0.172

# Let Fable calculate the out-of-sample one-step ahead forecast MASE
beer_fit %>% forecast(h = 1) %>% accuracy(recent_production)
# A tibble: 2 × 10
  .model .type     ME  RMSE   MAE    MPE  MAPE  MASE RMSSE  ACF1
  <chr>  <chr>  <dbl> <dbl> <dbl>  <dbl> <dbl> <dbl> <dbl> <dbl>
1 Mean   Test   -9.45  9.45  9.45  -2.21  2.21 0.649 0.554    NA
2 Naïve  Test  -64    64    64    -15.0  15.0  4.40  3.75     NA

# The warning message occurs because h=1 and is not related (to me knowledge)

I picked the "mean estimator" intentionally because it clearly depends on the entire history in the training set so the "shared" training set should be beer_train. First question is how does FABLE even calculate the MASE of the naive estimator on the training set different from 1. Then why can't I reproduce the naive training MAE that Fable uses to calculate the MASE (and the MASE as well)?

$\endgroup$

1 Answer 1

0
$\begingroup$

For quarterly data, the denominator in the MASE is based on seasonal naive forecasts, not naive forecasts. See https://otexts.com/fpp3/accuracy.html#scaled-errors for the definitions.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.