MASE and handling nan-values

Question

I'd like to ask advice on how to correctly compute Mean Absolute Scaled Error (2006, Hyndman, Rob J., and Anne B. Koehler.) over the following example:

y_hat = [1, 2, 3, 4, 0, 0, 0, 0, 9]
y_true = [1, 2, 3, 4, np.nan, 5, 6, 7, 9]

Should I delete NaN and look at two separate non-nan subsets averaging their MASE scores? As far as I get from this discussion and post by Dr. Hyndman MASE represents simple MAE, divided by the mean absolute error of the one-step "naive forecast method" (i.e. some constant to scale MAE error). So I face situation where I have 2 choices:

y_hat = [1, 2, 3, 4, 0, 0, 0, 9]
y_true = [1, 2, 3, 4, 5, 6, 7, 9]

vs

y_hat_1 = [1, 2, 3, 4], y_hat_2 = [0, 0, 0, 9]
y_true_1 = [1, 2, 3, 4], y_true_2 = [5, 6, 7, 9]

where MASE = (MASE(y_hat_1, y_true_1) + MASE(y_hat_2, y_true_2)).mean()

What is correct way to compute MASE here?

Answer 1

There is no commonly accepted way of calculating the MASE in the presence of missing values that I am aware of.

As you write, the MASE is defined as the ratio between the MAE and a normalizing constant that gives the MAE of some benchmark method. In the original paper, Hyndman & Koehler (2006) proposed using the MAE of the naive one-step-ahead no-change forecast in-sample (this detail is frequently overlooked). I have also seen people use a seasonal naive forecast for this benchmark. You can really use whatever you like as long as you are explicit about what benchmark you use.

In your question, it looks like you are confusing the calculation of the MAE with the calculation of this normalizing factor.

• For the MAE in your example, I would simply discard the missing value, along with its forecast. We get an MAE of $\frac{5+6+7}{8}=2.25$ as the numerator.
• For the denominator, you would use the in-sample random walk MAE in the original formulation. If you want to use the random walk MAE for the actual forecast sample as a benchmark, I would simply remove the missing value and compute the MAE of a forecast [1,2,3,4,5,6,7] for actuals [2,3,4,5,6,7,9], yielding an MAE of $\frac{8}{7}\approx 1.14$.

The MASE would then be $\frac{2.25}{1.12}\approx 1.97$.