1
$\begingroup$

I've looked through many of the other posts concerning the Mean Absolute Scaled Error (MASE) forecast metric and haven't been able to sort out my problem just yet.

I'm working with some weather model forecast data (hourly forecasts from 0 to the 18th hour out) for surface temperatures and comparing the weather model data to a weather station that records surface temperature. I calculated MASE and ended up with values in the ballpark of 2 to 3. If I understand correctly, this indicates that my mean absolute error in the forecast is 2 to 3 times greater than that of a naive forecast.

If that's the case, why don't we just use naive forecasts for forecasting surface temperatures instead of a complex weather model?

I've included my code snippet below and the values. It won't be terribly useful without the data which I think may be difficult for me to provide but perhaps someone will see an egregious error in my method for calculating MASE. I've also included the MAE (numerator) and MAE naive forecast (denominator) output from running my code.

def mase_calc(df, obs):
    # Naive forecast calculation
    df = df.dropna() ##
    naive = df[obs].shift(1, freq='1H')
    naive_res = abs(df[obs] - naive)
    n = len(naive_res.dropna())
    print('naive_res n: ' +str(n))
    mae_naive = naive_res.sum()/n

    # RWIS - Observation MAE
    #hrrr_rwis_res = abs(df['sfc_tmp'] - df[obs])
    hrrr_rwis_res = abs(df['hrrr_sfc_tmp'] - df[obs])
    n = len(hrrr_rwis_res.dropna())
    print('hrrr_rwis_res n: ' +str(n))

    mae = hrrr_rwis_res.sum()/n
    print('MAE: ' + str(mae) + '     MAE_naive: ' + str(mae_naive))
    mase = mae/mae_naive

    return(mase)

Here is the output I got back from each forecast hour.

MAE and MAE naive

Rob J. Hyndman's paper discussing MASE can be found here.

My MASE results can be summarized in the following graph.

MASE vs Forecast Hour

$\endgroup$

2 Answers 2

0
$\begingroup$

If a better answer is posted, I'll accept it.

After tinkering around with the data some more and thinking through all of this I realized that the MAE of the naive forecast was set at a 1 hour naive forecast where as my forecast data from the weather model provides up to 18 hours out. So the 18th hour forecast was being scaled by a 1 hour naive prediction. This means that the 18th hour forecast from my weather model has errors that are about 2.8 times greater than the naive forecast from the hour before that (17th hour in model terms).

So in summary, the naive forecast is better than the first hour of the model but the naive falls apart miserably if I attempt to push it farther than an hour (which I expected). A potential fix to the naive forecast problem is to use historical data (the seasonal version of MASE) but that's something I don't have currently.

$\endgroup$
0
$\begingroup$

I would think the mistake you made might be more serious than you expected.

The way you calculated "Naive", could only be the control of your one-hour prediction.

Then since you only had/showed one sample of forecast data (Fix me if I misunderstanding), there is no "MEAN" here, but only the "Absolute Scaled Error".


On the other hand, there is nothing here for your 18-hours forecast to defeat yet.

If you consider the original "Naive", it should be calculated by shifting 18 hours.

Well, in my opinion, if you want to compare your forecast with "Naive", except for the "MEAN" problem, I wouldn't consider the last observation of 18 hours ago would be "Naive" enough. I didn't know much about the weather forecast, but a Daily "Naive" might be better. Anyway, the competitor you chose for your defeat shows how confident you are.

$\endgroup$
3
  • $\begingroup$ Correct me if I'm wrong but I think what you're saying is my naive prediction (denominator in MASE) will only work as a comparison to the 1 hour prediction from my weather model. I should instead be using a naive forecast that shifts 18 hours to compare to the 18 hour forecast. Could you further elaborate on what you mean by "I didn't know much about the weather forecast, but a Daily "Native" might be better."? $\endgroup$ Jul 3, 2019 at 14:47
  • $\begingroup$ @Foxhound013 It is not the main point I wanted to warm you. As I saw, you only have a set of prediction (1-hour to 18-hour ) from one specific timing (t) by using the data before that. But this was just ONE forecast, so only one forecast error, so one ASE. For the "MASE", you should do things like: do the forecast for the period after time t by using the data before t; do the forecast for the period after time t-1 by using the data before t-1; and so on. That's how you could calculate "MEAN". $\endgroup$
    – Jason
    Jul 3, 2019 at 20:04
  • $\begingroup$ "Naive", sorry. $\endgroup$
    – Jason
    Jul 3, 2019 at 23:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.