I've looked through many of the other posts concerning the Mean Absolute Scaled Error (MASE) forecast metric and haven't been able to sort out my problem just yet.
I'm working with some weather model forecast data (hourly forecasts from 0 to the 18th hour out) for surface temperatures and comparing the weather model data to a weather station that records surface temperature. I calculated MASE and ended up with values in the ballpark of 2 to 3. If I understand correctly, this indicates that my mean absolute error in the forecast is 2 to 3 times greater than that of a naive forecast.
If that's the case, why don't we just use naive forecasts for forecasting surface temperatures instead of a complex weather model?
I've included my code snippet below and the values. It won't be terribly useful without the data which I think may be difficult for me to provide but perhaps someone will see an egregious error in my method for calculating MASE. I've also included the MAE (numerator) and MAE naive forecast (denominator) output from running my code.
def mase_calc(df, obs): # Naive forecast calculation df = df.dropna() ## naive = df[obs].shift(1, freq='1H') naive_res = abs(df[obs] - naive) n = len(naive_res.dropna()) print('naive_res n: ' +str(n)) mae_naive = naive_res.sum()/n # RWIS - Observation MAE #hrrr_rwis_res = abs(df['sfc_tmp'] - df[obs]) hrrr_rwis_res = abs(df['hrrr_sfc_tmp'] - df[obs]) n = len(hrrr_rwis_res.dropna()) print('hrrr_rwis_res n: ' +str(n)) mae = hrrr_rwis_res.sum()/n print('MAE: ' + str(mae) + ' MAE_naive: ' + str(mae_naive)) mase = mae/mae_naive return(mase)
Here is the output I got back from each forecast hour.
Rob J. Hyndman's paper discussing MASE can be found here.
My MASE results can be summarized in the following graph.