My company uses an inventory model (call it IM) to order parts. I have two years of data (Jan 2016 to Dec 2017) that show how many parts were actually ordered and as well as a forecast from Jan-Dec 2017 that shows how many parts the IM model said to order.
I calculated the MASE for the (a) IM Model, (b) Mean, and (c) Naive forecasts using Jan-Dec 2016 as my training set and Jan-Dec 2017 as my test set.
For over 6000 parts, there were more parts that showed that the Mean and Naive forecasts had MASE values less than 1 than the IM model, so I would conclude from that that the IM model was not very good when compared to a Mean or Naive forecast. The problem is, when I apply a Shapiro-Wilks test or a Doornik-Hansen test to the residuals for the period Jan to Dec 2017, there are more parts where normality is satisfied for the IM model than for the Mean and Naive forecasts, which leaves me with two different answers? Could someone please interpret what I'm missing?