1
$\begingroup$

Let's say we have 3 time series for three different fruits sales over one year. Although they are all fruits, their daily sold volume is very different.

For example, imagine apple being sold consistently from day to day and in general the volume is large (e.g. between 100 to 1000 daily apple sold), vs some other rare seasonal fruit (most days zero sales, and on a few days of the year, there might be ~5 sold).

Now the task is to build model (single or one for each?) to predict the sales in the future, and evaluate how well the model performs.

For model training (e.g. using sequential model like an RNN), I would need to normalize the data, I suppose I could either normalize it within each time series (so apple's time series divide by it's max sales for example) or normalize it over all three. For inference, I'd need to multiply the normalization factor back to get the actual sale prediction.

To evaluate the model, by default I'm thinking to use MAE or RMSE on the sale prediction, but it's obviously scale dependent, and then I read in 3.4 Evaluating forecast accuracy, it seems that I could use scaled error instead (it basically divides over naive prediction, which carries the scale so the end value is sort of scale free), then I can combine or compare the 3 different time series' performance.

My question is that, since I normalized the training time series to all between 0 and 1, can't I just compare the 3 time series using rmse on raw predictions (i.e. without multiplying the normalization factor back)? If so, would the scaled error listed in above textbook still be relevant somehow?

$\endgroup$
2
  • $\begingroup$ I'm sorry, I'm not sure I understand your question. Are you asking A) Whether you should train a single model for all, or separate model for each series? or B) How to compare prediction errors between the time series (regardless of prediction method used)? $\endgroup$ Jan 20, 2022 at 3:19
  • $\begingroup$ The main question is B, although I would appreciate any opinions on A if that's possible, thanks! $\endgroup$
    – Sam
    Jan 20, 2022 at 14:27

1 Answer 1

0
$\begingroup$

First of all, unless you're rather confident that the three time series exhibit the exact same patterns/dynamics, then I would advise against training a single model on all 3. You should do separate ones - since there are presumably different patterns to learn.

To your main question, though - how to compare prediction errors?

Unless there's some larger objective you're trying to achieve by which to measure success (such as some kind of trading based on predictions), I don't think there's a truly objective way to compare prediction errors, as you're (forgive me) literally comparing apples to oranges if the prediction problems are different - the patterns to be learned are different, the scales are different, the variance is different etc. It'll be some kind of judgement call (because the distribution you're predicting is just different from one case to another).

If you scaled by normalizing, then I don't see a problem with comparing the RMSE of the scaled predictions of the 3 time series.

I think the proposed scaled error is certainly worth a try! It seems that conceptually, using the scaled error as described in the chapter is like scaling not only for the units, but also for the "difficulty" of the problem, since you're scaling by the error of a basic model (so if a basic model does well, it means it's a relatively easy/predictable prediction problem and vice versa).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.