2
$\begingroup$

Is there a rule of thumb when significance tests should be undertaken?

I have one large timeseries with ~750000 periods. In my incremental learning approach (no batch learning approach with fixed train/test sample size ) I try to predict the outcome of the next time step and at the end of the study I calculate the MAE.

I have a reference algorithm which has a MAE of, say, 2.00. Now I experiment with different algorithms and the MAE decreases to say 1.99 or 1.95 or 1.9.

May i conclude on each of the three examples that the adjusted algorithms are better than the reference algorithm? I thought that for very large timeseries, such "smaller" improvements of the MAE lead to significance because of the law of large numbers.

EDIT: A bit more background info because of @Chris Umphlett's Post:

My forecast model originally is based on the Multi-armed bandit problem. Here the player (=the forecasting algorithm), has to decide which of the machines (=expert advises or say predictions) he should play to minimize his loss as much as possible.

In my setting, the expert predictions (there are several of them) have their own models (which are not of interest and can be "anything") and they try to predict the next time step. I solely rely on these predictions.

Precisely, in a sequential (or say incremental) setting, my forecasting algorithm regularly re-considers which of the expert predictions to "trust" for the next time step. It does this, by weighting the different expert predictions for the next time step and forms his/her own prediction. Again: The forecasting algorithm can't predict on it's own, it always uses a mixture of the expert predictions based on their past performance. Sometimes, if some experts over-predict and the other part under-predicts and my algorithm is exactly between, I have an improvement against all experts for this particular period.

After the experiment, I have different MAEs (i can easily calculate other metrics too).

MAE of each expert (there are actually 10 experts) MAE of the combination algorithm

More information on data: To be more precisely, I have 32 different datasets containing each of them roughly about 24000 periods (half hourly). I let the algorithm run for each of the datasets and compare the MAEs. I don't know much about significance tests and I would like to know if I can speak about "improvements" when the MAE of the forecasting algorithm is lower.

$\endgroup$

1 Answer 1

0
$\begingroup$

You didn't mention what prediction or forecasting method was used to generate the predictions. If it was a time-series/forecasting method then it is likely that you could drop much/most of the 750,000 periods without much of a change in your predictions.

If you read forecasting papers you will often find that the researcher utilizes more than one metric when comparing methods. I have not seen any attempt at establishing whether the magnitude changes in a particular metric are significant. That's my advice-- use multiple metrics, and consider utilizing more than one holdout sample. If the data is hourly, then take 13 weeks of hours and predict one week of hours and calculate your metrics. Then do that again, with a different set of 14 weeks.

$\endgroup$
4
  • $\begingroup$ Thanks for your thoughtful comment, I added some additional information. Regarding the multiple metrics: I can use the RMSE too, the situation there is similar. $\endgroup$ Aug 9, 2018 at 18:29
  • $\begingroup$ Interesting. do the experts use the same methodology over time? you could try the multiple holdout sample approach by collecting the data and then re-sampling. Besides Forecast error, you could also use other types of metrics like information criteria (AIC or BIC) and bias to evaluate the experts'. With my models at work, there are rules that govern which model we choose but we also create these other metrics to review and see if they are worse in the other types of metrics. $\endgroup$ Aug 9, 2018 at 19:40
  • $\begingroup$ I don't know much about the experts. I know that their models are based on the large numerical weather prediction models. $\endgroup$ Aug 10, 2018 at 11:49
  • $\begingroup$ I don't have much else to add on evaluating single predictions on single time series. You could do a comparison test (like ANOVA) using the MAE's of all predictions for each expert to see what groups are produced in terms of their overall accuracy. You could also plot it (ggplot with geom_violin for instance where each expert has its own violin composed of all 32 predictions). $\endgroup$ Aug 10, 2018 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.