Is there a rule of thumb when significance tests should be undertaken?
I have one large timeseries with ~750000 periods. In my incremental learning approach (no batch learning approach with fixed train/test sample size ) I try to predict the outcome of the next time step and at the end of the study I calculate the MAE.
I have a reference algorithm which has a MAE of, say, 2.00. Now I experiment with different algorithms and the MAE decreases to say 1.99 or 1.95 or 1.9.
May i conclude on each of the three examples that the adjusted algorithms are better than the reference algorithm? I thought that for very large timeseries, such "smaller" improvements of the MAE lead to significance because of the law of large numbers.
EDIT: A bit more background info because of @Chris Umphlett's Post:
My forecast model originally is based on the Multi-armed bandit problem. Here the player (=the forecasting algorithm), has to decide which of the machines (=expert advises or say predictions) he should play to minimize his loss as much as possible.
In my setting, the expert predictions (there are several of them) have their own models (which are not of interest and can be "anything") and they try to predict the next time step. I solely rely on these predictions.
Precisely, in a sequential (or say incremental) setting, my forecasting algorithm regularly re-considers which of the expert predictions to "trust" for the next time step. It does this, by weighting the different expert predictions for the next time step and forms his/her own prediction. Again: The forecasting algorithm can't predict on it's own, it always uses a mixture of the expert predictions based on their past performance. Sometimes, if some experts over-predict and the other part under-predicts and my algorithm is exactly between, I have an improvement against all experts for this particular period.
After the experiment, I have different MAEs (i can easily calculate other metrics too).
MAE of each expert (there are actually 10 experts) MAE of the combination algorithm
More information on data: To be more precisely, I have 32 different datasets containing each of them roughly about 24000 periods (half hourly). I let the algorithm run for each of the datasets and compare the MAEs. I don't know much about significance tests and I would like to know if I can speak about "improvements" when the MAE of the forecasting algorithm is lower.