4
$\begingroup$

I have created two autoregressive models for forecasting: a basic intercept-only model and an AR-NN (autoregressive neural network) model. Both models show similar performance based on recursive one-step-ahead pseudo out-of-sample RMSEs from a 70/30 train-test split, with the intercept-only model performing slightly better.The attached picture shows the forecasts: Black line = real testing data, red = Intercept-only forecast, blue = AR-NN forecast. RMSE Intercept-only = 0.018, RMSE AR-NN = 0.0209

However, the intercept-only model doesn't offer variance in its forecasts, especially compared to the AR-NN model.

Given this, does it make sense to prefer the AR-NN model to obtain future forecasts that offer more variation even though being less accurate?

Picture shows forecasts: Black line = real testing data, red = Intercept-only forecast, blue = AR-NN forecast

This is the data (Unfortunately, the seed was not set probably, so these number differ a bit from the posted graph, still the situation remains the same):

data_test =  c( -0.0023267409, 0.0100361961,0.0031162867,-0.0104408353,0.0275584206,0.0094189017,-0.0053767755,-0.0066241215,-0.0077336373,0.0065507481,0.0108000000,-0.0010410827,0.0081810961,0.0011107585,-0.0073529412,0.0150358183,-0.0195024077,-0.0122674466,0.0106895997,-0.0095748134,0.0038797284,-0.0004852014,0.0119846596,-0.0238874346,0.0185467684,-0.0002409252,-0.0073618639,0.0065102074,-0.0103126269,0.0020259319,-0.0300500835,-0.0007518169,-0.0244758237,-0.1248556026,-0.0025091681,0.0679140056,0.0474680833,0.0161847762,0.0279416585,0.0052979053,0.0163553275,0.0066103855,0.0025421036,-0.0027881781,0.0296823066,-0.0015483471,-0.0130980392,0.0066225166,-0.0050900548,0.0046765394,-0.0038338158,0.0101456010,0.0283693280,0.0118233195,-0.0150200015,0.0018081820,-0.0184147932,0.0064039033,0.0161266127,-0.0058091286,0.0050292749,0.0067844628,0.0000000000,-0.0071332032,-0.0181178809,-0.0071604558,0.0026875528,-0.0128324778,0.0222796745,-0.0155212355,0.0086503866,-0.0086479809,-0.0025545750,0.0034714187,-0.0006175222,0.0074318112,-0.0104513432,0.0065374558,-0.0059574468,0.0156130998)

forecast_wn = c(0.003838468,0.003819081,0.003838570,0.003836313,0.003791836,0.003865645,0.003882838,0.003854259,0.003822018,0.003786571,0.003795024,
0.003816380,0.003801616,0.003814887,0.003806718,0.003773104,0.003806926,0.003737138,0.003689363,0.003710197,0.003670776,0.003671394,
0.003659132,0.003683619,0.003602766,0.003646462,0.003635128,0.003603160,0.003611586,0.003571343,0.003566889,0.003470289,0.003458191,
0.003378380,0.003013041,0.002997353,0.003181253,0.003306357,0.003342634,0.003411732,0.003417016,0.003453156,0.003461951,0.003459395,
0.003442089,0.003514576,0.003500629,0.003455028,0.003463706,0.003440335,0.003443703,0.003423927,0.003442143,0.003509514,0.003531923,
0.003482053,0.003477565,0.003419029,0.003426989,0.003460765,0.003436176,0.003440390,0.003449214,0.003440137,0.003412385,0.003356024,
0.003328565,0.003326896,0.003284924,0.003334133,0.003285411,0.003299238,0.003268526,0.003253595,0.003254152,0.003244275,0.003254930,
0.003220143,0.003228541,0.003205344)

forecast_nn = c(-0.0055742370,0.0158053225,0.0131748284,-0.0037854409,0.0151352256,-0.0028655890,0.0017027026,0.0127561264,0.0069359377,
0.0064302675,0.0129493060,-0.0107558082,0.0177475651,0.0074675720,0.0042049967,0.0104959131,0.0061647484,0.0273562074,
-0.0298913539,-0.0027861523,-0.0034990786,-0.0078612675,0.0136435813,-0.0133381446,0.0218311663,0.0011608844,-0.0076247119,
0.0212541515,-0.0095312359,-0.0057810926,0.0052415389,-0.0396474175,-0.0239578101,-0.0059993996,-0.0582889012,-0.0142563977,
0.0034204033,0.0321841871,0.0228072932,0.0098939801,0.0277782414,0.0157948274,0.0088082394,0.0035054735,-0.0055461901,
-0.0094388197,-0.0054676945,0.0071107150,-0.0036656321,-0.0058958911,-0.0002254639,0.0109870903,-0.0080940848,-0.0027239072,
0.0031855604,-0.0055587381,0.0211536626,-0.0072454644,0.0108820874,0.0083630764,0.0015537198,-0.0073377180,-0.0001160707,
0.0023619604,0.0090099197,-0.0196249917,-0.0290056639,0.0010064923,0.0205880529,0.0028138888,0.0066251122,-0.0048052213,
0.0029243110,-0.0059962882,-0.0003584435,0.0032325902,-0.0041919858,0.0151261213,-0.0038535528,0.0025962838)

sqrt(mean((data_test-forecast_nn)^2))
[1] 0.02340158

sqrt(mean((data_test-forecast_wn)^2))
[1] 0.02088056


train_data=c(
-0.0290370370,-0.0068615752,0.0074030204,0.0200232153,-0.0117439812,0.0249069568,0.0025699600,-0.0066110951,0.0108046631,
0.0067777464,-0.0088319088,0.0244580322,0.0024951483,-0.0061366806,0.0088471109,-0.0089260809,0.0115798180,0.0160065111,
-0.0148678414,0.0103542234,0.0083761146,-0.0046145494,0.0094111320,-0.0119727891,0.0110333692,0.0077436582,-0.0002670940,
-0.0113452188,0.0077727151,0.0077127660,-0.0002660282,0.0164835165,0.0052056221,0.0069785474,0.0170223577,0.0007616146,
0.0083081571,0.0231185440,0.0012281995,0.0075572891,0.0217028381,-0.0157461240,0.0346117867,0.0016343684,0.0025617140,
0.0240909091,0.0222222222,-0.0114632502,0.0040295500,0.0156456589,-0.0124944221,0.0239547038,0.0058454211,-0.0158346162,
0.0189859763,0.0030114003,0.0006448839,0.0064075182,0.0053112386,0.0195792543,0.0057983019,-0.0070907195,0.0149958915,
0.0059219931,-0.0115678579,0.0222177338,0.0004037957,0.0026178010,0.0207059752,-0.0011846002,0.0019704433,0.0221579961,
-0.0038684720,0.0015449981,0.0346756152,-0.0026168224,0.0031675051,0.0254221899,-0.0153023599,0.0163220892,0.0089863408,
-0.0155137799,0.0161608906,0.0058907533,0.0032028470,0.0030157885,0.0056447345,-0.0141323792,0.0088652482,-0.0014204545,
-0.0107681263,0.0193593805,-0.0069112174,0.0110410095,0.0033187773,-0.0057976107,0.0114623133,0.0135343498,0.0008558713,
0.0073067120,0.0180210245,0.0066302006,0.0013242841,0.0082088327,0.0113617919,0.0212867355,0.0095972310,0.0042299859,
0.0178488998,0.0016897081,0.0074706510,0.0311669129,-0.0166691695,0.0147950880,0.0171586448,-0.0008732353,0.0123616501,
0.0038659794,0.0002862869,-0.0012899527,-0.0120394546,-0.0045169751,-0.0104534747,-0.0065204505,-0.0019302153,-0.0120210368,
0.0032948929,-0.0183010523,0.0159087498,-0.0039174326,-0.0152975371,0.0215536596,0.0111012433,0.0041273585,0.0160986222,
-0.0081883316,0.0175262175,0.0082632854,-0.0001424907,0.0091769024,-0.0007064142,0.0128312413,-0.0161564626,0.0228500208,
-0.0041718815,0.0015273535,0.0035971223,0.0000000000,-0.0036101083,0.0170601883,-0.0148199446,0.0352752539,0.0162986330,
0.0023603462,0.0197943445,-0.0085558724,0.0191989828,0.0108162495,0.0081087824,0.0042236025,0.0169739895,-0.0052786644,
-0.0071711177,0.0124542125,0.0070320078,0.0118605487,0.0111361213,-0.0034474560,-0.0054984461,0.0041661707,0.0265353418,
0.0018505667,0.0038022814,-0.0103608847,0.0152470480,0.0073964497,0.0071178398,0.0113928292,0.0243025283,-0.0045982045,
0.0094349854,0.0127408994,0.0017101325,0.0024522870,0.0098184122,-0.0123984609,0.0139123103,0.0120783007,0.0003122723,
0.0146666667,0.0050005103,0.0041666667,0.0028374544,-0.0007098672,0.0046431816,0.0022157317,0.0055088141,0.0037916584,
0.0101728395,-0.0045639448,0.0114750883,0.0010776918,-0.0014717425,0.0268308985,-0.0079884504,-0.0169325634,0.0129456091,
-0.0032955316,-0.0079132474,0.0118737330,-0.0089607480,-0.0097364280,-0.0105346850,-0.0678127985,-0.0601935194,-0.0512123004,
-0.0010655932,-0.0041612175,0.0086044319,-0.0109628217,0.0245263280,0.0226084981,-0.0019351167,0.0342970210,0.0146230503,
0.0011900898,0.0261300179,-0.0019001372,0.0108593505,0.0193528569,-0.0001024066,0.0243780597,0.0124321658,-0.0053566114,
0.0027698091,0.0000000000,0.0160599572,0.0080139036,-0.0011599807,0.0128816794,-0.0048902100,0.0045814642,-0.0076945273,
0.0085820540,-0.0048869299,0.0077018161,0.0093255463,-0.0099895348,-0.0002854968,-0.0006666032,0.0007612523,-0.0018112488,
0.0052157421,0.0068751177,-0.0099876344,0.0261232052,-0.0093501636,-0.0024369669,-0.0002812676,0.0064275734,-0.0086441793,
0.0017820296,0.0014984079,0.0155803448,-0.0104331626,0.0063865235,0.0037805440,0.0002765487,-0.0102439933,0.0095923261,
0.0047732697,-0.0044255947,0.0101305102,0.0023672949,-0.0060456169,0.0119467825,-0.0020859786,-0.0043723811,0.0105452907,
-0.0028925246,-0.0009047317,0.0089661974,-0.0022465852,0.0173951435,-0.0023898035,-0.0017733641,0.0103545104,0.0004385580,
-0.0009657594,-0.0123544574,0.0040718775,-0.0121852881,0.0154375441,0.0015853444,-0.0016762241,0.0050908453,0.0059331646,
-0.0077376242,0.0095793782,-0.0142201025,0.0143640637,-0.0133203952,0.0063984574,-0.0042249802,0.0142299349,-0.0130087018,
0.0189704234,-0.0043301290,-0.0027789839,0.0224957555,0.0095013874,-0.0059206631,-0.0016944845,0.0196029571,-0.0136398080,
0.0166418281,-0.0013264799)
Improve this question
$\endgroup$
11
  • $\begingroup$ How do you calculate explained variance? If the RMSE of the flat line is lower than that of the NN, then the standard $R^2$ should be lower, since it's a simple monotonic transformation. Are you sure you used the holdout data for both calculations? $\endgroup$
    – Stephan Kolassa
    Commented May 29 at 9:00
  • $\begingroup$ See this answer: the denominator is the same for both methods, since it does not depend on the forecast, only the actuals; and the numerator is just the sum of squared errors. So I don't see how one method could have a smaller RMSE and a smaller $R^2$ than another method. Unless you are calculating "explained variance" in some other way than $R^2$, or using some other data (e.g., one calculation using only holdout data, but the other one using in-sample). $\endgroup$
    – Stephan Kolassa
    Commented May 29 at 9:03
  • 1
    $\begingroup$ Thanks. I have changed it. $\endgroup$
    – george1994
    Commented May 29 at 11:03
  • 1
    $\begingroup$ However, the intercept-only model doesn't explain as much variance in its forecasts compared to the AR-NN model. How do you figure? What calculation (if it’s even a calculation) shows this? $\endgroup$
    – Dave
    Commented May 29 at 11:18
  • 1
    $\begingroup$ @Dave That is wrong. Thanks for the hint, I edited it. The intercept-only model actually explains moreof the variance. However, its forecasts offer, of course, less variation than the AR-NN's. $\endgroup$
    – george1994
    Commented May 29 at 11:36

1 Answer 1

Reset to default
6
$\begingroup$

The $R^2$ of the flat line forecast is lower than that of the NN. So the flat forecast explains more variation than the NN. It is better. Case closed.

does it make sense to prefer the AR-NN model to obtain future forecasts that offer more variation even though being less accurate?

It depends. What are your ultimate aims in forecasting? If you need an accurate forecast, then your answer should be obvious: the flat line forecast is more accurate than the NN one. However, if you have strategic guidance to use AI/ML in forecasting, and this overrides the quest for accuracy (accuracy is rarely or never the only concern in forecasting), then it may well be prudent to use a more wiggly and more "interesting" ML forecast.

I know forecasters whose flat forecast was most accurate. But their customers complained that they could have calculated a historical mean themselves, so why were they paying an expensive expert? (Insert story about car mechanic and high invoice for simple operation here.) So the forecaster simply added some random noise to the flat line. This made the forecast worse, and the customer happy. See Kolassa (2023, Foresight).

Of course, it is unintuitive that a flat, non-varying, forecasts "explains variation" better than a variable NN forecast. This is because, at least to the best knowledge of your NN, there is no variation to be explained. The NN cannot explain any variation, it can only output variable forecasts that are noise (evidence: its RMSE is higher than that of the flat line forecast).

It is extremely common for flat historical mean forecasts to outperform more complex ones, whether ARIMA or ML-based: Is it unusual for the MEAN to outperform ARIMA? Even if your goal is to predict variations, that does not mean that you can do this with the information you have at hand. Your choice is between a better, less sexy flat forecast and a more "interesting" worse forecast that is more "interesting" solely because it injects useless noise.

If there is no variation that can be predicted based on the time series itself, you can start looking for external drivers that do so: How to know that your machine learning problem is hopeless?

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ So the forecaster simply added some random noise to the flat line. This made the forecast worse, and the customer happy. Amazing how the psychology factors into this. $\endgroup$
    – Dave
    Commented May 29 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.