1
$\begingroup$

We have data values pertaining to BPS (bits per second) traffic on a networking device. We have data from for a particular month (October) from the past 4 years. The data points are available in a 1 minute interval.

So we have 31 days * 24 hours * 60 minutes = 44640 values for each year. Multiply that by 4 and we get ~180,000 data points in our CSV file. We have tried several model including TBATS, ARIMA etc. to make future predictions. For example, we need to predict 44640 values for the october of next year. Problem is that so many data values means that fitting a model takes forever and it's not worth waiting an entire day of processing just to find that model is predicting a straight line trend around the average of previous values.

We are looking for possibly reducing this data using something like exponential smoothing but we do not know how. The past data values are as follows:

8839
29191985
3825997
439694949
5186727
5747251
4814919
489752985
481456366
53712118
51364413
57449919
48123322
473151317
529185483
51284866
528115232
597333333
535883672
594275668
549679615
589267353
54916916
756419719
65492594
587599734
616325563
68434481
63351749
624134894
61665387
697646113
61722678
689499647
6884953
618223888
67283625
7451432
773956231
72682555
748525567
682498934
71892441
8527712
752342356
68912676
746693391
7241629
712685465
748971655
74339677
773571787
81173992
9369364
885665416
969439265
99578482
13281261
127297176
1577597
129853832
13882798
1184388675
115559261
118735937
121685158
1128946618
1157798227
1143165165
11632918
122479785
11341628
116385628
12621439
1172845976
1214564385
1795176
12957522
1183316274
12619916
12519533
135765784
1399453354
1399224864
1372868436
1331569834
137852813
128497677
1297789678
135918171
1294935824
1384582825
1362893276
145228865
142459451
1523728929
1553973554
155186563
149211641
159253766
141712263
14764913
148991924
1562214535
164371933
1546871
163188462
168156746
168938876
16835799
171595761
1663196329
1692558573
17636281
18258581
177213887
1652531676
19852331
1789876462
1789629233
1748867173
181994385
176165681
1969791999
19861387
1947295162
287128848
235583965
1968433253
217279852
2212697598
1953822855
2212983294
2166245238
29418584
28276258
22111712
21361513
2114169137
2314153846
216195463
1948538537
2131395686
22873135
2121744212
2261766416
1952463426
231837712
211836243
21321957
231673786
23586221
237934824
23857991
226835959
22527878
229163528
223611724
242565252
2451523242
242146954
22592296
2524295439
24288788
239426786
2488167389
23614618
2387528327
224687321
2352352153
219349398
211514732
223242859
2114838493
2275546998
224398369
2271632485
2237118326
231972341
223867472
232943687
2616184865
2264386319
241637212
2577586277
2473823845
247444156
261553512
264819946
2643896421
274781277
26189985
272488724
2727773421
269784662
2923184161
2835866726
29476972
313529872
3899199
2979386981
37853218
38881954
297738289
3113766229
32723531
2773715317
3137525998
367757942
36456197
297769411
2882461831
342295362
344496963
39439679
3136141447
3324496997
3253434742
338259
2895698259
31183592
374252594
38459536
312441788
3239434239
3161928
3166617496
31916915
3162371549
319837457
3141362857
32638795
3157587728
3281771348
3142241484
3368347612
327583987
3241925869
33183412
32491351
3383213
3573783926
3299445212
3268651
33138667
329333539
329314786
343676884
342544137
3286497278
34854846
31553839
3553121791
3295782535
333871824
3357511167
364861848
3412626294
33294747
348641163
327124157
3392738132
325626931
33883856
334594742
32942374
31897973
3834926556
365132813
35475637
336384187
366552233
36141892
34887985
34695147
3576451651
3335458644
3326826563
3341539
339894997
337912327
336649223
3555534642
329266359
356461957
3439773899
328435177
3758339514
346635125
361774558
335482465
3486783351
349275
341392357
37215
3497621877
364242974
3624311875
377361582
367461755
363526377
34877241
47832182
371281677
376216515
3741615717
3695335388
3628351931
372717255
3792287845
36549945
372238998
3869247316
3822289851
386173797
372368834
345429379
417153116
38749739
395119594
3746367111
383839372
391378292
367872746
372373178
3625754
379946415
37778181
3746261571
3918932444
386892529
3695653853
3959862748
415346593
42977194
412162553
41582129
41732773
471311973
4415543
3838746827
43135679
41259122
416451147
382524677
3829914759
396922256
392669399
383285533
3915829759
497197157
471337265
494296438
395495
41562899
3973355519
398198495
376359951
397532419
438115941
383579951
39116435
425944659
3961366459
3997619677
4575215
415522986
3947337112
394636114
392714147
385221299
47237153
...and so on

Keeping in mind that we have ~180,000 such past data values, and we wish to predict the next 44640 future values, how do we go about making such a prediction in R?

We are new at R so actual code rather than abstract concepts would help a lot!

EDIT to show how out ARIMA model got stuck in computations:

This is the code we used for auto-arima fitting that got stuck for hours until we had to abort:

mydata <- load.csv("2.csv")
mydata <- ts(mydata, start = c(2012,1), frequency = 44640)
require(forecast)
arimafit <- auto.arima(mydata)

What are we doing wrong in the ARIMA model that's taking so long?

$\endgroup$
  • 1
    $\begingroup$ Do you actually need forecasts on minute level? What process will consume these forecasts, and what decisions will be taken based on these forecasts? I suspect that you will be able to go up to 15 minute buckets, or even hourly ones. (For instance, call center and electric load/price forecasting typically happens in 15 minute buckets.) $\endgroup$ – Stephan Kolassa Jul 8 '16 at 19:49
  • $\begingroup$ We were debating this and turns out we may be able to get away with data observed over a daily interval. This would significantly reduce the computations I guess. $\endgroup$ – learnerX Jul 8 '16 at 19:56
  • $\begingroup$ 180,000 should be a manageable number? What problems are you having? 180000 * 50 variables (totally made up by me) * 8 bytes (sizeof double precision floating point) = 72 megabytes (which isn't a lot for a design matrix). Eg. on my computer, computing X'X for 180000 by 50 matrix takes 0.1 seconds $\endgroup$ – Matthew Gunn Jul 8 '16 at 20:53
  • 1
    $\begingroup$ @Matthew Try running one of these compute-intensive, ML based methods like TBATS sometime. They can take phenomenally long. The issue isn't storage, it's serial CPU cycles. $\endgroup$ – whuber Jul 8 '16 at 21:14
  • $\begingroup$ @whuber You can estimate AR models with OLS in trivial time. Estimating an ARIMA(5,1,5) with 180000 observations using maximum likelihood takes about 12.5 seconds on my machine. Even an ARIMA(50,1,50) < 20 min. If the underlying maximum likelihood optimization problem reduces to a convex problem and is in a reasonable number of parameters, 180000 observations isn't infeasibly huge. (Convex solvers often obtain convergence in ~ 20 steps, regardless of size.) The complexity of the underlying optimization problem, what they're doing is iimportant, and that's missing. IMHO, more info is required. $\endgroup$ – Matthew Gunn Jul 8 '16 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.