1
$\begingroup$

I'm performing some processing of rather large amounts of data. I did a hundred tests with some constant number of records (i.e 1, 2, 3, 4, 5, …, 99, 100 millions) and measured execution time with time(1). So, I have the following CSV with results (the columns are the following: number of records, extra processing, elapsed time, user time, sys time):

1000000,false,4.59,13.39,0.49
1000000,true,7.84,22.82,0.69
2000000,false,10.77,33.84,1.26
2000000,true,18.71,56.82,1.98
3000000,false,19.29,62.54,2.17
3000000,true,28.17,85.45,2.87
4000000,false,26.89,85.87,3.20
4000000,true,43.55,136.69,4.50
5000000,false,41.09,121.94,4.46
5000000,true,64.75,187.28,6.41
6000000,false,56.55,162.98,6.41
6000000,true,82.02,235.38,9.03
7000000,false,70.87,207.14,8.05
7000000,true,123.77,309.63,12.70
8000000,false,107.07,275.97,11.68
8000000,true,109.04,326.34,12.25
9000000,false,89.05,270.70,10.78
9000000,true,124.43,373.96,13.74
10000000,false,98.57,301.04,11.70
10000000,true,136.19,408.59,15.19
11000000,false,97.29,300.00,11.72
11000000,true,143.84,430.53,16.11
12000000,false,107.98,333.13,13.11
12000000,true,153.02,460.15,16.36
13000000,false,119.23,364.84,14.61
13000000,true,180.09,517.06,20.12
14000000,false,140.31,431.75,17.30
14000000,true,196.17,589.55,22.31
15000000,false,145.82,449.05,18.06
15000000,true,198.22,597.39,21.88
16000000,false,150.61,462.79,18.15
16000000,true,209.92,631.77,22.54
17000000,false,160.53,493.84,19.93
17000000,true,219.94,662.96,24.75
18000000,false,167.71,516.39,20.07
18000000,true,232.46,700.50,25.96
19000000,false,171.61,527.73,20.61
19000000,true,242.55,730.42,26.71
20000000,false,183.04,563.93,21.68
20000000,true,257.45,773.88,28.53
21000000,false,195.61,596.88,23.33
21000000,true,264.99,802.26,28.98
22000000,false,200.51,615.76,24.07
22000000,true,276.23,834.92,31.05
23000000,false,212.92,654.44,24.84
23000000,true,296.50,880.92,31.98
24000000,false,288.01,800.15,32.92
24000000,true,314.12,1003.80,35.99
25000000,false,235.02,764.84,29.11
25000000,true,302.02,962.37,34.31
26000000,false,227.93,742.00,28.98
26000000,true,313.74,998.78,36.27
27000000,false,274.73,832.92,33.66
27000000,true,350.24,1122.48,41.00
28000000,false,263.68,864.71,32.52
28000000,true,361.61,1150.99,40.87
29000000,false,278.61,915.42,34.71
29000000,true,359.13,1152.33,40.00
30000000,false,266.44,876.82,31.88
30000000,true,362.48,1162.83,40.51
31000000,false,272.13,891.96,32.84
31000000,true,370.74,1188.92,42.13
32000000,false,275.06,902.61,34.28
32000000,true,394.96,1241.18,44.27
33000000,false,336.71,1051.36,40.17
33000000,true,424.30,1386.18,47.78
34000000,false,301.45,1007.21,37.04
34000000,true,406.05,1323.55,45.72
35000000,false,348.93,1082.58,40.53
35000000,true,534.95,1536.01,57.48
36000000,false,369.94,1183.06,46.72
36000000,true,462.07,1510.04,53.58
37000000,false,326.20,1080.18,40.61
37000000,true,443.66,1447.07,50.52
38000000,false,331.37,1100.33,41.04
38000000,true,446.12,1456.71,50.68
39000000,false,337.18,1118.88,42.16
39000000,true,464.21,1511.98,52.77
40000000,false,341.30,1138.45,42.58
40000000,true,469.74,1531.80,54.10
41000000,false,345.26,1151.26,43.82
41000000,true,479.28,1563.01,54.49
42000000,false,360.36,1201.11,45.85
42000000,true,489.55,1594.97,56.24
43000000,false,365.98,1218.86,45.41
43000000,true,499.44,1628.66,57.59
44000000,false,373.09,1245.24,46.16
44000000,true,508.68,1657.16,58.67
45000000,false,378.15,1258.53,46.75
45000000,true,526.17,1712.68,61.13
46000000,false,387.08,1290.10,48.22
46000000,true,535.58,1743.08,60.87
47000000,false,396.41,1325.43,48.35
47000000,true,544.20,1775.30,60.48
48000000,false,406.32,1357.40,49.43
48000000,true,551.06,1796.34,61.25
49000000,false,411.53,1372.70,50.83
49000000,true,558.44,1826.14,61.27
50000000,false,416.86,1387.44,51.33
50000000,true,575.04,1874.64,66.63
51000000,false,425.44,1421.51,52.12
51000000,true,590.92,1930.07,64.84
52000000,false,433.62,1449.04,53.13
52000000,true,592.16,1933.14,65.54
53000000,false,439.01,1465.96,53.55
53000000,true,611.13,1995.80,65.34
54000000,false,446.41,1490.79,54.27
54000000,true,617.46,2015.61,69.06
55000000,false,460.24,1536.30,56.68
55000000,true,632.65,2068.60,70.60
56000000,false,467.23,1560.03,56.48
56000000,true,645.12,2108.10,71.86
57000000,false,485.92,1624.96,59.39
57000000,true,649.07,2119.50,71.34
58000000,false,491.89,1636.55,59.68
58000000,true,667.78,2180.27,74.36
59000000,false,501.20,1669.71,61.40
59000000,true,677.74,2212.39,75.18
60000000,false,502.95,1680.30,60.06
60000000,true,681.43,2224.64,74.31
61000000,false,501.97,1678.04,61.37
61000000,true,693.97,2264.24,77.11
62000000,false,515.07,1714.68,62.53
62000000,true,704.49,2300.10,77.64
63000000,false,527.61,1761.55,64.81
63000000,true,722.49,2360.76,80.00
64000000,false,643.31,2000.42,75.64
64000000,true,827.64,2624.58,94.74
65000000,false,635.29,2033.04,80.60
65000000,true,791.18,2567.88,90.94
66000000,false,562.68,1870.04,70.88
66000000,true,773.64,2510.22,89.11
67000000,false,569.15,1893.32,71.40
67000000,true,775.40,2514.35,87.48
68000000,false,576.29,1920.36,72.44
68000000,true,783.38,2545.92,88.21
69000000,false,691.88,2155.46,84.36
69000000,true,974.73,2967.98,108.82
70000000,false,673.63,2222.80,84.96
70000000,true,855.05,2782.12,98.22
71000000,false,617.71,2053.45,77.34
71000000,true,829.90,2702.96,93.55
72000000,false,617.02,2048.64,77.98
72000000,true,846.30,2744.12,97.75
73000000,false,617.92,2055.50,77.24
73000000,true,853.64,2774.81,100.08
74000000,false,631.01,2100.00,79.01
74000000,true,855.87,2784.63,97.28
75000000,false,654.11,2167.72,82.09
75000000,true,919.01,2968.60,104.51
76000000,false,645.22,2146.43,80.83
76000000,true,885.67,2878.00,100.00
77000000,false,659.80,2199.18,80.80
77000000,true,911.96,2971.48,101.03
78000000,false,674.53,2247.21,81.24
78000000,true,921.12,3002.63,102.25
79000000,false,716.36,2309.28,86.00
79000000,true,1182.82,3486.01,124.27
80000000,false,829.22,2603.51,98.12
80000000,true,1155.12,3553.06,124.61
81000000,false,861.48,2704.98,104.04
81000000,true,1184.15,3618.10,126.96
82000000,false,869.57,2746.88,103.03
82000000,true,1275.98,3763.45,136.28
83000000,false,963.59,2893.50,113.94
83000000,true,1249.66,3818.42,136.10
84000000,false,876.93,2812.39,104.97
84000000,true,1276.48,3882.76,138.26
85000000,false,847.49,2804.06,106.89
85000000,true,1103.37,3582.76,124.32
86000000,false,768.63,2548.75,93.34
86000000,true,1230.58,3844.48,134.97
87000000,false,852.95,2814.36,106.42
87000000,true,1093.79,3527.53,126.06
88000000,false,802.80,2639.86,101.79
88000000,true,1085.99,3500.96,124.66
89000000,false,787.44,2599.36,99.24
89000000,true,1081.94,3477.62,126.75
90000000,false,792.19,2610.28,102.10
90000000,true,1091.05,3515.17,125.12
91000000,false,805.38,2658.47,101.26
91000000,true,1101.55,3552.66,126.05
92000000,false,820.35,2701.84,102.18
92000000,true,1107.64,3581.94,124.94
93000000,false,825.74,2723.28,101.97
93000000,true,1251.92,3919.69,140.60
94000000,false,1102.45,3239.35,129.50
94000000,true,1268.50,4087.58,141.99
95000000,false,853.01,2826.20,104.67
95000000,true,1154.24,3738.96,129.43
96000000,false,854.23,2832.28,105.32
96000000,true,1154.56,3740.70,129.46
97000000,false,848.93,2807.60,105.36
97000000,true,1159.24,3760.43,126.24
98000000,false,848.86,2810.26,103.06
98000000,true,1157.35,3756.42,126.70
99000000,false,860.61,2838.28,105.35
99000000,true,1170.72,3791.50,129.74
100000000,false,868.81,2876.12,108.43
100000000,true,1171.27,3795.40,130.19

How can I predict the time to process, for example, a billion of records? I'm going to use R to have the ability to visualize data as well.

Previously asked on StackOverflow.

$\endgroup$
0

1 Answer 1

1
$\begingroup$

It looks like your times are growing linearly, so it seems reasonable to extrapolate from a linear model, as commenter Zheyuan Li suggests.

enter image description here

The caveat is that who know what really happens after 100M? At some point you may cross some memory threshold which will trigger memory paging from the OS, for example. You can sort of see something like that happening with this data: sometimes it jumps above the line and then settles back down. If you can understand and model those jumps, your prediction will be better but still at the mercy of the extrapolation unknowns.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.