Timing functions in R [closed]

Question

1. I would like to measure the time that it takes to repeat the running of a function. Are replicate() and using for-loops equivalent? For example:

   system.time(replicate(1000, f()));
   system.time(for(i in 1:1000){f()});
   

   Which is the prefered method.

2. In the output of system.time(), is sys+user the actual CPU time for running the program? Is elapsed a good measure of time performance of the program?

Answer 1

For effective timing of programs, especially when you are interested in comparing alternative solutions, you need a control! A good way is to put the procedure you're timing into a function. Call the function within a timing loop. Write a stub procedure, essentially by stripping out all the code from your function and just returning from it (but leave all the arguments in). Put the stub into your timing loop and re-time. This measures all the overhead associated with the timing. Subtract the stub time from the procedure time to get the net: this should be an accurate measure of the actual time needed.

Because most systems nowadays can be peremptorily interrupted, it is important to do several timing runs to check for variability. Instead of doing one long run of $N$ seconds, do $m$ runs of about $N/m$ seconds each. It helps to do this in a double loop all in one go. Not only is that easier to handle, it introduces a little bit of negative correlation in each time series, which actually improves the estimates.

By using these basic principles of experimental design, you essentially control for any differences due to how you deploy the code (e.g., the difference between a for loop and replicate()). That makes your problem go away.

Answer 2

Regarding your two points:

1. It's stylistic. I like replicate() as it is functional.
2. I tend to focus on elapsed, i.e. the third number.

What I often do is

N <- someNumber
mean(replicate( N, system.time( f(...) )[3], trimmed=0.05) )

to get a trimmed mean of 90% of N repetitions of calling f().

(Edited, with thanks to Hadley for catching a thinko.)

Answer 3

You can also time with timesteps returned by Sys.time; this of course measures walltime, so real time computation time. Example code:

Sys.time()->start;
replicate(N,doMeasuredComputation());
print(Sys.time()-start);

Answer 4

Regarding which timing metric to use, I can not add to the other responders.

Regarding the function to use, I like using the ?benchmark from the rbenchmark package.

Answer 5

They do different things. Time what you wish done. replicate() returns a vector of results of each execution of the function. The for loop does not. Therefore, they're not equivalent statements.

In addition, time a number of ways you want something done. Then you can find the most efficient method.