# Timing functions in R

1. I would like to measure the time that it takes to repeat the running of a function. Are replicate() and using for-loops equivalent? For example:

system.time(replicate(1000, f()));
system.time(for(i in 1:1000){f()});


Which is the prefered method.

2. In the output of system.time(), is sys+user the actual CPU time for running the program? Is elapsed a good measure of time performance of the program?

-
Just for the record, since I'm clearly far too late to change the course of this question: this is the kind of issue that I think is best suited for StackOverflow. – Matt Parker Oct 2 '10 at 17:22
@Matt I agree that questions about how one times a program are well suited for SO. I also agree that a literal interpretation of this question (as taken by several of the answers) would place it off-topic here on CV. There does seem to be some statistical interest in designing a timing experiment and in analyzing the results of a such an experiment though. – whuber Jan 2 '15 at 15:31

For effective timing of programs, especially when you are interested in comparing alternative solutions, you need a control! A good way is to put the procedure you're timing into a function. Call the function within a timing loop. Write a stub procedure, essentially by stripping out all the code from your function and just returning from it (but leave all the arguments in). Put the stub into your timing loop and re-time. This measures all the overhead associated with the timing. Subtract the stub time from the procedure time to get the net: this should be an accurate measure of the actual time needed.

Because most systems nowadays can be peremptorily interrupted, it is important to do several timing runs to check for variability. Instead of doing one long run of $N$ seconds, do $m$ runs of about $N/m$ seconds each. It helps to do this in a double loop all in one go. Not only is that easier to handle, it introduces a little bit of negative correlation in each time series, which actually improves the estimates.

By using these basic principles of experimental design, you essentially control for any differences due to how you deploy the code (e.g., the difference between a for loop and replicate()). That makes your problem go away.

-

1. It's stylistic. I like replicate() as it is functional.
2. I tend to focus on elapsed, i.e. the third number.

What I often do is

N <- someNumber
mean(replicate( N, system.time( f(...) )[3], trimmed=0.05) )


to get a trimmed mean of 90% of N repetitions of calling f().

(Edited, with thanks to Hadley for catching a thinko.)

-
Don't you mean mean(replicate(N, system.time(f(...))[3]), trim = 0.05) ? – hadley Oct 1 '10 at 20:29
Yup. Thanks for catching that. – Dirk Eddelbuettel Oct 1 '10 at 20:31
Thinko. I like it. – Shane Oct 1 '10 at 20:56
If the f() call is long then it's fine. However, if the f() call is short then any timing call overhead will likely be increasing error measurement. With a single call for system.time() over many repetitions of f() one gets to divide out the error the call until it's some infinitesimal value (and it returns faster). – John Oct 2 '10 at 16:07
@John: Thanks but I don't quite get what you said. I am still wondering which is better, repeating f() inside or outside system.time()? – Tim Oct 4 '10 at 5:59

You can also time with timesteps returned by Sys.time; this of course measures walltime, so real time computation time. Example code:

Sys.time()->start;
replicate(N,doMeasuredComputation());
print(Sys.time()-start);

-

Regarding which timing metric to use, I can not add to the other responders.

Regarding the function to use, I like using the ?benchmark from the rbenchmark package.

-

They do different things. Time what you wish done. replicate() returns a vector of results of each execution of the function. The for loop does not. Therefore, they're not equivalent statements.

In addition, time a number of ways you want something done. Then you can find the most efficient method.

-
mod-tip: post the second part as a comment to the Dirk's answer. – mbq Oct 2 '10 at 9:28