# Timing functions in R [closed]

1. I would like to measure the time that it takes to repeat the running of a function. Are replicate() and using for-loops equivalent? For example:

system.time(replicate(1000, f()));
system.time(for(i in 1:1000){f()});


Which is the prefered method.

2. In the output of system.time(), is sys+user the actual CPU time for running the program? Is elapsed a good measure of time performance of the program?

• Just for the record, since I'm clearly far too late to change the course of this question: this is the kind of issue that I think is best suited for StackOverflow. Oct 2, 2010 at 17:22
• @Matt I agree that questions about how one times a program are well suited for SO. I also agree that a literal interpretation of this question (as taken by several of the answers) would place it off-topic here on CV. There does seem to be some statistical interest in designing a timing experiment and in analyzing the results of a such an experiment though.
– whuber
Jan 2, 2015 at 15:31

For effective timing of programs, especially when you are interested in comparing alternative solutions, you need a control! A good way is to put the procedure you're timing into a function. Call the function within a timing loop. Write a stub procedure, essentially by stripping out all the code from your function and just returning from it (but leave all the arguments in). Put the stub into your timing loop and re-time. This measures all the overhead associated with the timing. Subtract the stub time from the procedure time to get the net: this should be an accurate measure of the actual time needed.

Because most systems nowadays can be peremptorily interrupted, it is important to do several timing runs to check for variability. Instead of doing one long run of $N$ seconds, do $m$ runs of about $N/m$ seconds each. It helps to do this in a double loop all in one go. Not only is that easier to handle, it introduces a little bit of negative correlation in each time series, which actually improves the estimates.

By using these basic principles of experimental design, you essentially control for any differences due to how you deploy the code (e.g., the difference between a for loop and replicate()). That makes your problem go away.

Regarding your two points:

1. It's stylistic. I like replicate() as it is functional.
2. I tend to focus on elapsed, i.e. the third number.

What I often do is

N <- someNumber
mean(replicate( N, system.time( f(...) ), trimmed=0.05) )


to get a trimmed mean of 90% of N repetitions of calling f().

(Edited, with thanks to Hadley for catching a thinko.)

• Don't you mean mean(replicate(N, system.time(f(...))), trim = 0.05) ? Oct 1, 2010 at 20:29
• If the f() call is long then it's fine. However, if the f() call is short then any timing call overhead will likely be increasing error measurement. With a single call for system.time() over many repetitions of f() one gets to divide out the error the call until it's some infinitesimal value (and it returns faster).
– John
Oct 2, 2010 at 16:07
• @John: Thanks but I don't quite get what you said. I am still wondering which is better, repeating f() inside or outside system.time()?
– Tim
Oct 4, 2010 at 5:59
• Every call to the system.time() command has some variable time it takes to call that causes some amount of measurement error. This is a small amount. But what if f() is a very brief call? Then this error can be conflated with the time taken to call f(). So, when you call f() 1e5 times inside a single system.time() call the error gets divided down into 1e5 chunks. When you call system.time() for every f() it's impact could be meaningful if time for f() is small. Of course, if all you need is relative timing it doesn't much matter.
– John
Oct 4, 2010 at 10:13
• Oh, and the second part is that it would be faster to just call system.call() once.
– John
Oct 4, 2010 at 10:14

You can also time with timesteps returned by Sys.time; this of course measures walltime, so real time computation time. Example code:

Sys.time()->start;
replicate(N,doMeasuredComputation());
print(Sys.time()-start);


Regarding which timing metric to use, I can not add to the other responders.

Regarding the function to use, I like using the ?benchmark from the rbenchmark package.

They do different things. Time what you wish done. replicate() returns a vector of results of each execution of the function. The for loop does not. Therefore, they're not equivalent statements.

In addition, time a number of ways you want something done. Then you can find the most efficient method.

• mod-tip: post the second part as a comment to the Dirk's answer.
– user88
Oct 2, 2010 at 9:28