Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.
  1. I would like to measure the time that it takes to repeat the running of a function. Are replicate() and using for-loops equivalent? For example:

    system.time(replicate(1000, f()));
    system.time(for(i in 1:1000){f()});
    

    Which is the prefered method.

  2. In the output of system.time(), is sys+user the actual CPU time for running the program? Is elapsed a good measure of time performance of the program?

share|improve this question
3  
Just for the record, since I'm clearly far too late to change the course of this question: this is the kind of issue that I think is best suited for StackOverflow. –  Matt Parker Oct 2 '10 at 17:22

5 Answers 5

up vote 11 down vote accepted

For effective timing of programs, especially when you are interested in comparing alternative solutions, you need a control! A good way is to put the procedure you're timing into a function. Call the function within a timing loop. Write a stub procedure, essentially by stripping out all the code from your function and just returning from it (but leave all the arguments in). Put the stub into your timing loop and re-time. This measures all the overhead associated with the timing. Subtract the stub time from the procedure time to get the net: this should be an accurate measure of the actual time needed.

Because most systems nowadays can be peremptorily interrupted, it is important to do several timing runs to check for variability. Instead of doing one long run of $N$ seconds, do $m$ runs of about $N/m$ seconds each. It helps to do this in a double loop all in one go. Not only is that easier to handle, it introduces a little bit of negative correlation in each time series, which actually improves the estimates.

By using these basic principles of experimental design, you essentially control for any differences due to how you deploy the code (e.g., the difference between a for loop and replicate()). That makes your problem go away.

share|improve this answer

Regarding your two points:

  1. It's stylistic. I like replicate() as it is functional.
  2. I tend to focus on elapsed, i.e. the third number.

What I often do is

N <- someNumber
mean(replicate( N, system.time( f(...) )[3], trimmed=0.05) )

to get a trimmed mean of 90% of N repetitions of calling f().

(Edited, with thanks to Hadley for catching a thinko.)

share|improve this answer
1  
Don't you mean mean(replicate(N, system.time(f(...))[3]), trim = 0.05) ? –  hadley Oct 1 '10 at 20:29
    
Yup. Thanks for catching that. –  Dirk Eddelbuettel Oct 1 '10 at 20:31
1  
Thinko. I like it. –  Shane Oct 1 '10 at 20:56
2  
If the f() call is long then it's fine. However, if the f() call is short then any timing call overhead will likely be increasing error measurement. With a single call for system.time() over many repetitions of f() one gets to divide out the error the call until it's some infinitesimal value (and it returns faster). –  John Oct 2 '10 at 16:07
    
@John: Thanks but I don't quite get what you said. I am still wondering which is better, repeating f() inside or outside system.time()? –  Tim Oct 4 '10 at 5:59

You can also time with timesteps returned by Sys.time; this of course measures walltime, so real time computation time. Example code:

Sys.time()->start;
replicate(N,doMeasuredComputation());
print(Sys.time()-start);
share|improve this answer

Regarding which timing metric to use, I can not add to the other responders.

Regarding the function to use, I like using the ?benchmark from the rbenchmark package.

share|improve this answer

They do different things. Time what you wish done. replicate() returns a vector of results of each execution of the function. The for loop does not. Therefore, they're not equivalent statements.

In addition, time a number of ways you want something done. Then you can find the most efficient method.

share|improve this answer
    
mod-tip: post the second part as a comment to the Dirk's answer. –  mbq Oct 2 '10 at 9:28

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.